# An Agent That Fixes Its Own Mistakes - Saloni Gandhi & Ziwei Guo, LinkedIn

Ziwei Guo, LinkedIn · Saloni Gandhi, LinkedIn

[Talk page](https://ai.engineer/talks/yH7_SIIgI-Y-agent-that-fixes-its-own-mistakes) · [Video](https://www.youtube.com/watch?v=yH7_SIIgI-Y)

## An Agent That Fixes Its Own Mistakes

LinkedIn’s first GEPA-optimized prompt for its user-understanding agent reached production after measurement and human review. A separate review of the evaluators exposed a subtler error. The system could blame the scoring rubric when the judge had misapplied it.

### Which part of the agent should a production failure change?

[0:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1s)

A production trace records what LinkedIn’s career agent did on a real request. When something goes wrong, the engineers maintaining the agent still have to decide what to change. Ziwei Guo and Saloni Gandhi built a system around one part of the career experience, the agent that infers a member’s goals and preferences. Their starting point was an evaluator that could inspect the traces and identify failures that mattered to members.

The team calls the process recursive self-improvement, or RSI. The agent logs its output and the steps that produced it. An improvement engine uses those records to evaluate the output, investigate the cause of a failure and propose an edit. Once a validated edit enters production, the revised agent produces the traces for a later round.

Several parts of the agent harness, the software around the base model, can influence an answer. Planning logic chooses an execution path. Context and memory preserve session state. Tool interfaces give the agent access to APIs and datasets, and storage retains outputs and generation logs. The team chose prompt edits as the simplest changes to propose and test while keeping the base model fixed.

Guo describes a progression in what the optimizer might eventually be allowed to edit. It could change how the agent’s context is structured, design the workflow, then modify the harness code that runs it. The most ambitious step would let the optimizer revise its own code. The production work in this talk tackles the first step, changing the instructions.

### How an edit earns its place in the next run

[5:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=309s)

One iteration has seven steps. The first two produce evidence. Run the agent on a query, keeping its output and execution trace. Then have a judge evaluate the run and explain where the output succeeded or fell short.

Steps three and four turn a judgment into a proposed edit. A reflection agent investigates what could account for the score. Possibilities include a model error, a prompt that left something out, a tool problem or a mistake in evaluation. The diagnosis guides the proposal. This matters even when the only edit the optimizer can make is to a prompt. A low score by itself gives it little reason to change a particular instruction.

The last three steps test the proposal. The system evaluates candidates using the same judge and rubric, incorporates a promising edit into the prompt, then compares the revised agent with the previous version on a held-out set. That final comparison asks whether the change also helps on examples outside those used to choose it. A validated update becomes the baseline for the next iteration.

### From a member ID to an inferred job preference

[9:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=560s)

The user-understanding agent takes a member ID. It fetches profile data, search history, location and preferences, then synthesizes those signals into fields describing the member. One field records which job titles the person appears to prefer. Entries might include *network manager* or *account manager*. What would let a reviewer check that inference?

Alongside the titles, the agent supplies its confidence, citations to member evidence and an explanation. If *network manager* appears frequently in saved searches, a citation can point to the activity behind that inference. The explanation can distinguish the roles of two sources of evidence. A preference the member stated can establish the primary title, while saved searches can support a related title the member might also consider.

The first-level judge, L1, assesses the output across 15 task dimensions using product-defined rubrics. One member’s output received an overall score of 2.31. Underneath it, fields such as goal summary and preferred job title each get a score and notes explaining the verdict. On this scale, 1 means poor, 2 fair and 3 very good. The field-level notes give the next stage a specific claim to investigate when an otherwise reasonable output scores poorly.

### Counting recurring problems across different members

[13:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=837s)

Reflection starts by grouping failing rows from members’ traces according to the dimension that scored poorly. This first grouping just uses the existing scores. Within each group, a second-level judge, L2, examines each row and assigns a free-text diagnosis and a cause tag. A shared symptom, such as a low score for preferred titles, can have different causes across members.

Free text creates a counting problem. One diagnosis says the member’s profile data is missing. Another says the profile gives an incomplete picture of the member’s browsing history. In Guo’s example, both identify a problem at the same level of detail for the team’s purposes. The available profile evidence cannot support the agent’s conclusion. Merging these descriptions under a common issue label makes their combined frequency visible. Ranking the labels then requires no further model judgment.

Who uses the diagnoses after a run? Engineers get a scorecard showing which problems recur most often. L2 gets something different on a later run. The system saves prior diagnoses and supplies them as context when L2 investigates fresh failures.

Where a row’s diagnosis goes
The scorecard counts issues across members in this run. The return path carries prior diagnoses into a later run, when L2 has new rows to examine.
- Failing rows — Group traces by the dimension that scored poorly.
- L2 diagnosis — Assign each row a cause tag and a free-text explanation.
- Common issue labels — Merge descriptions of the same underlying problem.
- Scorecard for engineers — Rank recurring issues by frequency.
- Saved root causes — Keep diagnoses available for future investigations.
- Failing rows → L2 diagnosis (Pass each row for diagnosis)
- L2 diagnosis → Common issue labels (Collect near-duplicate diagnoses)
- Common issue labels → Scorecard for engineers (Count occurrences of each issue)
- Common issue labels → Saved root causes (Retain the identified causes)
- Saved root causes → L2 diagnosis (Supply prior diagnoses on a later run)

### The first GEPA-optimized prompt reaches production

[18:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1117s)

GEPA uses the judge’s evidence and the reflection results to propose changes to the agent’s instructions. It works over mini-batches of sampled traces. Candidate prompts advance to validation, where the team can compare how well they address the observed problems. A prompt that survives the held-out comparison can replace the previous production baseline.

The team used human annotations to improve both L1 and L2 and studied variation across agent runs and model judgments. Meanwhile, engineers packaged the seven steps into a workflow that could suggest edits weekly, every two weeks or monthly, depending on need. Scheduling proposals left the release decision with people. The first GEPA-optimized pull request shipped after quantitative measurement and human review.

Gandhi describes a second outcome from comparing the evaluators’ decisions with human reviews. Disagreements exposed recurring faults in both the scoring rules and the system that explained failures. That raised two separate questions. Could L1 recognize a poor answer? And when L1 flagged one, could L2 identify what had gone wrong?

### Human reviewers agreed most often with a score of 3

[25:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1512s)

Where was L1 reliable enough to guide an edit? The team analyzed about 180–200 human reviews, grouping them first by the score L1 had assigned. Full agreement was roughly 55% for outputs scored 1, 57% for those scored 2 and 80% for those scored 3. The judge recognized strong work more reliably than it distinguished degrees of weak work. Those low scores were precisely the ones that sent examples into reflection. [Score agreement in the recording, 25:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1531s)

Grouping the reviews by field reveals a different source of disagreement. Preferred job titles had among the weakest agreement. For domain or area of expertise, agreement was about 47%, and for work type it was about 50%. The two breakdowns point to different questions for the rubric. What distinguishes a 1 from a 2? What evidence should count for this particular field?

The annotations put concrete cases behind those figures.

- **Preferred titles.** L1 could accept a plausible title even when the agent had overlooked an important signal of the member’s intent. A reviewer checking the input could spot a missing preference that a check of the returned titles alone would miss.
- **Fields of expertise.** In the team’s example, *software engineering* is a domain, *machine learning engineer* is a role and *machine learning* is a skill. The rubric blurred those categories. It also gave little guidance on how much credit to award when an answer mixed correct and incorrect entries.
- **Formatting and inference.** The judge sometimes penalized harmless differences in how a value was written. A harder disagreement arose when the agent inferred a preference from current employment and the member had never stated that preference explicitly. Reviewers needed a rule for when that inference was justified.

Of the 37 judgments reviewers identified as too harsh, 31 involved an output L1 had scored 1. That concentration made partial credit a practical priority. A rule for recognizing the correct parts of a mixed answer could keep that answer from being treated as a complete failure.

### When the scoring rule is wrong, and when the judge is wrong

[28:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1711s)

Take the work-type value *full-time*. Capitalization alone could cost the agent credit. The revised rubric accepted formatting variants with the same meaning. The team also added guidance for normalizing job titles, handling concurrent employment and assessing inferences about seniority and geography. It revised the scoring rules to distinguish a poor answer from a partially correct one.

After the rubric changes, reported L1 agreement with human reviewers rose from about 57% to 64%. Mean error also fell, though the scale of that measure was unspecified. The agreement gain measures a change in the evaluator. Engineers had made the product’s definition of a correct judgment more explicit.

Reviewing L2 exposed a different problem. Human reviewers agreed more often when L2 attributed a failure to the underlying model. Agreement was much lower when it blamed the rubric. The original list of cause tags gave L2 no way to say that L1 had made a mistake. Faced with a reasonable rule that L1 had applied incorrectly, L2 sometimes labeled the rule itself defective.

L2 also sometimes repeated L1’s rationale without checking it against the agent’s output. A single mistaken judgment could then look like a finding that two evaluators agreed on. If GEPA acted on that diagnosis, it could propose an edit to a component that had behaved correctly. Improving L2’s ability to identify the cause became a priority in its own right.

For decisions about individual edits, the team used majority voting to reduce the influence of any one model judgment and held-out evaluations to check whether gains carried over. Gandhi also reports an improvement involving the threshold for adopting an edit and the reported confidence interval, without specifying a value or direction of change for either measure.

### What another agent team would need

[33:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1983s)

At the scale of a team, adoption poses a different question. What fraction of its prompt changes can RSI reliably generate and validate? LinkedIn also wants to measure downstream response quality and the time engineers save maintaining an agent. These are proposed measures for broader use. A count of edits would rise even if the quality of the answers stayed flat.

Many teams already have an agent that answers questions, summarizes material or completes a workflow. Their ability to evaluate it varies. Some have bespoke tests, others inspect outputs by hand, and relatively few have a reliable evaluation they can run repeatedly. That gap limits what a shared optimizer could do for them. To propose a targeted fix, it needs a diagnosis. To investigate the cause, the diagnostic system needs a dependable way to identify a failure in the first place.

LinkedIn’s proposed shared infrastructure would give teams five capabilities.

- **Evaluation.** Reuse evaluation frameworks and metrics, with judges that teams can calibrate against human feedback.
- **Feedback collection.** Collect execution traces continuously so recurring failures can be detected and investigated.
- **Optimization.** Use reflection and prompt optimization to generate and test candidate changes in a repeatable way.
- **Validation.** Compare candidates on held-out examples and run regression tests for behavior an edit might damage.
- **Self-service adoption.** Package the components so a team with an existing agent can use them without building the whole pipeline.

Gandhi frames the longer-term goal in terms of a slope. The target agent infers member preferences. The improvement engine finds failures, investigates them and tests changes. If agent quality is a function of time, an individual release can raise quality at one point on the curve. The ambition for the engine is to increase the rate of improvement across releases. For that slope to mean anything, the team has to be able to trust that a measured gain reflects a better answer for the member.

## Key takeaways

- Checking the titles an agent returned tests plausibility. Checking those titles against the member’s intent signals can also reveal a preference the agent omitted.
- A diagnostic judge needs a way to identify an incorrect score and evidence against which to check it. LinkedIn’s reflection agent sometimes blamed a sound rubric or repeated the first judge’s error.
- LinkedIn compared proposed prompt edits under a common rubric, tested the selected change on held-out examples and kept people involved in the production decision. The first GEPA-optimized pull request passed measurement and human review before release.

## Slides

### Outcomes and Learnings

Saloni introduces a lesson from the team’s work on RSI reliability. The quality of the evaluator mattered as much as the quality of the optimization system.

[23:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1406s) [The words “Outcomes and Learnings” appear on two lines in large dark type at the upper left of a white slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/2f46cd3d927d2e908eb9617aa1bbdc3f4f24f9cc17d0681a190a2cb2cdd61ce6.webp)

Outcomes and
Learnings

in
PROPRIETARY + CONFIDENTIAL
12

### RSI production deployment and evaluation quality

LinkedIn shipped its first RSI-generated changes after human review and quantitative measurement. The team studied where its judge disagreed with reviewers, refined the rubric and reduced evaluation variance.

[23:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1436s) [Two panels report that the first RSI-generated changes reached production after human review and quantitative measurement, and that human annotation helped improve the evaluation judge and rubric and reduce evaluation variance.](https://assets.ai.engineer/sites/aiecode2025/sha256/a57e018dfd148b6cdba801daaa51a836f0436defdccddcf56e57d519f37c9ef3.webp)

1
First RSI system changes landed in production 🚀🚀🔥
Successfully deployed initial system updates generated by RSI after rigorous human review and quantitative measurement.

2
Eval Judge and Rubric quality
• Dedicated substantial effort to human-annotating the MU eval judge
• Improved overall judge quality and significantly reduced evaluation variance
• Enhanced rubric robustness for more reliable automated evaluation

in
PROPRIETARY + CONFIDENTIAL
13

### MU Eval L1 judge findings and rubric recalibration

Of the 37 judgments reviewers flagged as too harsh, 31 gave a score of 1. The team proposed clearer partial-credit rules so the judge would reserve score 1 for substantially incorrect or unusable answers.

[26:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1598s) [Bullets describe missed job-title preferences, inconsistent partial credit for expertise, formatting penalties and disputes over inferred preferences. Of 37 judgments reviewers marked too harsh, 31 received score 1. The slide calls for clearer rules for partial credit and inferred evidence.](https://assets.ai.engineer/sites/aiecode2025/sha256/86dd45128618494ec7ee6fe23d80579a98e8c9a69cc8ed7118f55d2f0272e5a9.webp)

Human Review - MU Eval (L1) Judge findings

Summary of Key Patterns

• Preferred Job titles: The judge often checks whether returned titles are plausible but misses expected titles or intent signals.
• Primary field of Expertise: The distinction between a domain, function, and skill is unclear, and partial credit for mixed-quality lists is inconsistent in the rubric.
• Formatting & Inferences: The judge sometimes penalizes formatting differences too severely or inconsistently infers target preferences from current employment.
• Score Alignment & Rubric Recalibration: Scores 1 and 2 have similar full-judgment alignment, 55.7% and 57.5%, while score 3 reaches 80.4%. 31 of the 37 too-harsh annotations occur on score 1 decisions.
  ○ This suggests we calibrate the boundary between scores 1 and 2, with clearer partial-credit rules through rubric changes so that score 1 is reserved for outputs that are substantially incorrect or unusable.
• Recurring Disputes:
  ○ Key Areas: Employment status, Geo, expertise & rank: Human reviewers frequently accept reasonable inferences, whereas the L1 rubric strictly required explicit evidence.
  ○ Gap: Need aligned guidelines on implicit vs. explicit candidate signal handling.

in
PROPRIETARY + CONFIDENTIAL
15

### Human Review – RSI Reflection Findings

Reviewers agreed in 70% of the 72 cases where the reflection agent blamed the model, but only 27% of the 22 where it blamed the rubric. The agent had no way to flag an error by the L1 judge, so it sometimes blamed the rubric instead.

[29:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1790s) [Four charts compare L2 reflections with human judgments by assigned cause, dimension and stated confidence. Reviewers agreed with 70% of model attributions, 50% of context attributions and 27% of rubric attributions. Agreement was 68% in the middle confidence band and 64% in the highest.](https://assets.ai.engineer/sites/aiecode2025/sha256/f2975d1dbebdce1a1e1a4d187073a16974d6fdb85a6225341f183a65fdad0d98.webp)

Human Review - RSI Reflection findings

Score alignment pattern
• Humans agree mostly with model issues
• low agreement on other causes – L2 could cheat → blame “model”
• Confidence scores did not align with performance

Dimension alignment pattern
• Lowest: employment type 33%, workplace type 44%, career stage 50%

Why humans disagreed
• Wrong root cause 18: no way to blame L1 judge→ fell back to "rubric"
• Nothing failed 15: 13 were L1 too harsh
• L2 reused L1's note; L2 didn’t put validation efforts

Human agrees, by cause L2 named
Y-axis: %
Y-axis ticks: 0, 50, 100
model (n=72): 70%
context (n=4): 50%
rubric (n=22): 27%

Cause: L2 vs human (n=35)
Rows: L2 said
Columns: Human said
         model  rubric  other
model       13       7      3
rubric       8       0      2
context      0       1      1

Human agrees, by dimension
X-axis ticks: 0, 20, 40, 60, 80, 100
Job nature: 33%
Work environment: 44%
Employment status: 50%
Career level: 50%
Desired locations: 57%
Field knowledge: 62%
Experience level: 62%
Preferred domains: 71%
Career goals: 80%
Sector type: 83%

Human agrees, by confidence L2 stated
Y-axis: %
Y-axis ticks: 0, 50, 100
<0.85 (n=16): 31%
0.85-0.95 (n=28): 68%
≥0.95 (n=54): 64%

in
PROPRIETARY + CONFIDENTIAL
17

### Quality Improvements

The team wanted the L1 judge to score more like a human reviewer and the L2 reflection to identify what went wrong. A mistaken diagnosis could send a prompt change to the wrong part of the system.

[31:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1886s) [Section divider headed “Quality Improvements” with the subtitle “RSI reflections and Judge.”](https://assets.ai.engineer/sites/aiecode2025/sha256/b4c9f79d9a0957a23e10aee75c6919a68a316b4d47b2fd45dcf5a492cb2daebd.webp)

Quality
Improvements

RSI reflections and Judge

in
PROPRIETARY + CONFIDENTIAL
18

### RSI Reflection Quality Improvements

Once L2 could flag an L1 judging error, it blamed the rubric in 8 of 300 verdicts, down from 37 of 300. Human agreement with those rubric diagnoses rose from 22% to 62%. Across all verdicts, agreement rose from 77% to 80%.

[31:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1890s) [A table lists changes to the L1 and L2 judges. The chart shows overall human agreement rising from 77% to 80%. For verdicts that attributed the issue to the rubric, agreement rose from 22% of 37 cases to 62% of 8 cases.](https://assets.ai.engineer/sites/aiecode2025/sha256/ab4ae22a4107af0fb593e84dbb9ec7c90ba33852c939f3723fd7a8991f9536a1.webp)

RSI Reflection Quality Improvements

Issue | Fix | Before → After
No "L1 judge was wrong" cause | Added "L1 judge wrong" cause | rubric blames 37 → 8
L2 reused L1's note as evidence | Reads agent output + reasoning | repeats agree 80 → 88%
Confidence float uninformative | Low / medium / high bands | shipped
L1 too harsh (37 of 180 reviews) | Production rubric + formatting rules | matches human 24 → 42%
No "nothing failed" verdict (15) | Open: deeper analysis ongoing | next

Human agrees, by cause L2 named · before vs after
Y-axis: Human agrees (%)
Y-axis ticks: 0, 25, 50, 75, 100
Legend: Before, After
All verdicts: Before 77% (n=300), After 80% (n=300)
"model": Before 87% (n=253), After 83% (n=275)
"rubric": Before 22% (n=37), After 62% (n=8)
"l1_judge" (new): After 33% (n=15)

in
PROPRIETARY + CONFIDENTIAL
19

### Acceleration Plan

The speakers propose standardizing evaluation, reflection and optimization so other agent teams could reuse the self-improvement loop developed for user understanding.

[33:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2028s) [White section divider with the large heading “Acceleration Plan” in the upper left.](https://assets.ai.engineer/sites/aiecode2025/sha256/3106134f99dfcef742f10a9cd61b617bd6d300957459ade9f93bef3d279b304b.webp)

Acceleration Plan

in
PROPRIETARY + CONFIDENTIAL
21

### End-to-end Evals Current State

Some teams evaluate agents with fragile custom workflows or one-off checks. Reliable evaluations help agents diagnose failures. Most agents lack both that reflection step and the recursive self-improvement loop that builds on it.

[34:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2055s) [Four layers run from Base Agent at the bottom through Agentic Evals and Agentic Reflection to RSI Loop at the top. Annotations describe bespoke or one-off evaluations and say most agents lack the upper two layers.](https://assets.ai.engineer/sites/aiecode2025/sha256/0b656b0e4450919e206f5bf62d8136717a171269fcb2633cb3b8c54a4f9ec2bf.webp)

End-to-end Evals Current State
Leveling up Evals is prerequisite to RSI

RSI Loop
Does not exists for most agents

Agentic Reflection
Does not exists for most agents

Agentic Evals
Some have it as a bespoke unstable workflow. Some just do one-off analysis.
Almost every team is asking for a platform

Base Agent

in
PROPRIETARY + CONFIDENTIAL
22

### Conclusion: Accelerating the Rate of AI Improvement

LinkedIn’s improvement engine diagnoses failures in its user understanding agent, proposes changes, and tests whether they help. The goal is a steeper performance curve as the agent improves faster over time.

[35:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2136s) [Four bullets describe the target agent, the improvement engine, repeated evaluation and validation, and the goal of faster improvement. A formula expresses that goal as increasing the rate of change in agent performance over time.](https://assets.ai.engineer/sites/aiecode2025/sha256/041c094424aad90de37ea6ee3b225688f48b7d8894f00c46a34df23959ee0ae2.webp)

Conclusion: Accelerating the Rate of AI Improvement

Recursive Self-Improvement isn't just about building better AI agents. It's about building systems that make AI agents better, faster.

• Target Agent: The AI system whose performance we're continuously improving.
• Improvement Engine: The feedback-driven system that evaluates, diagnoses failures, and proposes targeted changes.
• Closed-Loop Learning: Repeated cycles of evaluation, reflection, optimization, and validation drive measurable improvements.
• The Bigger Goal: Move beyond improving performance at a single point in time to accelerating the rate of improvement itself.

We're not just optimizing agent performance. We're optimizing how quickly agent performance improves.

RSI Goal: Increase d(Agent Performance)/dt

[36:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2217s) [Four bullets describe the target agent, the improvement engine, closed-loop learning, and the goal of faster improvement. An equation expresses that goal as increasing the rate of change in agent performance over time.](https://assets.ai.engineer/sites/aiecode2025/sha256/8b5853f84c97542ff6a105ad1797048341c79b6d523c10f5b433dbf5efc1a421.webp)

Conclusion: Accelerating the Rate of AI Improvement

Recursive Self-Improvement isn't just about building better AI agents. It's about building systems that make AI agents better, faster.

• Target Agent: The AI system whose performance we're continuously improving.
• Improvement Engine: The feedback-driven system that evaluates, diagnoses failures, and proposes targeted changes.
• Closed-Loop Learning: Repeated cycles of evaluation, reflection, optimization, and validation drive measurable improvements.
• The Bigger Goal: Move beyond improving performance at a single point in time to accelerating the rate of improvement itself.

We're not just optimizing agent performance. We're optimizing how quickly agent performance improves.

RSI Goal: Increase d(Agent Performance)/dt

### Thank you.

As the thank-you slide appears, the speaker closes with a question for recursive self-improvement: how can the team trust that its changes make the system better?

[38:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2306s) [Large white text reads “Thank you.” on a navy background with a blue border and a LinkedIn logo in the lower right.](https://assets.ai.engineer/sites/aiecode2025/sha256/33ff996b207fc4c3d4e89c8cdce38f90ab26c0598c4b2be7d6cb98de1244c9a6.webp)

Thank you.

in

## Full transcript

[0:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1s) Uh hello everyone. Uh so my name is uh

[0:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=4s) Uh hello everyone. Uh so my name is uh

[0:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=4s) Uh hello everyone. Uh so my name is uh Zui and today uh together with uh Saloni

[0:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=7s) Zui and today uh together with uh Saloni

[0:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=7s) Zui and today uh together with uh Saloni so we are going to present the recursive

[0:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=9s) so we are going to present the recursive

[0:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=9s) so we are going to present the recursive self-improvement uh project that we are

[0:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=12s) self-improvement uh project that we are

[0:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=12s) self-improvement uh project that we are doing at linking and for this RSI work

[0:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=15s) doing at linking and for this RSI work

[0:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=15s) doing at linking and for this RSI work we focus on the user understanding and

[0:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=19s) we focus on the user understanding and

[0:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=19s) we focus on the user understanding and yeah super excited to share the uh RSI

[0:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=22s) yeah super excited to share the uh RSI

[0:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=22s) yeah super excited to share the uh RSI work we did at linking. So first uh just

[0:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=25s) work we did at linking. So first uh just

[0:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=25s) work we did at linking. So first uh just a high level or overview on what is a

[0:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=27s) a high level or overview on what is a

[0:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=27s) a high level or overview on what is a recursive self-improvement.

[0:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=30s) recursive self-improvement.

[0:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=30s) recursive self-improvement. So the recursive self-improvement is a

[0:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=31s) So the recursive self-improvement is a

[0:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=32s) So the recursive self-improvement is a build around some uh AI system and uh

[0:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=34s) build around some uh AI system and uh

[0:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=34s) build around some uh AI system and uh this is the purpose of doing this is we

[0:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=37s) this is the purpose of doing this is we

[0:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=37s) this is the purpose of doing this is we want to get a system that

[0:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=39s) want to get a system that

[0:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=39s) want to get a system that self-improvements uh self-improves. So

[0:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=42s) self-improvements uh self-improves. So

[0:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=42s) self-improvements uh self-improves. So it can learns from issues in production

[0:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=44s) it can learns from issues in production

[0:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=44s) it can learns from issues in production trace and that work. So the AI system

[0:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=47s) trace and that work. So the AI system

[0:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=47s) trace and that work. So the AI system can be some uh agent and I linking. So

[0:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=50s) can be some uh agent and I linking. So

[0:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=50s) can be some uh agent and I linking. So one example is uh our career uh agent to

[0:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=54s) one example is uh our career uh agent to

[0:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=54s) one example is uh our career uh agent to help uh our uh linking users members to

[0:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=57s) help uh our uh linking users members to

[0:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=57s) help uh our uh linking users members to achieve their career goals and from the

[1:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=60s) achieve their career goals and from the

[1:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=60s) achieve their career goals and from the agent so uh and there's uh many daily

[1:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=63s) agent so uh and there's uh many daily

[1:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=63s) agent so uh and there's uh many daily development work that we discover the

[1:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=66s) development work that we discover the

[1:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=66s) development work that we discover the issues we want to address and also we

[1:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=69s) issues we want to address and also we

[1:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=69s) issues we want to address and also we will get the production traces and that

[1:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=72s) will get the production traces and that

[1:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=72s) will get the production traces and that we can uh so they like we can log the

[1:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=74s) we can uh so they like we can log the

[1:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=74s) we can uh so they like we can log the issues we found during the production

[1:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=76s) issues we found during the production

[1:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=76s) issues we found during the production traffic and once we have this

[1:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=79s) traffic and once we have this

[1:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=79s) traffic and once we have this informations So we will have some

[1:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=81s) informations So we will have some

[1:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=81s) informations So we will have some evaluation engine for example the ar

[1:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=83s) evaluation engine for example the ar

[1:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=83s) evaluation engine for example the ar judge and to uh check our uh traces and

[1:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=87s) judge and to uh check our uh traces and

[1:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=87s) judge and to uh check our uh traces and to identify the issues and gives

[1:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=89s) to identify the issues and gives

[1:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=89s) to identify the issues and gives feedbacks and based on the feedbacks and

[1:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=92s) feedbacks and based on the feedbacks and

[1:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=92s) feedbacks and based on the feedbacks and we will uh use those feedback to keep uh

[1:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=95s) we will uh use those feedback to keep uh

[1:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=95s) we will uh use those feedback to keep uh continuously uh iterate our system. So

[1:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=98s) continuously uh iterate our system. So

[1:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=98s) continuously uh iterate our system. So we build this kind of like a loop to

[1:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=101s) we build this kind of like a loop to

[1:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=101s) we build this kind of like a loop to keep uh improve our um AI system. And uh

[1:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=105s) keep uh improve our um AI system. And uh

[1:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=105s) keep uh improve our um AI system. And uh one key thing we want to uh call out is

[1:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=107s) one key thing we want to uh call out is

[1:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=107s) one key thing we want to uh call out is uh during this uh recursive self

[1:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=109s) uh during this uh recursive self

[1:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=109s) uh during this uh recursive self improvement process the evaluation is

[1:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=112s) improvement process the evaluation is

[1:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=112s) improvement process the evaluation is very critical and it must be objective

[1:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=115s) very critical and it must be objective

[1:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=115s) very critical and it must be objective and well aligned with the user feedback

[1:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=117s) and well aligned with the user feedback

[1:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=117s) and well aligned with the user feedback because this is a key to uh like

[1:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=119s) because this is a key to uh like

[1:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=119s) because this is a key to uh like identify where the bottleneck is and

[2:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=121s) identify where the bottleneck is and

[2:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=121s) identify where the bottleneck is and guide the next step for the uh system AI

[2:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=124s) guide the next step for the uh system AI

[2:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=124s) guide the next step for the uh system AI system improvement.

[2:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=126s) system improvement.

[2:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=126s) system improvement. And uh next we want to uh uh like

[2:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=129s) And uh next we want to uh uh like

[2:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=129s) And uh next we want to uh uh like describe like show uh what kind of like

[2:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=132s) describe like show uh what kind of like

[2:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=132s) describe like show uh what kind of like uh which component of the AI system we

[2:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=134s) uh which component of the AI system we

[2:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=134s) uh which component of the AI system we can uh optimize. Uh so here take the

[2:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=138s) can uh optimize. Uh so here take the

[2:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=138s) can uh optimize. Uh so here take the harness uh as as an example or like as

[2:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=141s) harness uh as as an example or like as

[2:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=141s) harness uh as as an example or like as orchestrator. So there are different

[2:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=143s) orchestrator. So there are different

[2:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=143s) orchestrator. So there are different components we can uh use we can uh

[2:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=146s) components we can uh use we can uh

[2:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=146s) components we can uh use we can uh optimize by using the RSI. So for

[2:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=149s) optimize by using the RSI. So for

[2:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=149s) optimize by using the RSI. So for example the first is a planning and

[2:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=151s) example the first is a planning and

[2:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=151s) example the first is a planning and logics which is used to orchestrator

[2:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=154s) logics which is used to orchestrator

[2:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=154s) logics which is used to orchestrator orchestrates execution and make a

[2:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=156s) orchestrates execution and make a

[2:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=156s) orchestrates execution and make a decision uh past and the second is a

[2:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=158s) decision uh past and the second is a

[2:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=158s) decision uh past and the second is a context and the memory where we preserve

[2:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=161s) context and the memory where we preserve

[2:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=161s) context and the memory where we preserve and manage the session state and the

[2:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=164s) and manage the session state and the

[2:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=164s) and manage the session state and the third one is a two interface that

[2:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=166s) third one is a two interface that

[2:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=166s) third one is a two interface that interacts with external and internal

[2:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=168s) interacts with external and internal

[2:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=168s) interacts with external and internal APIs and data sets. The last uh last but

[2:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=171s) APIs and data sets. The last uh last but

[2:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=171s) APIs and data sets. The last uh last but not least is artifact storage uh that

[2:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=174s) not least is artifact storage uh that

[2:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=174s) not least is artifact storage uh that where we stores outputs and the

[2:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=176s) where we stores outputs and the

[2:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=176s) where we stores outputs and the generation logs and so they like uh and

[2:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=179s) generation logs and so they like uh and

[2:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=179s) generation logs and so they like uh and we are on top of those we definitely

[3:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=181s) we are on top of those we definitely

[3:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=181s) we are on top of those we definitely have some like evaluation uh logic and

[3:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=184s) have some like evaluation uh logic and

[3:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=184s) have some like evaluation uh logic and uh so that to help us to evaluate uh

[3:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=186s) uh so that to help us to evaluate uh

[3:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=186s) uh so that to help us to evaluate uh each um component and help us to keep uh

[3:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=189s) each um component and help us to keep uh

[3:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=189s) each um component and help us to keep uh improve the uh AI system and we want to

[3:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=193s) improve the uh AI system and we want to

[3:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=193s) improve the uh AI system and we want to uh like uh uh and another thing to uh

[3:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=196s) uh like uh uh and another thing to uh

[3:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=196s) uh like uh uh and another thing to uh keep in mind that all those uh like

[3:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=198s) keep in mind that all those uh like

[3:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=198s) keep in mind that all those uh like harness is a powered by the base uh iron

[3:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=202s) harness is a powered by the base uh iron

[3:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=202s) harness is a powered by the base uh iron model but for the RSI uh so it's

[3:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=206s) model but for the RSI uh so it's

[3:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=206s) model but for the RSI uh so it's currently the optimize of this base

[3:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=209s) currently the optimize of this base

[3:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=209s) currently the optimize of this base model uh itself is not in uh scope but

[3:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=212s) model uh itself is not in uh scope but

[3:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=212s) model uh itself is not in uh scope but we focus on the uh harness optimization

[3:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=216s) we focus on the uh harness optimization

[3:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=216s) we focus on the uh harness optimization uh first

[3:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=218s) uh first

[3:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=218s) uh first and uh here we want to uh like show uh

[3:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=221s) and uh here we want to uh like show uh

[3:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=221s) and uh here we want to uh like show uh like this is the ladders that we can

[3:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=223s) like this is the ladders that we can

[3:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=223s) like this is the ladders that we can climb to optimize the uh AI system by

[3:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=226s) climb to optimize the uh AI system by

[3:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=226s) climb to optimize the uh AI system by using the uh RSI. So uh at the beginning

[3:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=230s) using the uh RSI. So uh at the beginning

[3:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=230s) using the uh RSI. So uh at the beginning so typically so this is a start from

[3:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=232s) so typically so this is a start from

[3:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=232s) so typically so this is a start from simple uh process at the beginning we

[3:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=234s) simple uh process at the beginning we

[3:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=234s) simple uh process at the beginning we can focus on the prompts uh updates

[3:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=237s) can focus on the prompts uh updates

[3:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=237s) can focus on the prompts uh updates because this is a most uh this is the

[3:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=239s) because this is a most uh this is the

[3:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=239s) because this is a most uh this is the easiest and the most straightforward way

[4:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=242s) easiest and the most straightforward way

[4:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=242s) easiest and the most straightforward way to uh keep iterate our AI system and

[4:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=246s) to uh keep iterate our AI system and

[4:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=246s) to uh keep iterate our AI system and after that uh we can uh continue to

[4:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=248s) after that uh we can uh continue to

[4:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=248s) after that uh we can uh continue to optimize the structured context of the

[4:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=251s) optimize the structured context of the

[4:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=251s) optimize the structured context of the AI system and uh uh the and the for the

[4:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=255s) AI system and uh uh the and the for the

[4:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=255s) AI system and uh uh the and the for the this is the L1 uh ladder and for the L2

[4:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=258s) this is the L1 uh ladder and for the L2

[4:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=258s) this is the L1 uh ladder and for the L2 tool ladder and which is after the

[4:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=260s) tool ladder and which is after the

[4:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=260s) tool ladder and which is after the structure context optimization. We can

[4:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=263s) structure context optimization. We can

[4:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=263s) structure context optimization. We can uh expand our scope to design the auto

[4:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=266s) uh expand our scope to design the auto

[4:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=266s) uh expand our scope to design the auto automatically design the entire uh

[4:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=268s) automatically design the entire uh

[4:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=268s) automatically design the entire uh workflow and uh uh so this is kind of

[4:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=271s) workflow and uh uh so this is kind of

[4:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=271s) workflow and uh uh so this is kind of like uh like a meta information uh

[4:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=274s) like uh like a meta information uh

[4:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=274s) like uh like a meta information uh design and uh after that and we can

[4:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=277s) design and uh after that and we can

[4:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=277s) design and uh after that and we can automize the harness code uh itself and

[4:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=280s) automize the harness code uh itself and

[4:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=280s) automize the harness code uh itself and as at the end we can uh go very

[4:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=283s) as at the end we can uh go very

[4:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=283s) as at the end we can uh go very compactly to optimize the optimizer code

[4:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=286s) compactly to optimize the optimizer code

[4:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=286s) compactly to optimize the optimizer code uh itself. So this is kind of the

[4:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=288s) uh itself. So this is kind of the

[4:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=288s) uh itself. So this is kind of the different optimization ladders uh with

[4:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=291s) different optimization ladders uh with

[4:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=291s) different optimization ladders uh with increasing complexity and uh uh for our

[4:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=295s) increasing complexity and uh uh for our

[4:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=295s) increasing complexity and uh uh for our project. So currently uh our active

[4:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=297s) project. So currently uh our active

[4:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=297s) project. So currently uh our active focus is a prompt uh optimization but uh

[5:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=301s) focus is a prompt uh optimization but uh

[5:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=301s) focus is a prompt uh optimization but uh uh definitely like moving forward we are

[5:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=303s) uh definitely like moving forward we are

[5:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=303s) uh definitely like moving forward we are extend our scope to uh more and more

[5:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=306s) extend our scope to uh more and more

[5:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=306s) extend our scope to uh more and more complicated uh optimization uh in future

[5:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=309s) complicated uh optimization uh in future

[5:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=309s) complicated uh optimization uh in future and uh here we want to show our uh uh

[5:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=313s) and uh here we want to show our uh uh

[5:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=313s) and uh here we want to show our uh uh RSI uh strategy. So we break down our uh

[5:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=317s) RSI uh strategy. So we break down our uh

[5:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=317s) RSI uh strategy. So we break down our uh RSI uh uh process uh into seven uh

[5:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=320s) RSI uh uh process uh into seven uh

[5:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=320s) RSI uh uh process uh into seven uh steps. So the first step is to uh run

[5:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=323s) steps. So the first step is to uh run

[5:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=323s) steps. So the first step is to uh run the agent and uh generate some uh

[5:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=326s) the agent and uh generate some uh

[5:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=326s) the agent and uh generate some uh samples. So basically like for our

[5:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=329s) samples. So basically like for our

[5:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=329s) samples. So basically like for our career agent uh that we are optimizing

[5:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=331s) career agent uh that we are optimizing

[5:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=331s) career agent uh that we are optimizing for. So given a query we'll generate

[5:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=334s) for. So given a query we'll generate

[5:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=334s) for. So given a query we'll generate some uh output and we will lock those uh

[5:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=337s) some uh output and we will lock those uh

[5:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=337s) some uh output and we will lock those uh process in the uh traces for example the

[5:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=339s) process in the uh traces for example the

[5:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=340s) process in the uh traces for example the last miss traces and uh once so this is

[5:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=343s) last miss traces and uh once so this is

[5:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=343s) last miss traces and uh once so this is basically just the uh like the uh kind

[5:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=345s) basically just the uh like the uh kind

[5:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=345s) basically just the uh like the uh kind of the agent uh trajectories and the

[5:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=348s) of the agent uh trajectories and the

[5:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=348s) of the agent uh trajectories and the traces that we can uh uh use to uh

[5:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=351s) traces that we can uh uh use to uh

[5:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=351s) traces that we can uh uh use to uh optimize and once we got those

[5:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=353s) optimize and once we got those

[5:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=353s) optimize and once we got those information the next step step two is we

[5:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=356s) information the next step step two is we

[5:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=356s) information the next step step two is we uh runs through the evation uh engine

[5:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=359s) uh runs through the evation uh engine

[5:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=359s) uh runs through the evation uh engine for example the judge and to uh get some

[6:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=362s) for example the judge and to uh get some

[6:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=362s) for example the judge and to uh get some judge uh output to tell us uh where we

[6:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=366s) judge uh output to tell us uh where we

[6:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=366s) judge uh output to tell us uh where we are uh good at and we are where we are

[6:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=368s) are uh good at and we are where we are

[6:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=368s) are uh good at and we are where we are not good at. So to get the uh uh future

[6:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=371s) not good at. So to get the uh uh future

[6:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=371s) not good at. So to get the uh uh future iteration and then uh once we get the uh

[6:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=375s) iteration and then uh once we get the uh

[6:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=375s) iteration and then uh once we get the uh um judge output then we go uh we we are

[6:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=379s) um judge output then we go uh we we are

[6:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=379s) um judge output then we go uh we we are going to the RSI components and uh

[6:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=382s) going to the RSI components and uh

[6:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=382s) going to the RSI components and uh starting from the uh step three. So this

[6:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=384s) starting from the uh step three. So this

[6:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=384s) starting from the uh step three. So this is a kind of the RSI in the inside the

[6:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=386s) is a kind of the RSI in the inside the

[6:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=386s) is a kind of the RSI in the inside the RSI loop and the first step is we will

[6:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=390s) RSI loop and the first step is we will

[6:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=390s) RSI loop and the first step is we will uh we will do the agent reflection uh

[6:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=393s) uh we will do the agent reflection uh

[6:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=393s) uh we will do the agent reflection uh sometime we also call it a root causing

[6:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=395s) sometime we also call it a root causing

[6:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=395s) sometime we also call it a root causing so they like based on the judge uh

[6:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=397s) so they like based on the judge uh

[6:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=397s) so they like based on the judge uh output we want to understand what is a

[6:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=399s) output we want to understand what is a

[6:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=399s) output we want to understand what is a root cause uh that we don't have a good

[6:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=403s) root cause uh that we don't have a good

[6:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=403s) root cause uh that we don't have a good uh output for example is this is a agent

[6:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=405s) uh output for example is this is a agent

[6:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=405s) uh output for example is this is a agent issue a model issue u prompt issue or

[6:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=409s) issue a model issue u prompt issue or

[6:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=409s) issue a model issue u prompt issue or like a two uh two two issue or some

[6:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=411s) like a two uh two two issue or some

[6:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=411s) like a two uh two two issue or some other issues or sometimes it's Even it

[6:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=414s) other issues or sometimes it's Even it

[6:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=414s) other issues or sometimes it's Even it can even be a judge issue or rubric

[6:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=416s) can even be a judge issue or rubric

[6:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=416s) can even be a judge issue or rubric issue. And uh once we got the root cause

[6:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=419s) issue. And uh once we got the root cause

[6:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=419s) issue. And uh once we got the root cause and uh uh we will uh the next step is to

[7:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=422s) and uh uh we will uh the next step is to

[7:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=422s) and uh uh we will uh the next step is to uh based on the root cause uh we

[7:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=425s) uh based on the root cause uh we

[7:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=425s) uh based on the root cause uh we identified we want uh we can generate

[7:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=427s) identified we want uh we can generate

[7:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=427s) identified we want uh we can generate some uh addits proposal. So uh currently

[7:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=431s) some uh addits proposal. So uh currently

[7:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=431s) some uh addits proposal. So uh currently the addits proposal are just uh prompt

[7:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=434s) the addits proposal are just uh prompt

[7:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=434s) the addits proposal are just uh prompt but as we are climbing the climbing the

[7:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=436s) but as we are climbing the climbing the

[7:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=436s) but as we are climbing the climbing the ladder uh for the SI loop. So in future

[7:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=439s) ladder uh for the SI loop. So in future

[7:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=439s) ladder uh for the SI loop. So in future this uh additis can also be some like a

[7:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=442s) this uh additis can also be some like a

[7:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=442s) this uh additis can also be some like a new design of the workflow the harness

[7:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=444s) new design of the workflow the harness

[7:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=444s) new design of the workflow the harness code or even the optimizer optimizer

[7:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=447s) code or even the optimizer optimizer

[7:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=447s) code or even the optimizer optimizer code and uh uh so yeah that's uh step

[7:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=451s) code and uh uh so yeah that's uh step

[7:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=451s) code and uh uh so yeah that's uh step four to generate the edits proposal and

[7:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=453s) four to generate the edits proposal and

[7:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=453s) four to generate the edits proposal and the next step is to uh validate our

[7:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=457s) the next step is to uh validate our

[7:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=457s) the next step is to uh validate our edits because uh some edits can be uh

[7:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=460s) edits because uh some edits can be uh

[7:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=460s) edits because uh some edits can be uh good some can be bad uh so we will use

[7:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=462s) good some can be bad uh so we will use

[7:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=462s) good some can be bad uh so we will use the same uh evaluation engine for

[7:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=464s) the same uh evaluation engine for

[7:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=464s) the same uh evaluation engine for example the same judge with the same

[7:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=467s) example the same judge with the same

[7:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=467s) example the same judge with the same rubrics to uh to uh evaluate uh our

[7:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=470s) rubrics to uh to uh evaluate uh our

[7:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=470s) rubrics to uh to uh evaluate uh our addit validation and uh uh once we found

[7:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=474s) addit validation and uh uh once we found

[7:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=474s) addit validation and uh uh once we found some like a good candidates and we will

[7:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=476s) some like a good candidates and we will

[7:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=476s) some like a good candidates and we will uh naturally the next step is to uh

[7:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=479s) uh naturally the next step is to uh

[7:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=479s) uh naturally the next step is to uh evolve our uh prompt to adapt the uh the

[8:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=483s) evolve our uh prompt to adapt the uh the

[8:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=483s) evolve our uh prompt to adapt the uh the uh the proposed edits uh we have and uh

[8:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=487s) uh the proposed edits uh we have and uh

[8:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=487s) uh the proposed edits uh we have and uh uh once we uh once the prompt is uh

[8:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=489s) uh once we uh once the prompt is uh

[8:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=489s) uh once we uh once the prompt is uh updated. So the very last step which is

[8:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=492s) updated. So the very last step which is

[8:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=492s) updated. So the very last step which is a step seven here is we run the

[8:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=494s) a step seven here is we run the

[8:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=494s) a step seven here is we run the evaluation uh again on a hold up side

[8:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=497s) evaluation uh again on a hold up side

[8:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=497s) evaluation uh again on a hold up side and to confirm uh our uh this kind of

[8:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=500s) and to confirm uh our uh this kind of

[8:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=500s) and to confirm uh our uh this kind of the uh new system is indeed performs

[8:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=503s) the uh new system is indeed performs

[8:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=503s) the uh new system is indeed performs better uh than the old system after the

[8:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=506s) better uh than the old system after the

[8:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=506s) better uh than the old system after the SI optimization and uh uh this will and

[8:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=510s) SI optimization and uh uh this will and

[8:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=510s) SI optimization and uh uh this will and also like uh after this hold out

[8:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=513s) also like uh after this hold out

[8:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=513s) also like uh after this hold out validation uh like we will make the uh

[8:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=516s) validation uh like we will make the uh

[8:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=516s) validation uh like we will make the uh prompt updates and this will become our

[8:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=519s) prompt updates and this will become our

[8:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=519s) prompt updates and this will become our uh new baseline and uh and we are uh in

[8:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=523s) uh new baseline and uh and we are uh in

[8:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=523s) uh new baseline and uh and we are uh in future we will like continue to do this

[8:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=525s) future we will like continue to do this

[8:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=526s) future we will like continue to do this uh RSI uh iteration to keep improving uh

[8:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=529s) uh RSI uh iteration to keep improving uh

[8:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=529s) uh RSI uh iteration to keep improving uh our system uh with some uh cadence.

[8:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=533s) our system uh with some uh cadence.

[8:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=533s) our system uh with some uh cadence. Uh okay. Yeah. Next uh like uh we are we

[8:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=535s) Uh okay. Yeah. Next uh like uh we are we

[8:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=535s) Uh okay. Yeah. Next uh like uh we are we can uh like dig deeper into uh each step

[8:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=539s) can uh like dig deeper into uh each step

[8:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=539s) can uh like dig deeper into uh each step so that we can get uh so so to help you

[9:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=541s) so that we can get uh so so to help you

[9:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=542s) so that we can get uh so so to help you to understand more details on what we

[9:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=543s) to understand more details on what we

[9:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=544s) to understand more details on what we are uh doing here. for example, what is

[9:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=545s) are uh doing here. for example, what is

[9:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=545s) are uh doing here. for example, what is the agent we are optimizing and what is

[9:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=548s) the agent we are optimizing and what is

[9:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=548s) the agent we are optimizing and what is like a steps uh like a detail steps uh

[9:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=551s) like a steps uh like a detail steps uh

[9:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=551s) like a steps uh like a detail steps uh in each uh this high level steps to uh

[9:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=554s) in each uh this high level steps to uh

[9:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=554s) in each uh this high level steps to uh help us uh to help you to understand

[9:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=556s) help us uh to help you to understand

[9:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=556s) help us uh to help you to understand more of of our uh SI work here. So the

[9:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=560s) more of of our uh SI work here. So the

[9:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=560s) more of of our uh SI work here. So the first step I I want to show uh so this

[9:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=563s) first step I I want to show uh so this

[9:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=563s) first step I I want to show uh so this is the first uh the step we generated

[9:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=565s) is the first uh the step we generated

[9:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=565s) is the first uh the step we generated the output and the generated trace and

[9:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=567s) the output and the generated trace and

[9:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=567s) the output and the generated trace and uh so first we want to I want to uh help

[9:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=570s) uh so first we want to I want to uh help

[9:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=570s) uh so first we want to I want to uh help you to understand the agent uh we are

[9:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=572s) you to understand the agent uh we are

[9:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=572s) you to understand the agent uh we are optimizing here. So this is a uh user uh

[9:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=575s) optimizing here. So this is a uh user uh

[9:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=575s) optimizing here. So this is a uh user uh understanding uh agent uh at linking. So

[9:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=579s) understanding uh agent uh at linking. So

[9:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=579s) understanding uh agent uh at linking. So uh so like you can see we uh in this

[9:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=581s) uh so like you can see we uh in this

[9:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=581s) uh so like you can see we uh in this slide we are showing the input and the

[9:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=583s) slide we are showing the input and the

[9:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=583s) slide we are showing the input and the output of this uh agent. So the input is

[9:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=586s) output of this uh agent. So the input is

[9:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=586s) output of this uh agent. So the input is just a simply member ID also it's just a

[9:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=590s) just a simply member ID also it's just a

[9:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=590s) just a simply member ID also it's just a user ID and uh uh by uh given user ID we

[9:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=594s) user ID and uh uh by uh given user ID we

[9:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=594s) user ID and uh uh by uh given user ID we can fetch uh some information uh of this

[9:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=597s) can fetch uh some information uh of this

[9:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=597s) can fetch uh some information uh of this user for example this user search

[9:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=599s) user for example this user search

[9:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=599s) user for example this user search history uh profile data uh and also some

[10:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=602s) history uh profile data uh and also some

[10:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=602s) history uh profile data uh and also some like a location data and the preference

[10:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=604s) like a location data and the preference

[10:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=604s) like a location data and the preference data those kind of things and we have

[10:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=607s) data those kind of things and we have

[10:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=607s) data those kind of things and we have this user understanding agent to uh like

[10:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=609s) this user understanding agent to uh like

[10:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=609s) this user understanding agent to uh like take those users uh history or behavior

[10:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=612s) take those users uh history or behavior

[10:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=612s) take those users uh history or behavior data profile data as a input and uh do

[10:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=615s) data profile data as a input and uh do

[10:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=615s) data profile data as a input and uh do the uh do a lot like this kind of um uh

[10:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=618s) the uh do a lot like this kind of um uh

[10:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=618s) the uh do a lot like this kind of um uh agentic uh synthesize steps and we

[10:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=621s) agentic uh synthesize steps and we

[10:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=621s) agentic uh synthesize steps and we generate the uh output. So for this

[10:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=624s) generate the uh output. So for this

[10:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=624s) generate the uh output. So for this output we call it is a uh memor sending

[10:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=626s) output we call it is a uh memor sending

[10:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=626s) output we call it is a uh memor sending or the user understanding uh output. So

[10:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=629s) or the user understanding uh output. So

[10:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=629s) or the user understanding uh output. So here is a high level uh structure of the

[10:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=632s) here is a high level uh structure of the

[10:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=632s) here is a high level uh structure of the uh this output is so given a member. So

[10:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=635s) uh this output is so given a member. So

[10:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=635s) uh this output is so given a member. So we will generate some uh fields to

[10:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=638s) we will generate some uh fields to

[10:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=638s) we will generate some uh fields to describe uh this uh member based on our

[10:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=641s) describe uh this uh member based on our

[10:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=641s) describe uh this uh member based on our agent understanding. For example, what

[10:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=643s) agent understanding. For example, what

[10:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=643s) agent understanding. For example, what is the preferred uh job title uh of this

[10:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=645s) is the preferred uh job title uh of this

[10:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=646s) is the preferred uh job title uh of this member? This is just one uh field of

[10:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=648s) member? This is just one uh field of

[10:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=648s) member? This is just one uh field of this uh this user understanding uh

[10:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=650s) this uh this user understanding uh

[10:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=650s) this uh this user understanding uh output. So for this one, this is just a

[10:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=653s) output. So for this one, this is just a

[10:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=654s) output. So for this one, this is just a example of the preferred job titles. So

[10:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=657s) example of the preferred job titles. So

[10:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=657s) example of the preferred job titles. So it can be some like a network manager,

[10:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=659s) it can be some like a network manager,

[10:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=659s) it can be some like a network manager, account manager those kind of things and

[11:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=661s) account manager those kind of things and

[11:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=661s) account manager those kind of things and we'll give a confidence on like how

[11:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=663s) we'll give a confidence on like how

[11:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=663s) we'll give a confidence on like how confidence this uh agent think uh like

[11:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=667s) confidence this uh agent think uh like

[11:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=667s) confidence this uh agent think uh like this is indeed the preferred job titles

[11:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=670s) this is indeed the preferred job titles

[11:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=670s) this is indeed the preferred job titles for this user and uh also uh we'll get

[11:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=673s) for this user and uh also uh we'll get

[11:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=673s) for this user and uh also uh we'll get uh generate some uh citation data. So

[11:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=676s) uh generate some uh citation data. So

[11:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=676s) uh generate some uh citation data. So they like this is the uh like why the

[11:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=678s) they like this is the uh like why the

[11:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=678s) they like this is the uh like why the this is where we can tell why the agent

[11:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=681s) this is where we can tell why the agent

[11:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=681s) this is where we can tell why the agent think those uh titles is a preferred job

[11:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=684s) think those uh titles is a preferred job

[11:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=684s) think those uh titles is a preferred job title for this particular uh member. Uh

[11:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=687s) title for this particular uh member. Uh

[11:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=687s) title for this particular uh member. Uh for example it could be uh the reason

[11:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=689s) for example it could be uh the reason

[11:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=689s) for example it could be uh the reason could be like uh in the users uh save

[11:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=692s) could be like uh in the users uh save

[11:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=692s) could be like uh in the users uh save the search history like those job titles

[11:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=695s) the search history like those job titles

[11:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=695s) the search history like those job titles uh appears very frequently. So the agent

[11:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=698s) uh appears very frequently. So the agent

[11:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=698s) uh appears very frequently. So the agent think oh this is a strong evidence uh

[11:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=700s) think oh this is a strong evidence uh

[11:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=700s) think oh this is a strong evidence uh that to support that the preferred job

[11:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=702s) that to support that the preferred job

[11:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=702s) that to support that the preferred job title is a network uh manager uh for

[11:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=704s) title is a network uh manager uh for

[11:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=704s) title is a network uh manager uh for this uh for this uh user and uh uh as a

[11:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=709s) this uh for this uh user and uh uh as a

[11:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=709s) this uh for this uh user and uh uh as a uh and uh finally this agent can also

[11:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=712s) uh and uh finally this agent can also

[11:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=712s) uh and uh finally this agent can also output the reasoning on like why like

[11:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=715s) output the reasoning on like why like

[11:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=715s) output the reasoning on like why like just a reasoning like why the agent

[11:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=717s) just a reasoning like why the agent

[11:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=717s) just a reasoning like why the agent think those are preferred job titles

[11:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=719s) think those are preferred job titles

[11:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=719s) think those are preferred job titles like for example the state preference

[12:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=721s) like for example the state preference

[12:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=721s) like for example the state preference anchors the primary title save search

[12:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=723s) anchors the primary title save search

[12:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=723s) anchors the primary title save search agent uh adjacent one those kind of

[12:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=727s) agent uh adjacent one those kind of

[12:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=727s) agent uh adjacent one those kind of So this is a kind of uh uh one example

[12:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=729s) So this is a kind of uh uh one example

[12:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=730s) So this is a kind of uh uh one example of the user understanding agent that we

[12:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=733s) of the user understanding agent that we

[12:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=733s) of the user understanding agent that we are optimizing.

[12:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=735s) are optimizing.

[12:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=735s) are optimizing. So the step two is once we uh got this

[12:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=738s) So the step two is once we uh got this

[12:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=738s) So the step two is once we uh got this uh agents trace and we'll run the judge

[12:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=742s) uh agents trace and we'll run the judge

[12:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=742s) uh agents trace and we'll run the judge and to uh get a quality readings of this

[12:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=746s) and to uh get a quality readings of this

[12:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=746s) and to uh get a quality readings of this uh of the agent uh output and uh for

[12:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=749s) uh of the agent uh output and uh for

[12:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=749s) uh of the agent uh output and uh for this particular user agent uh the score

[12:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=751s) this particular user agent uh the score

[12:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=752s) this particular user agent uh the score is over uh 15 uh task of the dimensions

[12:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=755s) is over uh 15 uh task of the dimensions

[12:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=755s) is over uh 15 uh task of the dimensions by judge where are some um product uh

[12:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=758s) by judge where are some um product uh

[12:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=758s) by judge where are some um product uh defender uh rubrics. So here is one

[12:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=761s) defender uh rubrics. So here is one

[12:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=761s) defender uh rubrics. So here is one example of the uh judge uh output. So

[12:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=764s) example of the uh judge uh output. So

[12:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=764s) example of the uh judge uh output. So you can say for particular um member the

[12:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=767s) you can say for particular um member the

[12:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=767s) you can say for particular um member the user and we will generate a judge uh

[12:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=770s) user and we will generate a judge uh

[12:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=770s) user and we will generate a judge uh score which is 2.31 and uh we will show

[12:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=773s) score which is 2.31 and uh we will show

[12:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=773s) score which is 2.31 and uh we will show and it will also tell like this score is

[12:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=775s) and it will also tell like this score is

[12:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=776s) and it will also tell like this score is for uh which um and [clears throat]

[12:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=778s) for uh which um and [clears throat]

[12:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=778s) for uh which um and [clears throat] it'll show the breakdowns uh of the you

[13:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=781s) it'll show the breakdowns uh of the you

[13:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=781s) it'll show the breakdowns uh of the you know the each dimensions score for

[13:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=783s) know the each dimensions score for

[13:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=783s) know the each dimensions score for example the go summary dimensions and we

[13:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=786s) example the go summary dimensions and we

[13:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=786s) example the go summary dimensions and we also have some other dimension for the

[13:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=788s) also have some other dimension for the

[13:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=788s) also have some other dimension for the the second one is a prefer job title

[13:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=790s) the second one is a prefer job title

[13:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=790s) the second one is a prefer job title dimensions and for each this kind of

[13:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=792s) dimensions and for each this kind of

[13:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=792s) dimensions and for each this kind of dimensions we give a detailed uh score

[13:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=795s) dimensions we give a detailed uh score

[13:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=795s) dimensions we give a detailed uh score uh for each dimension and also the

[13:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=797s) uh for each dimension and also the

[13:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=797s) uh for each dimension and also the notes. So the notes is uh just the

[13:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=800s) notes. So the notes is uh just the

[13:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=800s) notes. So the notes is uh just the rationale also the uh vertx like why uh

[13:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=804s) rationale also the uh vertx like why uh

[13:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=804s) rationale also the uh vertx like why uh we give this score uh to this uh to this

[13:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=808s) we give this score uh to this uh to this

[13:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=808s) we give this score uh to this uh to this particular uh dimension and for this

[13:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=810s) particular uh dimension and for this

[13:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=810s) particular uh dimension and for this particular uh judge is the agent. So the

[13:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=814s) particular uh judge is the agent. So the

[13:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=814s) particular uh judge is the agent. So the uh judge score is one to three uh scale.

[13:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=817s) uh judge score is one to three uh scale.

[13:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=817s) uh judge score is one to three uh scale. So the score one means uh this is not a

[13:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=820s) So the score one means uh this is not a

[13:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=820s) So the score one means uh this is not a high uh good is not a good yeah the

[13:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=825s) high uh good is not a good yeah the

[13:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=825s) high uh good is not a good yeah the quality is not good and the two is a

[13:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=827s) quality is not good and the two is a

[13:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=827s) quality is not good and the two is a fair uh quality and the three means this

[13:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=830s) fair uh quality and the three means this

[13:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=830s) fair uh quality and the three means this is a very good quality. Yeah. And uh uh

[13:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=833s) is a very good quality. Yeah. And uh uh

[13:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=834s) is a very good quality. Yeah. And uh uh so yeah that's our uh um judge and uh

[13:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=837s) so yeah that's our uh um judge and uh

[13:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=837s) so yeah that's our uh um judge and uh after we got the judge score uh for each

[14:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=841s) after we got the judge score uh for each

[14:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=841s) after we got the judge score uh for each dimension of our user understanding

[14:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=843s) dimension of our user understanding

[14:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=843s) dimension of our user understanding agent and the next step is we will run

[14:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=846s) agent and the next step is we will run

[14:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=846s) agent and the next step is we will run this uh agent reflection process. So as

[14:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=849s) this uh agent reflection process. So as

[14:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=849s) this uh agent reflection process. So as we mentioned earlier so this is a kind

[14:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=851s) we mentioned earlier so this is a kind

[14:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=851s) we mentioned earlier so this is a kind of uh uh process for us to understand

[14:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=855s) of uh uh process for us to understand

[14:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=855s) of uh uh process for us to understand okay if this is a low score then what is

[14:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=857s) okay if this is a low score then what is

[14:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=857s) okay if this is a low score then what is the root cause uh lead to the uh this

[14:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=861s) the root cause uh lead to the uh this

[14:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=861s) the root cause uh lead to the uh this low score. So uh this is a so for this

[14:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=864s) low score. So uh this is a so for this

[14:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=864s) low score. So uh this is a so for this uh step three so actually uh internally

[14:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=867s) uh step three so actually uh internally

[14:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=867s) uh step three so actually uh internally there are four uh like we can call it a

[14:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=870s) there are four uh like we can call it a

[14:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=870s) there are four uh like we can call it a mini steps for the agent reflection. So

[14:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=873s) mini steps for the agent reflection. So

[14:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=873s) mini steps for the agent reflection. So the first step is a clustering. So we

[14:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=876s) the first step is a clustering. So we

[14:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=876s) the first step is a clustering. So we are group the failing uh rows because

[14:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=879s) are group the failing uh rows because

[14:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=879s) are group the failing uh rows because like in a production trace so we are we

[14:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=881s) like in a production trace so we are we

[14:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=881s) like in a production trace so we are we will have a lot of like uh examples a

[14:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=884s) will have a lot of like uh examples a

[14:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=884s) will have a lot of like uh examples a lot of users and we will group uh the

[14:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=887s) lot of users and we will group uh the

[14:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=888s) lot of users and we will group uh the those users also the traces by the

[14:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=890s) those users also the traces by the

[14:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=890s) those users also the traces by the failing uh dimension. So for this uh

[14:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=894s) failing uh dimension. So for this uh

[14:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=894s) failing uh dimension. So for this uh step so this is just a pure clustering

[14:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=896s) step so this is just a pure clustering

[14:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=896s) step so this is just a pure clustering uh without uh any um cost and uh once we

[15:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=900s) uh without uh any um cost and uh once we

[15:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=900s) uh without uh any um cost and uh once we get the clustering and uh uh so the next

[15:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=903s) get the clustering and uh uh so the next

[15:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=903s) get the clustering and uh uh so the next step is we call it L2 uh judge so b

[15:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=907s) step is we call it L2 uh judge so b

[15:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=907s) step is we call it L2 uh judge so b within each uh class. So we are run the

[15:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=911s) within each uh class. So we are run the

[15:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=911s) within each uh class. So we are run the um uh cause to diagnose uh each row and

[15:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=916s) um uh cause to diagnose uh each row and

[15:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=916s) um uh cause to diagnose uh each row and uh and then for each row we are assign a

[15:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=919s) uh and then for each row we are assign a

[15:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=919s) uh and then for each row we are assign a root cause and plus tag. So b like this

[15:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=922s) root cause and plus tag. So b like this

[15:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=922s) root cause and plus tag. So b like this is a u uh place that we uh can like rely

[15:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=927s) is a u uh place that we uh can like rely

[15:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=927s) is a u uh place that we uh can like rely on the to uh let us know uh uh if we

[15:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=931s) on the to uh let us know uh uh if we

[15:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=931s) on the to uh let us know uh uh if we think uh this is a for example this

[15:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=934s) think uh this is a for example this

[15:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=934s) think uh this is a for example this dimension has a low score is that due to

[15:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=936s) dimension has a low score is that due to

[15:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=936s) dimension has a low score is that due to the prompt issue uh agent issue harness

[15:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=938s) the prompt issue uh agent issue harness

[15:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=938s) the prompt issue uh agent issue harness issue code issue or like even the judge

[15:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=941s) issue code issue or like even the judge

[15:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=941s) issue code issue or like even the judge issue or rubric issue. So they will be

[15:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=944s) issue or rubric issue. So they will be

[15:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=944s) issue or rubric issue. So they will be covered by the root cause and tag uh

[15:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=946s) covered by the root cause and tag uh

[15:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=946s) covered by the root cause and tag uh here. So uh the then once we get the

[15:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=951s) here. So uh the then once we get the

[15:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=951s) here. So uh the then once we get the perole level root cause and the tag so

[15:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=953s) perole level root cause and the tag so

[15:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=953s) perole level root cause and the tag so the next step is to merge the uh issues

[15:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=956s) the next step is to merge the uh issues

[15:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=956s) the next step is to merge the uh issues because uh in the second step uh so we

[16:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=960s) because uh in the second step uh so we

[16:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=960s) because uh in the second step uh so we it's just a we is based right so so the

[16:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=965s) it's just a we is based right so so the

[16:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=965s) it's just a we is based right so so the root cost is more like the free tax but

[16:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=968s) root cost is more like the free tax but

[16:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=968s) root cost is more like the free tax but uh uh some root cause uh even the tax is

[16:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=972s) uh uh some root cause uh even the tax is

[16:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=972s) uh uh some root cause uh even the tax is not um uh exactly the same but they may

[16:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=976s) not um uh exactly the same but they may

[16:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=976s) not um uh exactly the same but they may just describe very similar thing or uh

[16:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=979s) just describe very similar thing or uh

[16:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=979s) just describe very similar thing or uh so that's why like we want to do this

[16:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=981s) so that's why like we want to do this

[16:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=981s) so that's why like we want to do this kind of uh dduping so we merge the

[16:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=984s) kind of uh dduping so we merge the

[16:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=984s) kind of uh dduping so we merge the issues if we think uh the root cause is

[16:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=988s) issues if we think uh the root cause is

[16:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=988s) issues if we think uh the root cause is uh near uh duplicates so uh I can give

[16:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=991s) uh near uh duplicates so uh I can give

[16:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=991s) uh near uh duplicates so uh I can give one example here so for example uh for

[16:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=994s) one example here so for example uh for

[16:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=994s) one example here so for example uh for one row for one member uh the root cause

[16:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=997s) one row for one member uh the root cause

[16:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=997s) one row for one member uh the root cause can be uh the members uh profile data is

[16:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1001s) can be uh the members uh profile data is

[16:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1001s) can be uh the members uh profile data is uh missing uh and another row uh uh so

[16:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1005s) uh missing uh and another row uh uh so

[16:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1005s) uh missing uh and another row uh uh so the arm can

[16:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1007s) the arm can

[16:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1007s) the arm can the profile data is uh not showing the

[16:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1010s) the profile data is uh not showing the

[16:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1010s) the profile data is uh not showing the full picture of this users uh like a

[16:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1013s) full picture of this users uh like a

[16:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1013s) full picture of this users uh like a browsing history or whatever. So b like

[16:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1015s) browsing history or whatever. So b like

[16:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1015s) browsing history or whatever. So b like they are uh both of them are describing

[16:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1018s) they are uh both of them are describing

[16:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1018s) they are uh both of them are describing the same thing. So the profile data is

[17:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1020s) the same thing. So the profile data is

[17:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1020s) the same thing. So the profile data is not supporting the uh this agent output.

[17:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1024s) not supporting the uh this agent output.

[17:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1024s) not supporting the uh this agent output. So uh but they just use a different

[17:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1026s) So uh but they just use a different

[17:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1026s) So uh but they just use a different text. So that's we are like that's why

[17:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1029s) text. So that's we are like that's why

[17:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1029s) text. So that's we are like that's why we need this um kind like a dduplication

[17:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1032s) we need this um kind like a dduplication

[17:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1032s) we need this um kind like a dduplication uh step to merge the similar issues so

[17:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1035s) uh step to merge the similar issues so

[17:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1035s) uh step to merge the similar issues so we can get some uh canonical uh issue I

[17:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1037s) we can get some uh canonical uh issue I

[17:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1037s) we can get some uh canonical uh issue I say the roo uh labels

[17:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1041s) say the roo uh labels

[17:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1041s) say the roo uh labels and uh once we finish the step three so

[17:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1043s) and uh once we finish the step three so

[17:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1043s) and uh once we finish the step three so the next step is to based on the uh

[17:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1045s) the next step is to based on the uh

[17:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1046s) the next step is to based on the uh canonical issues we found we are group

[17:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1048s) canonical issues we found we are group

[17:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1048s) canonical issues we found we are group and rank them by the uh frequency and uh

[17:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1051s) and rank them by the uh frequency and uh

[17:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1051s) and rank them by the uh frequency and uh this step uh there's no

[17:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1054s) this step uh there's no

[17:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1054s) this step uh there's no am required so this is just to help us

[17:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1057s) am required so this is just to help us

[17:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1057s) am required so this is just to help us to understand uh what is the major

[17:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1060s) to understand uh what is the major

[17:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1060s) to understand uh what is the major issues bottlenecks uh in our uh AI

[17:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1063s) issues bottlenecks uh in our uh AI

[17:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1063s) issues bottlenecks uh in our uh AI system and uh once we uh do the group

[17:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1066s) system and uh once we uh do the group

[17:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1066s) system and uh once we uh do the group and ranking and we are generated the uh

[17:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1068s) and ranking and we are generated the uh

[17:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1068s) and ranking and we are generated the uh scorec card for the human to uh get uh

[17:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1071s) scorec card for the human to uh get uh

[17:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1071s) scorec card for the human to uh get uh to just uh get a human to act on to uh

[17:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1075s) to just uh get a human to act on to uh

[17:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1076s) to just uh get a human to act on to uh continue uh iterate our uh AI system and

[17:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1079s) continue uh iterate our uh AI system and

[17:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1079s) continue uh iterate our uh AI system and all uh and also we are uh uh like

[18:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1082s) all uh and also we are uh uh like

[18:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1082s) all uh and also we are uh uh like basically just a cache I say save those

[18:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1085s) basically just a cache I say save those

[18:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1085s) basically just a cache I say save those root cause uh somewhere and it will

[18:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1088s) root cause uh somewhere and it will

[18:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1088s) root cause uh somewhere and it will continue use this for the self question

[18:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1090s) continue use this for the self question

[18:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1090s) continue use this for the self question loop. So that's why so you can see uh

[18:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1093s) loop. So that's why so you can see uh

[18:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1093s) loop. So that's why so you can see uh like there's a arrow here like a go back

[18:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1096s) like there's a arrow here like a go back

[18:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1096s) like there's a arrow here like a go back to the uh L2 judge. So be like it will

[18:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1099s) to the uh L2 judge. So be like it will

[18:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1099s) to the uh L2 judge. So be like it will guide us on like what is basically the

[18:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1102s) guide us on like what is basically the

[18:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1102s) guide us on like what is basically the pri prior diagnose prime on the next

[18:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1104s) pri prior diagnose prime on the next

[18:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1104s) pri prior diagnose prime on the next turn. Uh so that is easier for the uh

[18:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1107s) turn. Uh so that is easier for the uh

[18:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1107s) turn. Uh so that is easier for the uh like to judge to identify the issues uh

[18:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1110s) like to judge to identify the issues uh

[18:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1110s) like to judge to identify the issues uh in future runs. So that's uh uh like the

[18:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1114s) in future runs. So that's uh uh like the

[18:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1114s) in future runs. So that's uh uh like the what we do for the agent uh reflection.

[18:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1117s) what we do for the agent uh reflection.

[18:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1117s) what we do for the agent uh reflection. Uh so the next step is a generator edits

[18:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1120s) Uh so the next step is a generator edits

[18:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1120s) Uh so the next step is a generator edits proposal because from the step three uh

[18:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1123s) proposal because from the step three uh

[18:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1123s) proposal because from the step three uh the previous slide we know uh we already

[18:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1125s) the previous slide we know uh we already

[18:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1125s) the previous slide we know uh we already know okay so what is the root cause

[18:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1127s) know okay so what is the root cause

[18:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1127s) know okay so what is the root cause right why like the agent fails so the

[18:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1130s) right why like the agent fails so the

[18:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1130s) right why like the agent fails so the next step is naturally like given uh

[18:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1133s) next step is naturally like given uh

[18:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1133s) next step is naturally like given uh those root cause those issues we want to

[18:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1135s) those root cause those issues we want to

[18:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1135s) those root cause those issues we want to fix so what what can we do right what is

[18:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1139s) fix so what what can we do right what is

[18:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1139s) fix so what what can we do right what is u new edits we want to propose and uh

[19:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1142s) u new edits we want to propose and uh

[19:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1142s) u new edits we want to propose and uh here like

[19:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1144s) here like

[19:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1144s) here like we have uh again like this kind of we

[19:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1147s) we have uh again like this kind of we

[19:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1147s) we have uh again like this kind of we call it the proposal uh life cycle. So

[19:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1149s) call it the proposal uh life cycle. So

[19:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1149s) call it the proposal uh life cycle. So based on the judge uh evidence and the

[19:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1152s) based on the judge uh evidence and the

[19:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1152s) based on the judge uh evidence and the reflection artifact we will propose the

[19:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1155s) reflection artifact we will propose the

[19:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1155s) reflection artifact we will propose the input and uh we will run uh some

[19:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1158s) input and uh we will run uh some

[19:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1158s) input and uh we will run uh some methodology. So uh for this project uh

[19:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1161s) methodology. So uh for this project uh

[19:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1161s) methodology. So uh for this project uh we are using the uh JPA to uh given the

[19:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1165s) we are using the uh JPA to uh given the

[19:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1165s) we are using the uh JPA to uh given the uh you know the uh lookout and to uh

[19:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1168s) uh you know the uh lookout and to uh

[19:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1168s) uh you know the uh lookout and to uh propose the new prompt change and uh and

[19:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1171s) propose the new prompt change and uh and

[19:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1172s) propose the new prompt change and uh and like given a trace sample we do it in a

[19:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1175s) like given a trace sample we do it in a

[19:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1175s) like given a trace sample we do it in a mini batch uh fashion and uh and

[19:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1178s) mini batch uh fashion and uh and

[19:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1178s) mini batch uh fashion and uh and generated the new uh prompt updates

[19:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1180s) generated the new uh prompt updates

[19:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1180s) generated the new uh prompt updates proposal and once we generate the promp

[19:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1183s) proposal and once we generate the promp

[19:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1183s) proposal and once we generate the promp uh updates proposal we'll uh select the

[19:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1186s) uh updates proposal we'll uh select the

[19:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1186s) uh updates proposal we'll uh select the window prompts and do the uh validation

[19:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1189s) window prompts and do the uh validation

[19:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1189s) window prompts and do the uh validation validation uh set

[19:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1192s) validation uh set

[19:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1192s) validation uh set and uh once we uh found a good prompt

[19:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1195s) and uh once we uh found a good prompt

[19:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1195s) and uh once we uh found a good prompt candidates and we run the uh you know

[19:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1198s) candidates and we run the uh you know

[19:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1198s) candidates and we run the uh you know the validation set and we think uh it is

[20:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1201s) the validation set and we think uh it is

[20:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1201s) the validation set and we think uh it is uh become uh it is a good uh prompt

[20:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1204s) uh become uh it is a good uh prompt

[20:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1204s) uh become uh it is a good uh prompt updates and uh we are just um updates

[20:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1208s) updates and uh we are just um updates

[20:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1208s) updates and uh we are just um updates our prompt in our system. So this will

[20:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1210s) our prompt in our system. So this will

[20:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1210s) our prompt in our system. So this will be and into the production and this will

[20:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1213s) be and into the production and this will

[20:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1213s) be and into the production and this will become our new uh kind of like a

[20:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1215s) become our new uh kind of like a

[20:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1215s) become our new uh kind of like a baseline and we will continue to uh do

[20:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1217s) baseline and we will continue to uh do

[20:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1217s) baseline and we will continue to uh do the uh like the future uh RSI uh

[20:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1221s) the uh like the future uh RSI uh

[20:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1221s) the uh like the future uh RSI uh iterations. So yeah that's uh kind of

[20:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1224s) iterations. So yeah that's uh kind of

[20:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1224s) iterations. So yeah that's uh kind of like the and uh yeah that's a kind of

[20:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1226s) like the and uh yeah that's a kind of

[20:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1226s) like the and uh yeah that's a kind of like the entire life cycle of this RSI

[20:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1229s) like the entire life cycle of this RSI

[20:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1229s) like the entire life cycle of this RSI uh iteration. And uh next this slide uh

[20:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1233s) uh iteration. And uh next this slide uh

[20:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1233s) uh iteration. And uh next this slide uh we want to show our primary focus for

[20:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1235s) we want to show our primary focus for

[20:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1235s) we want to show our primary focus for this RSI work. So we so currently we

[20:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1239s) this RSI work. So we so currently we

[20:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1239s) this RSI work. So we so currently we have three major focuses. So the first

[20:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1241s) have three major focuses. So the first

[20:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1241s) have three major focuses. So the first is the quality, second is automation and

[20:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1244s) is the quality, second is automation and

[20:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1244s) is the quality, second is automation and the third is a production uh updates. So

[20:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1247s) the third is a production uh updates. So

[20:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1247s) the third is a production uh updates. So quality uh like we are focused on the

[20:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1250s) quality uh like we are focused on the

[20:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1250s) quality uh like we are focused on the different uh uh things uh on the

[20:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1253s) different uh uh things uh on the

[20:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1253s) different uh uh things uh on the quality. So um and we uh did many uh

[20:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1256s) quality. So um and we uh did many uh

[20:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1256s) quality. So um and we uh did many uh iterations to trying to improve the uh

[20:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1259s) iterations to trying to improve the uh

[20:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1259s) iterations to trying to improve the uh quality of our agent uh judge and this

[21:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1263s) quality of our agent uh judge and this

[21:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1263s) quality of our agent uh judge and this RSI loop. For example, we did extensive

[21:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1265s) RSI loop. For example, we did extensive

[21:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1266s) RSI loop. For example, we did extensive human annotations so that we can improve

[21:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1269s) human annotations so that we can improve

[21:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1269s) human annotations so that we can improve the judge both the L1 judge and the L2

[21:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1272s) the judge both the L1 judge and the L2

[21:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1272s) the judge both the L1 judge and the L2 judge uh quality and we did uh many

[21:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1275s) judge uh quality and we did uh many

[21:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1275s) judge uh quality and we did uh many study and analysis on assess agent judge

[21:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1278s) study and analysis on assess agent judge

[21:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1278s) study and analysis on assess agent judge uh uh variance so that we can uh make

[21:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1281s) uh uh variance so that we can uh make

[21:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1281s) uh uh variance so that we can uh make sure we can kind of mitigate the impact

[21:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1284s) sure we can kind of mitigate the impact

[21:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1284s) sure we can kind of mitigate the impact from the non nondeterministic nature of

[21:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1287s) from the non nondeterministic nature of

[21:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1287s) from the non nondeterministic nature of the LM and uh uh the third one is we

[21:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1290s) the LM and uh uh the third one is we

[21:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1290s) the LM and uh uh the third one is we push quality updates to both Asian judge

[21:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1293s) push quality updates to both Asian judge

[21:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1293s) push quality updates to both Asian judge and RSI reflection uh based on our

[21:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1296s) and RSI reflection uh based on our

[21:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1296s) and RSI reflection uh based on our annotation and the variance analysis uh

[21:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1299s) annotation and the variance analysis uh

[21:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1300s) annotation and the variance analysis uh mentioned uh above. So that's on the CI

[21:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1303s) mentioned uh above. So that's on the CI

[21:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1303s) mentioned uh above. So that's on the CI side and the second focus is uh

[21:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1305s) side and the second focus is uh

[21:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1305s) side and the second focus is uh automation uh because as we mentioned so

[21:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1308s) automation uh because as we mentioned so

[21:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1308s) automation uh because as we mentioned so there are seven steps in the RSI

[21:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1311s) there are seven steps in the RSI

[21:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1311s) there are seven steps in the RSI iteration and uh so that's why like here

[21:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1314s) iteration and uh so that's why like here

[21:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1314s) iteration and uh so that's why like here like I we build a uh this kind like

[21:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1317s) like I we build a uh this kind like

[21:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1317s) like I we build a uh this kind like workflow so that we can kick off this

[22:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1320s) workflow so that we can kick off this

[22:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1320s) workflow so that we can kick off this workflow with some uh cadence for weekly

[22:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1323s) workflow with some uh cadence for weekly

[22:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1323s) workflow with some uh cadence for weekly or bi-weekly or monthly based on the

[22:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1325s) or bi-weekly or monthly based on the

[22:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1325s) or bi-weekly or monthly based on the needs uh so that it can u automatically

[22:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1328s) needs uh so that it can u automatically

[22:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1328s) needs uh so that it can u automatically running and keep suggesting the uh like

[22:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1331s) running and keep suggesting the uh like

[22:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1331s) running and keep suggesting the uh like the prompt updates uh with some uh

[22:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1333s) the prompt updates uh with some uh

[22:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1333s) the prompt updates uh with some uh cadence and the third focus is

[22:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1336s) cadence and the third focus is

[22:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1336s) cadence and the third focus is production uh updates. So definitely

[22:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1338s) production uh updates. So definitely

[22:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1338s) production uh updates. So definitely like we want to uh build a uh useful uh

[22:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1342s) like we want to uh build a uh useful uh

[22:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1342s) like we want to uh build a uh useful uh RSI uh system that can help uh keep

[22:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1345s) RSI uh system that can help uh keep

[22:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1346s) RSI uh system that can help uh keep continuously uh improve our uh you know

[22:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1348s) continuously uh improve our uh you know

[22:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1348s) continuously uh improve our uh you know product. So that's why like we the third

[22:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1351s) product. So that's why like we the third

[22:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1351s) product. So that's why like we the third focus uh for the our our uh work is to

[22:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1355s) focus uh for the our our uh work is to

[22:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1355s) focus uh for the our our uh work is to make updates to the current agent in

[22:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1357s) make updates to the current agent in

[22:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1357s) make updates to the current agent in production with human in the loop uh

[22:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1360s) production with human in the loop uh

[22:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1360s) production with human in the loop uh like for now to uh ensure like the

[22:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1363s) like for now to uh ensure like the

[22:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1363s) like for now to uh ensure like the production uh uh quality uh is there but

[22:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1366s) production uh uh quality uh is there but

[22:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1366s) production uh uh quality uh is there but later like once we think uh the quality

[22:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1369s) later like once we think uh the quality

[22:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1369s) later like once we think uh the quality is good enough like we gradually uh uh

[22:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1373s) is good enough like we gradually uh uh

[22:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1373s) is good enough like we gradually uh uh like reduce the human effort and uh and

[22:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1377s) like reduce the human effort and uh and

[22:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1377s) like reduce the human effort and uh and hopefully we can achieve a higher

[22:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1379s) hopefully we can achieve a higher

[22:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1379s) hopefully we can achieve a higher automation uh kind of uh state.

[23:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1383s) automation uh kind of uh state.

[23:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1383s) automation uh kind of uh state. Uh cool. Yeah. Next, I will hand over uh

[23:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1386s) Uh cool. Yeah. Next, I will hand over uh

[23:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1386s) Uh cool. Yeah. Next, I will hand over uh to Saloni to uh share more details about

[23:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1389s) to Saloni to uh share more details about

[23:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1389s) to Saloni to uh share more details about the outcomes and uh learnings of our our

[23:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1393s) the outcomes and uh learnings of our our

[23:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1393s) the outcomes and uh learnings of our our work.

[23:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1396s) I will share my screen.

[23:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1398s) I will share my screen.

[23:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1398s) I will share my screen. Sure. Yeah.

[23:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1409s) Um, can you see my screen?

[23:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1411s) Um, can you see my screen?

[23:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1411s) Um, can you see my screen? Yeah. Yeah.

[23:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1413s) Yeah. Yeah.

[23:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1413s) Yeah. Yeah. Yeah. Um, so yeah. Hi, I'm Salon. And so

[23:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1415s) Yeah. Um, so yeah. Hi, I'm Salon. And so

[23:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1416s) Yeah. Um, so yeah. Hi, I'm Salon. And so far we described how the RSI system is

[23:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1419s) far we described how the RSI system is

[23:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1419s) far we described how the RSI system is designed. Uh, but one of the most

[23:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1420s) designed. Uh, but one of the most

[23:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1420s) designed. Uh, but one of the most interesting aspects of this work has

[23:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1422s) interesting aspects of this work has

[23:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1422s) interesting aspects of this work has been understanding what happens when we

[23:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1425s) been understanding what happens when we

[23:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1425s) been understanding what happens when we actually try to make this process

[23:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1427s) actually try to make this process

[23:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1427s) actually try to make this process reliable. And in particular, we learned

[23:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1429s) reliable. And in particular, we learned

[23:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1429s) reliable. And in particular, we learned that the quality of the evaluator is

[23:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1432s) that the quality of the evaluator is

[23:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1432s) that the quality of the evaluator is just as important as the quality of the

[23:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1434s) just as important as the quality of the

[23:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1434s) just as important as the quality of the optimization system. So let's walk

[23:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1436s) optimization system. So let's walk

[23:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1436s) optimization system. So let's walk through some of those um findings. Um so

[24:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1440s) through some of those um findings. Um so

[24:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1440s) through some of those um findings. Um so there are two major outcomes we want to

[24:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1441s) there are two major outcomes we want to

[24:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1441s) there are two major outcomes we want to highlight. So the first is that we

[24:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1443s) highlight. So the first is that we

[24:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1443s) highlight. So the first is that we successfully like deployed our initial

[24:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1445s) successfully like deployed our initial

[24:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1445s) successfully like deployed our initial RSI improvements into uh production

[24:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1448s) RSI improvements into uh production

[24:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1448s) RSI improvements into uh production which was like um landing a first PR

[24:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1451s) which was like um landing a first PR

[24:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1451s) which was like um landing a first PR which was optimized by Japa and through

[24:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1453s) which was optimized by Japa and through

[24:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1453s) which was optimized by Japa and through the loop which we just described. Um so

[24:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1455s) the loop which we just described. Um so

[24:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1455s) the loop which we just described. Um so these changes went through human review

[24:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1457s) these changes went through human review

[24:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1457s) these changes went through human review and quantitative measurement before

[24:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1459s) and quantitative measurement before

[24:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1459s) and quantitative measurement before deployment and it was an important

[24:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1461s) deployment and it was an important

[24:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1461s) deployment and it was an important milestone in moving RSI like beyond just

[24:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1463s) milestone in moving RSI like beyond just

[24:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1463s) milestone in moving RSI like beyond just an experimental workflow. The second

[24:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1466s) an experimental workflow. The second

[24:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1466s) an experimental workflow. The second major improvement was around like um

[24:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1468s) major improvement was around like um

[24:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1468s) major improvement was around like um evaluation quality where we invested

[24:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1470s) evaluation quality where we invested

[24:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1470s) evaluation quality where we invested substantial effort and like human

[24:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1472s) substantial effort and like human

[24:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1472s) substantial effort and like human annotations for the user understanding

[24:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1475s) annotations for the user understanding

[24:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1475s) annotations for the user understanding judge and as well as like analyzing

[24:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1476s) judge and as well as like analyzing

[24:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1476s) judge and as well as like analyzing where its judgment sort of like

[24:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1478s) where its judgment sort of like

[24:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1478s) where its judgment sort of like disagreed with like human reviewers. And

[24:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1481s) disagreed with like human reviewers. And

[24:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1481s) disagreed with like human reviewers. And this process revealed several recurring

[24:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1483s) this process revealed several recurring

[24:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1483s) this process revealed several recurring weaknesses in both our evaluation

[24:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1485s) weaknesses in both our evaluation

[24:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1485s) weaknesses in both our evaluation rubrics as well as our like our

[24:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1486s) rubrics as well as our like our

[24:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1486s) rubrics as well as our like our reflection system. And by investigating

[24:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1489s) reflection system. And by investigating

[24:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1489s) reflection system. And by investigating those disagreements, we were able to

[24:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1490s) those disagreements, we were able to

[24:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1490s) those disagreements, we were able to like make targeted improvements which

[24:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1493s) like make targeted improvements which

[24:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1493s) like make targeted improvements which further strengthen the reliability as

[24:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1495s) further strengthen the reliability as

[24:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1495s) further strengthen the reliability as well as like reduce the evaluation

[24:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1496s) well as like reduce the evaluation

[24:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1496s) well as like reduce the evaluation variance which we have. And uh before we

[24:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1499s) variance which we have. And uh before we

[24:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1499s) variance which we have. And uh before we could trust a system to like improve an

[25:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1501s) could trust a system to like improve an

[25:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1501s) could trust a system to like improve an agent automatically, we had to

[25:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1502s) agent automatically, we had to

[25:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1502s) agent automatically, we had to understand whether the feedback driving

[25:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1504s) understand whether the feedback driving

[25:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1504s) understand whether the feedback driving those improvements was also itself

[25:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1506s) those improvements was also itself

[25:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1506s) those improvements was also itself trustworthy. And this was like a major

[25:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1509s) trustworthy. And this was like a major

[25:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1509s) trustworthy. And this was like a major area of our engineering work. Um to so

[25:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1512s) area of our engineering work. Um to so

[25:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1512s) area of our engineering work. Um to so to understand the quality of our like um

[25:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1514s) to understand the quality of our like um

[25:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1514s) to understand the quality of our like um L1 which was our evaluation judge. We

[25:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1516s) L1 which was our evaluation judge. We

[25:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1516s) L1 which was our evaluation judge. We analyzed about 180 to 200 human reviews.

[25:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1520s) analyzed about 180 to 200 human reviews.

[25:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1520s) analyzed about 180 to 200 human reviews. We were particularly interested in how

[25:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1522s) We were particularly interested in how

[25:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1522s) We were particularly interested in how the judge agreed with human reviewers

[25:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1524s) the judge agreed with human reviewers

[25:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1524s) the judge agreed with human reviewers both across different score categories

[25:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1526s) both across different score categories

[25:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1526s) both across different score categories and across the individual evaluation

[25:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1529s) and across the individual evaluation

[25:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1529s) and across the individual evaluation dimensions. Starting with the chart on

[25:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1531s) dimensions. Starting with the chart on

[25:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1531s) dimensions. Starting with the chart on the left, we see basically that um for

[25:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1533s) the left, we see basically that um for

[25:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1533s) the left, we see basically that um for outputs we assigned a score of like one

[25:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1536s) outputs we assigned a score of like one

[25:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1536s) outputs we assigned a score of like one um a full judgment alignment was

[25:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1538s) um a full judgment alignment was

[25:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1538s) um a full judgment alignment was approximately 55% and for score two it

[25:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1541s) approximately 55% and for score two it

[25:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1541s) approximately 55% and for score two it was around 57%. But for score three the

[25:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1544s) was around 57%. But for score three the

[25:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1544s) was around 57%. But for score three the alignment increased to approximately

[25:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1546s) alignment increased to approximately

[25:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1546s) alignment increased to approximately like 80%. Which indicates an interesting

[25:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1549s) like 80%. Which indicates an interesting

[25:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1549s) like 80%. Which indicates an interesting pattern which was that the judge was

[25:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1551s) pattern which was that the judge was

[25:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1551s) pattern which was that the judge was considerably like better at recognizing

[25:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1552s) considerably like better at recognizing

[25:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1552s) considerably like better at recognizing stronger outputs than it was at like

[25:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1555s) stronger outputs than it was at like

[25:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1555s) stronger outputs than it was at like distinguishing between different sort of

[25:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1557s) distinguishing between different sort of

[25:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1557s) distinguishing between different sort of like weaker outputs. And we found that

[25:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1559s) like weaker outputs. And we found that

[25:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1559s) like weaker outputs. And we found that many of its uh lowest scores were like

[26:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1562s) many of its uh lowest scores were like

[26:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1562s) many of its uh lowest scores were like also overly harsh. And um now looking at

[26:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1565s) also overly harsh. And um now looking at

[26:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1565s) also overly harsh. And um now looking at the chart on right agreement also varied

[26:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1568s) the chart on right agreement also varied

[26:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1568s) the chart on right agreement also varied significantly across like different

[26:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1569s) significantly across like different

[26:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1569s) significantly across like different dimensions where higher preferred

[26:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1572s) dimensions where higher preferred

[26:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1572s) dimensions where higher preferred dimensions like titles had like the

[26:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1574s) dimensions like titles had like the

[26:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1574s) dimensions like titles had like the weakest observed alignment as well as

[26:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1575s) weakest observed alignment as well as

[26:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1575s) weakest observed alignment as well as like uh domains or like the expertise or

[26:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1578s) like uh domains or like the expertise or

[26:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1578s) like uh domains or like the expertise or area of work was around like 47% and the

[26:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1582s) area of work was around like 47% and the

[26:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1582s) area of work was around like 47% and the sort of like work type was around 50%.

[26:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1585s) sort of like work type was around 50%.

[26:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1585s) sort of like work type was around 50%. which told us like the evaluation

[26:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1587s) which told us like the evaluation

[26:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1587s) which told us like the evaluation quality was uniform across all the tasks

[26:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1589s) quality was uniform across all the tasks

[26:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1589s) quality was uniform across all the tasks which we cared about and certain

[26:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1591s) which we cared about and certain

[26:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1591s) which we cared about and certain dimensions required more clearer

[26:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1593s) dimensions required more clearer

[26:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1593s) dimensions required more clearer definitions better evidence handling

[26:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1595s) definitions better evidence handling

[26:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1595s) definitions better evidence handling rules as well as like more explicit

[26:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1597s) rules as well as like more explicit

[26:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1597s) rules as well as like more explicit scoring guidance.

[26:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1600s) scoring guidance.

[26:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1600s) scoring guidance. Um so when we look more closely at the

[26:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1601s) Um so when we look more closely at the

[26:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1602s) Um so when we look more closely at the human annotations several recurring

[26:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1603s) human annotations several recurring

[26:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1603s) human annotations several recurring patterns emerged. the first involved

[26:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1606s) patterns emerged. the first involved

[26:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1606s) patterns emerged. the first involved like using job titles. We found cases

[26:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1608s) like using job titles. We found cases

[26:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1608s) like using job titles. We found cases where the judge would accept like a

[26:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1610s) where the judge would accept like a

[26:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1610s) where the judge would accept like a return title because it was plausible

[26:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1612s) return title because it was plausible

[26:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1612s) return title because it was plausible would but would miss the fact that the

[26:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1613s) would but would miss the fact that the

[26:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1613s) would but would miss the fact that the agent had omitted an expected um signal

[26:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1617s) agent had omitted an expected um signal

[26:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1617s) agent had omitted an expected um signal or like an ignored and basically an

[26:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1619s) or like an ignored and basically an

[26:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1619s) or like an ignored and basically an ignored and important intent signal. Um

[27:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1621s) ignored and important intent signal. Um

[27:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1622s) ignored and important intent signal. Um in other words, sometimes the judge was

[27:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1623s) in other words, sometimes the judge was

[27:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1623s) in other words, sometimes the judge was sometimes evaluating the plausibility

[27:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1625s) sometimes evaluating the plausibility

[27:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1625s) sometimes evaluating the plausibility rather than the completeness of the

[27:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1627s) rather than the completeness of the

[27:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1627s) rather than the completeness of the facts which were stated. Uh the second

[27:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1629s) facts which were stated. Uh the second

[27:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1629s) facts which were stated. Uh the second issue was primarily around primarily

[27:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1631s) issue was primarily around primarily

[27:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1631s) issue was primarily around primarily around the fields of expertise. Here the

[27:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1633s) around the fields of expertise. Here the

[27:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1634s) around the fields of expertise. Here the distinction between like a domain,

[27:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1635s) distinction between like a domain,

[27:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1635s) distinction between like a domain, function and a skill wasn't like always

[27:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1637s) function and a skill wasn't like always

[27:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1637s) function and a skill wasn't like always clear where um a domain could be like um

[27:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1640s) clear where um a domain could be like um

[27:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1640s) clear where um a domain could be like um say software engineering and a function

[27:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1643s) say software engineering and a function

[27:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1643s) say software engineering and a function or a role could be like for example a

[27:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1645s) or a role could be like for example a

[27:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1645s) or a role could be like for example a machine learning engineer while skill

[27:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1647s) machine learning engineer while skill

[27:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1647s) machine learning engineer while skill being uh machine learning. Um the rubric

[27:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1649s) being uh machine learning. Um the rubric

[27:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1649s) being uh machine learning. Um the rubric also like lack consistent partial credit

[27:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1652s) also like lack consistent partial credit

[27:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1652s) also like lack consistent partial credit rules for output containing like a

[27:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1654s) rules for output containing like a

[27:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1654s) rules for output containing like a mixture of like correct and incorrect

[27:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1655s) mixture of like correct and incorrect

[27:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1655s) mixture of like correct and incorrect information. And another issue was

[27:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1658s) information. And another issue was

[27:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1658s) information. And another issue was involving around formatting and

[27:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1660s) involving around formatting and

[27:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1660s) involving around formatting and inference. So in some cases the judge

[27:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1662s) inference. So in some cases the judge

[27:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1662s) inference. So in some cases the judge penalized harmless formatting

[27:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1663s) penalized harmless formatting

[27:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1663s) penalized harmless formatting differences. In other words, uh it made

[27:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1666s) differences. In other words, uh it made

[27:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1666s) differences. In other words, uh it made it inconsistent assumptions about what

[27:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1668s) it inconsistent assumptions about what

[27:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1668s) it inconsistent assumptions about what could be inferred from like a member's

[27:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1670s) could be inferred from like a member's

[27:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1670s) could be inferred from like a member's existing employment information too. We

[27:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1673s) existing employment information too. We

[27:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1673s) existing employment information too. We also examined like the distribution of

[27:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1675s) also examined like the distribution of

[27:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1675s) also examined like the distribution of overly harsh judgments. Out of like the

[27:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1677s) overly harsh judgments. Out of like the

[27:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1677s) overly harsh judgments. Out of like the 37 um judgments identified as too harsh,

[28:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1680s) 37 um judgments identified as too harsh,

[28:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1680s) 37 um judgments identified as too harsh, 31 of them involved outputs that had

[28:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1683s) 31 of them involved outputs that had

[28:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1683s) 31 of them involved outputs that had been assigned a score of one, which

[28:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1685s) been assigned a score of one, which

[28:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1685s) been assigned a score of one, which suggested that the boundaries between

[28:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1687s) suggested that the boundaries between

[28:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1687s) suggested that the boundaries between our scores needed to be like

[28:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1688s) our scores needed to be like

[28:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1688s) our scores needed to be like recalibrated.

[28:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1690s) recalibrated.

[28:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1690s) recalibrated. Um, another recurring source of

[28:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1693s) Um, another recurring source of

[28:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1693s) Um, another recurring source of disagreement was whether an inferred

[28:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1694s) disagreement was whether an inferred

[28:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1694s) disagreement was whether an inferred member's preference should be considered

[28:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1696s) member's preference should be considered

[28:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1696s) member's preference should be considered valid when the evidence wasn't like

[28:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1698s) valid when the evidence wasn't like

[28:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1698s) valid when the evidence wasn't like explicitly stated. And the broader

[28:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1701s) explicitly stated. And the broader

[28:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1701s) explicitly stated. And the broader lesson here was that many evaluation

[28:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1702s) lesson here was that many evaluation

[28:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1702s) lesson here was that many evaluation disagreements were not simply like model

[28:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1704s) disagreements were not simply like model

[28:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1704s) disagreements were not simply like model periods. They were reflecting

[28:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1706s) periods. They were reflecting

[28:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1706s) periods. They were reflecting ambiguities in the rubric as well as

[28:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1707s) ambiguities in the rubric as well as

[28:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1707s) ambiguities in the rubric as well as like the underlying evaluation policy.

[28:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1711s) like the underlying evaluation policy.

[28:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1711s) like the underlying evaluation policy. Um, based on these findings, we

[28:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1712s) Um, based on these findings, we

[28:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1712s) Um, based on these findings, we introduced several targeted improvements

[28:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1714s) introduced several targeted improvements

[28:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1714s) introduced several targeted improvements to our evaluation rubrics. One was like

[28:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1717s) to our evaluation rubrics. One was like

[28:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1717s) to our evaluation rubrics. One was like basically fixing harmless formatting

[28:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1719s) basically fixing harmless formatting

[28:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1719s) basically fixing harmless formatting differences. Um, previously like a value

[28:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1722s) differences. Um, previously like a value

[28:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1722s) differences. Um, previously like a value such as like you know full-time could

[28:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1724s) such as like you know full-time could

[28:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1724s) such as like you know full-time could receive an incorrect score because of

[28:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1726s) receive an incorrect score because of

[28:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1726s) receive an incorrect score because of like capitalization or incorrect

[28:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1727s) like capitalization or incorrect

[28:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1727s) like capitalization or incorrect formatting. We updated the rubrics so

[28:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1730s) formatting. We updated the rubrics so

[28:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1730s) formatting. We updated the rubrics so that these differences would not be

[28:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1731s) that these differences would not be

[28:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1731s) that these differences would not be treated as like semantic failures.

[28:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1733s) treated as like semantic failures.

[28:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1733s) treated as like semantic failures. Similarly, we introduced normalization

[28:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1735s) Similarly, we introduced normalization

[28:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1735s) Similarly, we introduced normalization guidance for like titles. Um we also

[28:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1738s) guidance for like titles. Um we also

[28:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1738s) guidance for like titles. Um we also clarified how the judge should handle

[29:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1740s) clarified how the judge should handle

[29:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1740s) clarified how the judge should handle concurrent employment like you know sort

[29:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1742s) concurrent employment like you know sort

[29:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1742s) concurrent employment like you know sort of impairing seniority as well as like

[29:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1745s) of impairing seniority as well as like

[29:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1745s) of impairing seniority as well as like impairing geographic references of the

[29:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1747s) impairing geographic references of the

[29:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1747s) impairing geographic references of the user and more broadly we recalibrated

[29:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1749s) user and more broadly we recalibrated

[29:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1749s) user and more broadly we recalibrated the rules uh separating low scores from

[29:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1751s) the rules uh separating low scores from

[29:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1751s) the rules uh separating low scores from like partially correct outputs and these

[29:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1754s) like partially correct outputs and these

[29:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1754s) like partially correct outputs and these changes resulted like resulted in like

[29:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1756s) changes resulted like resulted in like

[29:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1756s) changes resulted like resulted in like measurable improvements uh where human

[29:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1758s) measurable improvements uh where human

[29:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1758s) measurable improvements uh where human alignment increased from like 57% to

[29:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1760s) alignment increased from like 57% to

[29:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1760s) alignment increased from like 57% to like 64%

[29:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1762s) like 64%

[29:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1762s) like 64% and um at the same time the mean error

[29:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1764s) and um at the same time the mean error

[29:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1764s) and um at the same time the mean error decreased from like 96 to 74 What we're

[29:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1767s) decreased from like 96 to 74 What we're

[29:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1767s) decreased from like 96 to 74 What we're trying to achieve here is to get our

[29:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1768s) trying to achieve here is to get our

[29:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1768s) trying to achieve here is to get our system as high or like as highly

[29:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1771s) system as high or like as highly

[29:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1771s) system as high or like as highly aligning to humans as much as possible.

[29:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1774s) aligning to humans as much as possible.

[29:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1774s) aligning to humans as much as possible. And what we find particularly

[29:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1775s) And what we find particularly

[29:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1775s) And what we find particularly interesting here is that we didn't

[29:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1776s) interesting here is that we didn't

[29:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1776s) interesting here is that we didn't achieve these improvements by only like

[29:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1778s) achieve these improvements by only like

[29:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1778s) achieve these improvements by only like handling the model, but we also improved

[29:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1780s) handling the model, but we also improved

[29:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1780s) handling the model, but we also improved specifications of what like the correct

[29:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1782s) specifications of what like the correct

[29:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1782s) specifications of what like the correct evaluation should look like. That is

[29:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1783s) evaluation should look like. That is

[29:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1783s) evaluation should look like. That is like rubric design as well as like um

[29:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1787s) like rubric design as well as like um

[29:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1787s) like rubric design as well as like um evidences which our evaluator is like

[29:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1789s) evidences which our evaluator is like

[29:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1789s) evidences which our evaluator is like spitting out. Um

[29:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1792s) spitting out. Um

[29:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1792s) spitting out. Um moving forward uh we perform like a

[29:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1794s) moving forward uh we perform like a

[29:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1794s) moving forward uh we perform like a similar human review analysis for our

[29:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1796s) similar human review analysis for our

[29:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1796s) similar human review analysis for our reflection agent which was described

[29:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1798s) reflection agent which was described

[29:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1798s) reflection agent which was described earlier in our pipeline. We should

[30:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1800s) earlier in our pipeline. We should

[30:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1800s) earlier in our pipeline. We should recall that our L2 takes an evaluation

[30:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1802s) recall that our L2 takes an evaluation

[30:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1802s) recall that our L2 takes an evaluation failure and then it tries to identify

[30:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1804s) failure and then it tries to identify

[30:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1804s) failure and then it tries to identify the root cause. For example, it

[30:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1807s) the root cause. For example, it

[30:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1807s) the root cause. For example, it basically identifies was the problem

[30:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1808s) basically identifies was the problem

[30:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1808s) basically identifies was the problem caused by the agent itself by the

[30:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1810s) caused by the agent itself by the

[30:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1810s) caused by the agent itself by the evaluation rubric or by some other issue

[30:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1812s) evaluation rubric or by some other issue

[30:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1812s) evaluation rubric or by some other issue with the available context. What we

[30:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1815s) with the available context. What we

[30:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1815s) with the available context. What we discovered was that root cause

[30:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1816s) discovered was that root cause

[30:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1816s) discovered was that root cause attribution was substantially harder

[30:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1818s) attribution was substantially harder

[30:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1818s) attribution was substantially harder than like simply identifying a failure.

[30:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1821s) than like simply identifying a failure.

[30:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1821s) than like simply identifying a failure. Human reviewers agreed more often when

[30:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1823s) Human reviewers agreed more often when

[30:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1824s) Human reviewers agreed more often when L2 attributed a problem to like the

[30:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1827s) L2 attributed a problem to like the

[30:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1827s) L2 attributed a problem to like the underlying model, but agreement was much

[30:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1829s) underlying model, but agreement was much

[30:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1830s) underlying model, but agreement was much lower for other causes, particularly

[30:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1831s) lower for other causes, particularly

[30:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1831s) lower for other causes, particularly like rubric related issues. Um, one

[30:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1834s) like rubric related issues. Um, one

[30:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1834s) like rubric related issues. Um, one important issue was that our original

[30:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1837s) important issue was that our original

[30:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1837s) important issue was that our original reflection taxonomy didn't include a way

[30:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1839s) reflection taxonomy didn't include a way

[30:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1839s) reflection taxonomy didn't include a way to explicitly say that or Elvin judge

[30:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1841s) to explicitly say that or Elvin judge

[30:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1841s) to explicitly say that or Elvin judge could have been wrong. Um, as a result,

[30:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1843s) could have been wrong. Um, as a result,

[30:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1843s) could have been wrong. Um, as a result, sometimes the reflection agent fell back

[30:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1846s) sometimes the reflection agent fell back

[30:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1846s) sometimes the reflection agent fell back to blaming the rubric even when the real

[30:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1848s) to blaming the rubric even when the real

[30:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1848s) to blaming the rubric even when the real problem was like an incorrect um, L1

[30:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1850s) problem was like an incorrect um, L1

[30:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1850s) problem was like an incorrect um, L1 judgment. Um, and like another issue was

[30:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1853s) judgment. Um, and like another issue was

[30:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1853s) judgment. Um, and like another issue was like the uh, reflection agent sometimes

[30:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1855s) like the uh, reflection agent sometimes

[30:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1856s) like the uh, reflection agent sometimes used the L1 judge's rationale without

[30:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1858s) used the L1 judge's rationale without

[30:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1858s) used the L1 judge's rationale without like independently validating or

[30:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1859s) like independently validating or

[30:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1859s) like independently validating or independently thinking over it against

[31:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1861s) independently thinking over it against

[31:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1861s) independently thinking over it against the agents output. And um we exposed an

[31:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1866s) the agents output. And um we exposed an

[31:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1866s) the agents output. And um we exposed an important risk here which was like a

[31:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1867s) important risk here which was like a

[31:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1867s) important risk here which was like a reflection system can produce convincing

[31:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1869s) reflection system can produce convincing

[31:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1869s) reflection system can produce convincing explanations while still like

[31:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1871s) explanations while still like

[31:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1871s) explanations while still like identifying misidentifying like the

[31:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1873s) identifying misidentifying like the

[31:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1873s) identifying misidentifying like the underlying problem and if we use those

[31:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1876s) underlying problem and if we use those

[31:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1876s) underlying problem and if we use those explanations to generate prompt changes

[31:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1877s) explanations to generate prompt changes

[31:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1878s) explanations to generate prompt changes we could like end up modifying like the

[31:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1879s) we could like end up modifying like the

[31:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1879s) we could like end up modifying like the wrong component of the system. So

[31:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1881s) wrong component of the system. So

[31:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1881s) wrong component of the system. So improving the L2 uh like reflection

[31:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1884s) improving the L2 uh like reflection

[31:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1884s) improving the L2 uh like reflection chart like became our next priority.

[31:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1888s) chart like became our next priority.

[31:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1888s) chart like became our next priority. Um so to talk about some of the quality

[31:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1890s) Um so to talk about some of the quality

[31:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1890s) Um so to talk about some of the quality improvements um these findings led us to

[31:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1893s) improvements um these findings led us to

[31:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1893s) improvements um these findings led us to like focus on improving both layers of

[31:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1895s) like focus on improving both layers of

[31:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1895s) like focus on improving both layers of evaluation. At L1 the goal was to make

[31:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1898s) evaluation. At L1 the goal was to make

[31:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1898s) evaluation. At L1 the goal was to make sure that the scoring was more accurate

[31:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1900s) sure that the scoring was more accurate

[31:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1900s) sure that the scoring was more accurate and more better aligned with human

[31:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1902s) and more better aligned with human

[31:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1902s) and more better aligned with human judgment. And at L2 the goal was to

[31:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1904s) judgment. And at L2 the goal was to

[31:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1904s) judgment. And at L2 the goal was to improve the reliability of the root

[31:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1906s) improve the reliability of the root

[31:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1906s) improve the reliability of the root cause diagnosis.

[31:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1910s) And um the next this slide basically

[31:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1913s) And um the next this slide basically

[31:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1913s) And um the next this slide basically summarizes some of like the changes we

[31:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1914s) summarizes some of like the changes we

[31:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1914s) summarizes some of like the changes we made to the system and the impact that

[31:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1916s) made to the system and the impact that

[31:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1916s) made to the system and the impact that we observed. We introduced like several

[31:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1918s) we observed. We introduced like several

[31:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1918s) we observed. We introduced like several targeted changes to address the um the

[32:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1921s) targeted changes to address the um the

[32:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1921s) targeted changes to address the um the reflection failures

[32:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1923s) reflection failures

[32:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1923s) reflection failures um identified through like human review

[32:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1926s) um identified through like human review

[32:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1926s) um identified through like human review and um

[32:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1929s) and um

[32:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1929s) and um we we were basically like trying to

[32:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1931s) we we were basically like trying to

[32:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1931s) we we were basically like trying to measure the impact here in terms of like

[32:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1932s) measure the impact here in terms of like

[32:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1932s) measure the impact here in terms of like how do we translate all this evaluation

[32:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1935s) how do we translate all this evaluation

[32:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1935s) how do we translate all this evaluation and reflection work into like measurable

[32:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1937s) and reflection work into like measurable

[32:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1937s) and reflection work into like measurable impact. So there are two levels to

[32:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1939s) impact. So there are two levels to

[32:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1939s) impact. So there are two levels to consider here. The first is like

[32:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1941s) consider here. The first is like

[32:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1941s) consider here. The first is like improving the reliability of the RSI

[32:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1943s) improving the reliability of the RSI

[32:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1943s) improving the reliability of the RSI process itself. So by improving L1 we

[32:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1945s) process itself. So by improving L1 we

[32:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1945s) process itself. So by improving L1 we make the um evaluation scores more

[32:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1948s) make the um evaluation scores more

[32:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1948s) make the um evaluation scores more stable and like more better aligned with

[32:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1950s) stable and like more better aligned with

[32:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1950s) stable and like more better aligned with human expectations and by improving L2

[32:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1953s) human expectations and by improving L2

[32:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1953s) human expectations and by improving L2 we increase like the uh like the root

[32:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1955s) we increase like the uh like the root

[32:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1955s) we increase like the uh like the root cause diagnosis as we mentioned earlier.

[32:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1958s) cause diagnosis as we mentioned earlier.

[32:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1958s) cause diagnosis as we mentioned earlier. So we also use like majority voting sort

[32:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1960s) So we also use like majority voting sort

[32:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1960s) So we also use like majority voting sort of and like held out evaluations to like

[32:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1962s) of and like held out evaluations to like

[32:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1962s) of and like held out evaluations to like help mitigate um variance from like

[32:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1966s) help mitigate um variance from like

[32:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1966s) help mitigate um variance from like individual uh LLM judgments and together

[32:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1969s) individual uh LLM judgments and together

[32:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1969s) individual uh LLM judgments and together these improvements support like a more

[32:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1971s) these improvements support like a more

[32:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1971s) these improvements support like a more reliable optimization process.

[32:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1974s) reliable optimization process.

[32:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1974s) reliable optimization process. Um importantly like one of the observed

[32:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1977s) Um importantly like one of the observed

[32:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1977s) Um importantly like one of the observed improvement here was about the required

[32:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1978s) improvement here was about the required

[32:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1978s) improvement here was about the required adoption threshold and the reported

[33:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1980s) adoption threshold and the reported

[33:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1980s) adoption threshold and the reported confidence interval as well. Uh the

[33:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1983s) confidence interval as well. Uh the

[33:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1983s) confidence interval as well. Uh the second level of impact is where we want

[33:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1985s) second level of impact is where we want

[33:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1985s) second level of impact is where we want to measure the RSI where the RSI

[33:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1987s) to measure the RSI where the RSI

[33:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1987s) to measure the RSI where the RSI adoption grows. This includes the

[33:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1989s) adoption grows. This includes the

[33:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1989s) adoption grows. This includes the percentage of like agent prompt changes

[33:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1991s) percentage of like agent prompt changes

[33:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1991s) percentage of like agent prompt changes that can reliably be generated and

[33:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1992s) that can reliably be generated and

[33:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1992s) that can reliably be generated and validated through recursive

[33:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1994s) validated through recursive

[33:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1994s) validated through recursive self-improvements in like downstream

[33:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1996s) self-improvements in like downstream

[33:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1996s) self-improvements in like downstream response quality and also productivity

[33:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1998s) response quality and also productivity

[33:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1998s) response quality and also productivity gains for engineering for engineers like

[33:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2000s) gains for engineering for engineers like

[33:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2000s) gains for engineering for engineers like maintaining these systems. Um those

[33:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2002s) maintaining these systems. Um those

[33:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2002s) maintaining these systems. Um those broader metrics are a part of how we

[33:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2004s) broader metrics are a part of how we

[33:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2004s) broader metrics are a part of how we want to like basically think about

[33:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2006s) want to like basically think about

[33:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2006s) want to like basically think about evaluation evaluating these systems at

[33:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2008s) evaluation evaluating these systems at

[33:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2008s) evaluation evaluating these systems at scale because the ultimate goal isn't to

[33:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2010s) scale because the ultimate goal isn't to

[33:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2010s) scale because the ultimate goal isn't to like maximize the number of like

[33:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2012s) like maximize the number of like

[33:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2012s) like maximize the number of like automated prompt edits. It's to like

[33:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2014s) automated prompt edits. It's to like

[33:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2014s) automated prompt edits. It's to like produce measurable improvements in like

[33:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2016s) produce measurable improvements in like

[33:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2016s) produce measurable improvements in like agent quality while maintaining some

[33:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2018s) agent quality while maintaining some

[33:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2018s) agent quality while maintaining some sort of a confidence that those changes

[33:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2020s) sort of a confidence that those changes

[33:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2020s) sort of a confidence that those changes are like genuinely beneficial. And this

[33:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2023s) are like genuinely beneficial. And this

[33:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2023s) are like genuinely beneficial. And this brings up brings us to like the final

[33:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2025s) brings up brings us to like the final

[33:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2025s) brings up brings us to like the final part of our talk which is how do we

[33:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2027s) part of our talk which is how do we

[33:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2027s) part of our talk which is how do we scale these um capabilities

[33:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2030s) scale these um capabilities

[33:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2030s) scale these um capabilities um which is like through an acceleration

[33:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2032s) um which is like through an acceleration

[33:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2032s) um which is like through an acceleration plan. So so far we focused on like one

[33:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2035s) plan. So so far we focused on like one

[33:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2035s) plan. So so far we focused on like one implementation of RSI for user

[33:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2037s) implementation of RSI for user

[33:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2037s) implementation of RSI for user understanding but like the broader

[33:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2038s) understanding but like the broader

[33:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2038s) understanding but like the broader opportunity is like much larger. If we

[34:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2041s) opportunity is like much larger. If we

[34:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2041s) opportunity is like much larger. If we are able to like um kind of like

[34:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2043s) are able to like um kind of like

[34:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2043s) are able to like um kind of like standardize evaluation, reflection and

[34:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2045s) standardize evaluation, reflection and

[34:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2045s) standardize evaluation, reflection and optimization, we could potentially uh

[34:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2048s) optimization, we could potentially uh

[34:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2048s) optimization, we could potentially uh reuse this loop for like or the cycle

[34:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2050s) reuse this loop for like or the cycle

[34:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2050s) reuse this loop for like or the cycle for like many other agent teams. And um

[34:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2054s) for like many other agent teams. And um

[34:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2054s) for like many other agent teams. And um moving forward like um to understand the

[34:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2057s) moving forward like um to understand the

[34:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2057s) moving forward like um to understand the end toend evals the current state is

[34:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2059s) end toend evals the current state is

[34:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2059s) end toend evals the current state is like this diagram basically shows

[34:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2060s) like this diagram basically shows

[34:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2060s) like this diagram basically shows progression of capabilities that we

[34:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2062s) progression of capabilities that we

[34:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2062s) progression of capabilities that we believe are like necessary to support

[34:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2063s) believe are like necessary to support

[34:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2063s) believe are like necessary to support RSI at scale. At the foundation, we

[34:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2066s) RSI at scale. At the foundation, we

[34:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2066s) RSI at scale. At the foundation, we obviously have like the base agent where

[34:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2068s) obviously have like the base agent where

[34:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2068s) obviously have like the base agent where most teams already have agents that

[34:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2069s) most teams already have agents that

[34:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2069s) most teams already have agents that perform like useful tasks whether that's

[34:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2071s) perform like useful tasks whether that's

[34:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2071s) perform like useful tasks whether that's like question answering, summarizing,

[34:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2073s) like question answering, summarizing,

[34:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2073s) like question answering, summarizing, chat agents um or like just completing

[34:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2075s) chat agents um or like just completing

[34:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2076s) chat agents um or like just completing workflows as such. The next layer is

[34:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2077s) workflows as such. The next layer is

[34:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2077s) workflows as such. The next layer is agent evaluation and this is where we

[34:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2079s) agent evaluation and this is where we

[34:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2079s) agent evaluation and this is where we begin to like see gaps because some

[34:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2081s) begin to like see gaps because some

[34:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2081s) begin to like see gaps because some teams have built their own bespoke

[34:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2083s) teams have built their own bespoke

[34:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2083s) teams have built their own bespoke evaluation workflows. Other rely on like

[34:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2085s) evaluation workflows. Other rely on like

[34:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2085s) evaluation workflows. Other rely on like one-off analysis or like manually

[34:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2087s) one-off analysis or like manually

[34:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2087s) one-off analysis or like manually inspecting outputs. But only a few of

[34:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2090s) inspecting outputs. But only a few of

[34:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2090s) inspecting outputs. But only a few of them have like relatively standardized

[34:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2092s) them have like relatively standardized

[34:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2092s) them have like relatively standardized reliable evaluation processes that can

[34:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2093s) reliable evaluation processes that can

[34:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2094s) reliable evaluation processes that can be continuously reused. And above um

[34:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2098s) be continuously reused. And above um

[34:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2098s) be continuously reused. And above um evaluation we have agentic reflection

[35:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2100s) evaluation we have agentic reflection

[35:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2100s) evaluation we have agentic reflection which means being able to systematically

[35:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2101s) which means being able to systematically

[35:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2101s) which means being able to systematically investigate or do some sort of like root

[35:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2104s) investigate or do some sort of like root

[35:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2104s) investigate or do some sort of like root cause diagnosis. Um and finally at the

[35:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2106s) cause diagnosis. Um and finally at the

[35:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2106s) cause diagnosis. Um and finally at the top we have the RSI loop itself which

[35:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2108s) top we have the RSI loop itself which

[35:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2108s) top we have the RSI loop itself which connects evaluation and reflection to

[35:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2110s) connects evaluation and reflection to

[35:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2110s) connects evaluation and reflection to like optimization and validation. The

[35:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2113s) like optimization and validation. The

[35:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2113s) like optimization and validation. The the important observation here is that

[35:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2115s) the important observation here is that

[35:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2115s) the important observation here is that these layers like they depend on one

[35:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2117s) these layers like they depend on one

[35:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2117s) these layers like they depend on one another and you can't build like a

[35:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2119s) another and you can't build like a

[35:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2119s) another and you can't build like a trustworthy loop without like reliable

[35:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2121s) trustworthy loop without like reliable

[35:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2121s) trustworthy loop without like reliable diagnosis. That's why we see like

[35:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2124s) diagnosis. That's why we see like

[35:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2124s) diagnosis. That's why we see like leveling up evaluation infrastructure as

[35:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2126s) leveling up evaluation infrastructure as

[35:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2126s) leveling up evaluation infrastructure as a prerequisite to like scaling RSI and

[35:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2129s) a prerequisite to like scaling RSI and

[35:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2129s) a prerequisite to like scaling RSI and rather than asking every team to like

[35:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2131s) rather than asking every team to like

[35:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2131s) rather than asking every team to like independently build all these

[35:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2133s) independently build all these

[35:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2133s) independently build all these components, there's an opportunity over

[35:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2134s) components, there's an opportunity over

[35:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2134s) components, there's an opportunity over here to like provide some sort of like a

[35:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2136s) here to like provide some sort of like a

[35:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2136s) here to like provide some sort of like a shared um infrastructure.

[35:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2139s) shared um infrastructure.

[35:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2139s) shared um infrastructure. So uh what would it like basically take

[35:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2141s) So uh what would it like basically take

[35:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2141s) So uh what would it like basically take to make RSI like a reusable capability

[35:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2143s) to make RSI like a reusable capability

[35:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2143s) to make RSI like a reusable capability across an enterprise is like uh we've

[35:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2146s) across an enterprise is like uh we've

[35:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2146s) across an enterprise is like uh we've identified like five strong areas. First

[35:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2148s) identified like five strong areas. First

[35:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2148s) identified like five strong areas. First is like standardizing evaluation

[35:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2150s) is like standardizing evaluation

[35:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2150s) is like standardizing evaluation infrastructure. So we need reusable

[35:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2152s) infrastructure. So we need reusable

[35:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2152s) infrastructure. So we need reusable evaluation frameworks, metrics, judges

[35:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2154s) evaluation frameworks, metrics, judges

[35:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2154s) evaluation frameworks, metrics, judges which can be calibrated with human

[35:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2156s) which can be calibrated with human

[35:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2156s) which can be calibrated with human feedback and um the second is like

[35:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2158s) feedback and um the second is like

[35:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2158s) feedback and um the second is like automating feedback loops. So we want

[36:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2160s) automating feedback loops. So we want

[36:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2160s) automating feedback loops. So we want systems that can continuously collect

[36:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2162s) systems that can continuously collect

[36:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2162s) systems that can continuously collect execution traces, detect recurring

[36:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2164s) execution traces, detect recurring

[36:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2164s) execution traces, detect recurring failures and perform some sort of loop

[36:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2166s) failures and perform some sort of loop

[36:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2166s) failures and perform some sort of loop case analysis.

[36:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2168s) case analysis.

[36:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2168s) case analysis. Um the third one is we need to scale um

[36:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2170s) Um the third one is we need to scale um

[36:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2170s) Um the third one is we need to scale um automated optimization that includes

[36:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2172s) automated optimization that includes

[36:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2172s) automated optimization that includes prompt optimization, agent reflection.

[36:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2175s) prompt optimization, agent reflection.

[36:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2175s) prompt optimization, agent reflection. So instead of making like ad hoc

[36:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2177s) So instead of making like ad hoc

[36:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2177s) So instead of making like ad hoc changes, we want improvement processes

[36:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2179s) changes, we want improvement processes

[36:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2179s) changes, we want improvement processes where candidates where candidate updates

[36:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2181s) where candidates where candidate updates

[36:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2181s) where candidates where candidate updates can be like generated and tested in more

[36:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2183s) can be like generated and tested in more

[36:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2183s) can be like generated and tested in more like a repeatable way. Um the fourth

[36:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2185s) like a repeatable way. Um the fourth

[36:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2185s) like a repeatable way. Um the fourth area is like validation and trust. So

[36:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2187s) area is like validation and trust. So

[36:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2187s) area is like validation and trust. So any automated improvement needs evidence

[36:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2189s) any automated improvement needs evidence

[36:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2189s) any automated improvement needs evidence that it actually you know works which

[36:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2191s) that it actually you know works which

[36:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2192s) that it actually you know works which means like having robust hold out

[36:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2194s) means like having robust hold out

[36:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2194s) means like having robust hold out evaluations and some sort of regression

[36:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2196s) evaluations and some sort of regression

[36:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2196s) evaluations and some sort of regression testing. And finally, we want to like

[36:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2198s) testing. And finally, we want to like

[36:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2198s) testing. And finally, we want to like obviously democratize RSI across teams

[36:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2200s) obviously democratize RSI across teams

[36:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2200s) obviously democratize RSI across teams where the goal is to develop reusable

[36:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2202s) where the goal is to develop reusable

[36:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2202s) where the goal is to develop reusable components and self-service infra so

[36:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2204s) components and self-service infra so

[36:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2204s) components and self-service infra so that teams can adopt these capabilities

[36:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2206s) that teams can adopt these capabilities

[36:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2206s) that teams can adopt these capabilities without having to build an entire RSI

[36:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2209s) without having to build an entire RSI

[36:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2209s) without having to build an entire RSI pipeline themselves which is what our

[36:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2211s) pipeline themselves which is what our

[36:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2211s) pipeline themselves which is what our broader vision is. And I think that this

[36:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2213s) broader vision is. And I think that this

[36:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2214s) broader vision is. And I think that this is an important shift in how we approach

[36:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2215s) is an important shift in how we approach

[36:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2215s) is an important shift in how we approach the engineering of like AI systems.

[36:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2219s) the engineering of like AI systems.

[36:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2219s) the engineering of like AI systems. Um so in recursive self-improvement we

[37:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2222s) Um so in recursive self-improvement we

[37:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2222s) Um so in recursive self-improvement we basically just to summarize we have like

[37:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2224s) basically just to summarize we have like

[37:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2224s) basically just to summarize we have like two key components which is the

[37:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2225s) two key components which is the

[37:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2225s) two key components which is the improvement engine and the target agent.

[37:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2227s) improvement engine and the target agent.

[37:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2227s) improvement engine and the target agent. The target agent in the AI system is

[37:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2229s) The target agent in the AI system is

[37:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2229s) The target agent in the AI system is something we're trying to make better

[37:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2230s) something we're trying to make better

[37:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2230s) something we're trying to make better while the improvement engine is you know

[37:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2232s) while the improvement engine is you know

[37:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2232s) while the improvement engine is you know responsible for driving that change by

[37:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2235s) responsible for driving that change by

[37:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2235s) responsible for driving that change by evaluating performance identifying

[37:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2237s) evaluating performance identifying

[37:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2237s) evaluating performance identifying failures root causes and proposing

[37:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2239s) failures root causes and proposing

[37:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2239s) failures root causes and proposing improvements. But the objective of the

[37:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2241s) improvements. But the objective of the

[37:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2241s) improvements. But the objective of the RSI system isn't simply to improve the

[37:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2243s) RSI system isn't simply to improve the

[37:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2243s) RSI system isn't simply to improve the agents performance at like a given point

[37:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2245s) agents performance at like a given point

[37:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2245s) agents performance at like a given point in time. It's to like accelerate how

[37:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2247s) in time. It's to like accelerate how

[37:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2247s) in time. It's to like accelerate how quickly the agent gets better. And if we

[37:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2250s) quickly the agent gets better. And if we

[37:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2250s) quickly the agent gets better. And if we think of like the agent performance as a

[37:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2252s) think of like the agent performance as a

[37:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2252s) think of like the agent performance as a function of time, we are interested in

[37:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2254s) function of time, we are interested in

[37:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2254s) function of time, we are interested in increasing its first like its first

[37:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2256s) increasing its first like its first

[37:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2256s) increasing its first like its first derivative. If we think about it

[37:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2258s) derivative. If we think about it

[37:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2258s) derivative. If we think about it essentially like making the performance

[37:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2260s) essentially like making the performance

[37:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2260s) essentially like making the performance um improvement curve steeper and in our

[37:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2263s) um improvement curve steeper and in our

[37:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2263s) um improvement curve steeper and in our case like the target agent is our user

[37:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2265s) case like the target agent is our user

[37:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2265s) case like the target agent is our user understanding agent while the

[37:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2266s) understanding agent while the

[37:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2266s) understanding agent while the improvement engine is like the RSI

[37:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2268s) improvement engine is like the RSI

[37:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2268s) improvement engine is like the RSI pipeline which brings together

[37:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2270s) pipeline which brings together

[37:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2270s) pipeline which brings together evaluation, reflection and firmed

[37:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2272s) evaluation, reflection and firmed

[37:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2272s) evaluation, reflection and firmed optimization.

[37:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2274s) optimization.

[37:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2274s) optimization. Um before I close, we want to like

[37:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2276s) Um before I close, we want to like

[37:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2276s) Um before I close, we want to like acknowledge the people who helped make

[37:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2278s) acknowledge the people who helped make

[37:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2278s) acknowledge the people who helped make this work possible. This has been like

[38:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2280s) this work possible. This has been like

[38:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2280s) this work possible. This has been like an collaborative effort across the core

[38:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2282s) an collaborative effort across the core

[38:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2282s) an collaborative effort across the core team, our evaluation team and our

[38:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2284s) team, our evaluation team and our

[38:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2284s) team, our evaluation team and our engineering partners and leadership. Uh

[38:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2286s) engineering partners and leadership. Uh

[38:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2286s) engineering partners and leadership. Uh we we particularly like to thank all the

[38:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2288s) we we particularly like to thank all the

[38:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2288s) we we particularly like to thank all the team members and collaborators listed

[38:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2290s) team members and collaborators listed

[38:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2290s) team members and collaborators listed here for their contributions.

[38:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2292s) here for their contributions.

[38:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2292s) here for their contributions. Um a lot of the work we have described

[38:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2294s) Um a lot of the work we have described

[38:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2294s) Um a lot of the work we have described for evaluation quality to reflection and

[38:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2296s) for evaluation quality to reflection and

[38:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2296s) for evaluation quality to reflection and production required collaborations

[38:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2298s) production required collaborations

[38:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2298s) production required collaborations across these different areas of

[38:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2300s) across these different areas of

[38:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2300s) across these different areas of expertise and we're grateful for the

[38:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2302s) expertise and we're grateful for the

[38:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2302s) expertise and we're grateful for the support and um feedback we have

[38:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2304s) support and um feedback we have

[38:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2304s) support and um feedback we have received. Um

[38:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2307s) received. Um

[38:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2307s) received. Um yep so thank you and uh I guess we just

[38:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2310s) yep so thank you and uh I guess we just

[38:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2310s) yep so thank you and uh I guess we just want to conclude by saying that um most

[38:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2313s) want to conclude by saying that um most

[38:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2313s) want to conclude by saying that um most important part of like improvement

[38:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2315s) important part of like improvement

[38:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2315s) important part of like improvement processes is like being able to you know

[38:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2317s) processes is like being able to you know

[38:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2317s) processes is like being able to you know trust the changes we are making to make

[38:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2319s) trust the changes we are making to make

[38:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2319s) trust the changes we are making to make the system better. So that's some of the

[38:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2321s) the system better. So that's some of the

[38:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2321s) the system better. So that's some of the directions we're exploring with um

[38:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2323s) directions we're exploring with um

[38:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2323s) directions we're exploring with um recursive self-improvement here. So

[38:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2325s) recursive self-improvement here. So

[38:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2325s) recursive self-improvement here. So yeah, thank you so much for listening.