An Agent That Fixes Its Own Mistakes - Saloni Gandhi & Ziwei Guo, LinkedIn

A production trace records what LinkedIn’s career agent did on a real request. When something goes wrong, the engineers maintaining the agent still have to decide what to change. Ziwei Guo and Saloni Gandhi built a system around one part of the career experience, the agent that infers a member’s goals and preferences. Their starting point was an evaluator that could inspect the traces and identify failures that mattered to members.

The team calls the process recursive self-improvement, or RSI. The agent logs its output and the steps that produced it. An improvement engine uses those records to evaluate the output, investigate the cause of a failure and propose an edit. Once a validated edit enters production, the revised agent produces the traces for a later round.

Several parts of the agent harness, the software around the base model, can influence an answer. Planning logic chooses an execution path. Context and memory preserve session state. Tool interfaces give the agent access to APIs and datasets, and storage retains outputs and generation logs. The team chose prompt edits as the simplest changes to propose and test while keeping the base model fixed.

Guo describes a progression in what the optimizer might eventually be allowed to edit. It could change how the agent’s context is structured, design the workflow, then modify the harness code that runs it. The most ambitious step would let the optimizer revise its own code. The production work in this talk tackles the first step, changing the instructions.

One iteration has seven steps. The first two produce evidence. Run the agent on a query, keeping its output and execution trace. Then have a judge evaluate the run and explain where the output succeeded or fell short.

Steps three and four turn a judgment into a proposed edit. A reflection agent investigates what could account for the score. Possibilities include a model error, a prompt that left something out, a tool problem or a mistake in evaluation. The diagnosis guides the proposal. This matters even when the only edit the optimizer can make is to a prompt. A low score by itself gives it little reason to change a particular instruction.

The last three steps test the proposal. The system evaluates candidates using the same judge and rubric, incorporates a promising edit into the prompt, then compares the revised agent with the previous version on a held-out set. That final comparison asks whether the change also helps on examples outside those used to choose it. A validated update becomes the baseline for the next iteration.

The user-understanding agent takes a member ID. It fetches profile data, search history, location and preferences, then synthesizes those signals into fields describing the member. One field records which job titles the person appears to prefer. Entries might include network manager or account manager. What would let a reviewer check that inference?

Alongside the titles, the agent supplies its confidence, citations to member evidence and an explanation. If network manager appears frequently in saved searches, a citation can point to the activity behind that inference. The explanation can distinguish the roles of two sources of evidence. A preference the member stated can establish the primary title, while saved searches can support a related title the member might also consider.

The first-level judge, L1, assesses the output across 15 task dimensions using product-defined rubrics. One member’s output received an overall score of 2.31. Underneath it, fields such as goal summary and preferred job title each get a score and notes explaining the verdict. On this scale, 1 means poor, 2 fair and 3 very good. The field-level notes give the next stage a specific claim to investigate when an otherwise reasonable output scores poorly.

Reflection starts by grouping failing rows from members’ traces according to the dimension that scored poorly. This first grouping just uses the existing scores. Within each group, a second-level judge, L2, examines each row and assigns a free-text diagnosis and a cause tag. A shared symptom, such as a low score for preferred titles, can have different causes across members.

Free text creates a counting problem. One diagnosis says the member’s profile data is missing. Another says the profile gives an incomplete picture of the member’s browsing history. In Guo’s example, both identify a problem at the same level of detail for the team’s purposes. The available profile evidence cannot support the agent’s conclusion. Merging these descriptions under a common issue label makes their combined frequency visible. Ranking the labels then requires no further model judgment.

Who uses the diagnoses after a run? Engineers get a scorecard showing which problems recur most often. L2 gets something different on a later run. The system saves prior diagnoses and supplies them as context when L2 investigates fresh failures.

GEPA uses the judge’s evidence and the reflection results to propose changes to the agent’s instructions. It works over mini-batches of sampled traces. Candidate prompts advance to validation, where the team can compare how well they address the observed problems. A prompt that survives the held-out comparison can replace the previous production baseline.

The team used human annotations to improve both L1 and L2 and studied variation across agent runs and model judgments. Meanwhile, engineers packaged the seven steps into a workflow that could suggest edits weekly, every two weeks or monthly, depending on need. Scheduling proposals left the release decision with people. The first GEPA-optimized pull request shipped after quantitative measurement and human review.

Gandhi describes a second outcome from comparing the evaluators’ decisions with human reviews. Disagreements exposed recurring faults in both the scoring rules and the system that explained failures. That raised two separate questions. Could L1 recognize a poor answer? And when L1 flagged one, could L2 identify what had gone wrong?

Where was L1 reliable enough to guide an edit? The team analyzed about 180–200 human reviews, grouping them first by the score L1 had assigned. Full agreement was roughly 55% for outputs scored 1, 57% for those scored 2 and 80% for those scored 3. The judge recognized strong work more reliably than it distinguished degrees of weak work. Those low scores were precisely the ones that sent examples into reflection. Score agreement in the recording, 25:31

Grouping the reviews by field reveals a different source of disagreement. Preferred job titles had among the weakest agreement. For domain or area of expertise, agreement was about 47%, and for work type it was about 50%. The two breakdowns point to different questions for the rubric. What distinguishes a 1 from a 2? What evidence should count for this particular field?

The annotations put concrete cases behind those figures.

• Preferred titles. L1 could accept a plausible title even when the agent had overlooked an important signal of the member’s intent. A reviewer checking the input could spot a missing preference that a check of the returned titles alone would miss.
• Fields of expertise. In the team’s example, software engineering is a domain, machine learning engineer is a role and machine learning is a skill. The rubric blurred those categories. It also gave little guidance on how much credit to award when an answer mixed correct and incorrect entries.
• Formatting and inference. The judge sometimes penalized harmless differences in how a value was written. A harder disagreement arose when the agent inferred a preference from current employment and the member had never stated that preference explicitly. Reviewers needed a rule for when that inference was justified.

Of the 37 judgments reviewers identified as too harsh, 31 involved an output L1 had scored 1. That concentration made partial credit a practical priority. A rule for recognizing the correct parts of a mixed answer could keep that answer from being treated as a complete failure.

Take the work-type value full-time. Capitalization alone could cost the agent credit. The revised rubric accepted formatting variants with the same meaning. The team also added guidance for normalizing job titles, handling concurrent employment and assessing inferences about seniority and geography. It revised the scoring rules to distinguish a poor answer from a partially correct one.

After the rubric changes, reported L1 agreement with human reviewers rose from about 57% to 64%. Mean error also fell, though the scale of that measure was unspecified. The agreement gain measures a change in the evaluator. Engineers had made the product’s definition of a correct judgment more explicit.

Reviewing L2 exposed a different problem. Human reviewers agreed more often when L2 attributed a failure to the underlying model. Agreement was much lower when it blamed the rubric. The original list of cause tags gave L2 no way to say that L1 had made a mistake. Faced with a reasonable rule that L1 had applied incorrectly, L2 sometimes labeled the rule itself defective.

L2 also sometimes repeated L1’s rationale without checking it against the agent’s output. A single mistaken judgment could then look like a finding that two evaluators agreed on. If GEPA acted on that diagnosis, it could propose an edit to a component that had behaved correctly. Improving L2’s ability to identify the cause became a priority in its own right.

For decisions about individual edits, the team used majority voting to reduce the influence of any one model judgment and held-out evaluations to check whether gains carried over. Gandhi also reports an improvement involving the threshold for adopting an edit and the reported confidence interval, without specifying a value or direction of change for either measure.

At the scale of a team, adoption poses a different question. What fraction of its prompt changes can RSI reliably generate and validate? LinkedIn also wants to measure downstream response quality and the time engineers save maintaining an agent. These are proposed measures for broader use. A count of edits would rise even if the quality of the answers stayed flat.

Many teams already have an agent that answers questions, summarizes material or completes a workflow. Their ability to evaluate it varies. Some have bespoke tests, others inspect outputs by hand, and relatively few have a reliable evaluation they can run repeatedly. That gap limits what a shared optimizer could do for them. To propose a targeted fix, it needs a diagnosis. To investigate the cause, the diagnostic system needs a dependable way to identify a failure in the first place.

LinkedIn’s proposed shared infrastructure would give teams five capabilities.

• Evaluation. Reuse evaluation frameworks and metrics, with judges that teams can calibrate against human feedback.
• Feedback collection. Collect execution traces continuously so recurring failures can be detected and investigated.
• Optimization. Use reflection and prompt optimization to generate and test candidate changes in a repeatable way.
• Validation. Compare candidates on held-out examples and run regression tests for behavior an edit might damage.
• Self-service adoption. Package the components so a team with an existing agent can use them without building the whole pipeline.

Gandhi frames the longer-term goal in terms of a slope. The target agent infers member preferences. The improvement engine finds failures, investigates them and tests changes. If agent quality is a function of time, an individual release can raise quality at one point on the curve. The ambition for the engine is to increase the rate of improvement across releases. For that slope to mean anything, the team has to be able to trust that a measured gain reflects a better answer for the member.