Continual Learning for AI Agents: From Failures to Durable Improvements - Soheil Feizi, RELAI

Humans learn by interacting with the world and receiving feedback. Can an agent do the same—act, discover what went wrong, and improve without forgetting what it already knows? This is the continual-learning problem introduced by Soheil Feizi, who introduces himself as founder and CSO at RELAI and an associate professor of computer science at the University of Maryland.

An agent's world includes diverse users, complex tools, and changing data policies. Learning can affect several parts of that system: the model, through weight updates or model selection; the harness, through prompts, skills, tools, code, and workflows that supply context; and memory, both within a session and across sessions. The unit being improved is the whole agent, not just its language model.

Two questions organize the work. First, how do we know whether the agent did well, and what should it have done instead? Second, how do we act on that feedback—which component should change, and how? A useful learning system needs both a way to judge behavior and a way to select repairs.

During development, the feedback path is relatively straightforward. Run tasks from a curated benchmark, apply its evaluators, and obtain pass/fail grades, rewards, or feedback on behavior. The tasks and scoring rules already exist.

Production often supplies something less structured: a session log. A conversation may suggest that the user is dissatisfied without explicitly identifying the failure or the desired response. Two complementary routes can turn those logs into feedback:

• Automatic analysis: Models, LLMs, or code inspect the log. The agent can sometimes critique its own trajectory. This route offers scale.
• Human expert review: Specialists inspect a smaller selection of sessions and provide domain knowledge about what the agent should have done. Its lower volume does not diminish its role in aligning behavior with the application’s requirements.

A log records one thing that happened. Feedback describes what was wrong with it. Neither, by itself, lets us rerun the interaction with a modified agent and determine whether the repair works. The missing artifact is a replayable learning environment: a simulation with explicit grading for success.

Building that environment means inferring a distribution of relevant interactions from an observation. The original session supplies evidence, but the environment must specify how the surrounding world behaves:

1. Reconstruct tool behavior. Decide which tools can run for real and which need mocks. Supply the data necessary for those mocks to behave appropriately.
2. Infer user behavior. When the task involves a conversation, construct synthetic users that reproduce the relevant intent and interaction patterns.
3. Define success. Infer evaluators that distinguish acceptable behavior from the failure being repaired.

Each step is technically difficult: a plausible simulation is not automatically a faithful one. But when the reconstruction succeeds, the result is executable. Different agent candidates can face the same kinds of scenarios, receive grades, and expose whether a proposed change actually addresses the failure pattern. Feedback becomes something an optimizer can test against.

Once feedback is available, the next decision is where to apply it. Model updates use supervised fine-tuning or reinforcement-learning post-training and typically require substantial compute. Harness engineering can rewrite prompts, learn skills, alter tools, or change code around the LLM; approaches include GEPA and trace-to-harness repair. Memory updates retain facts or learned skills so the agent does not repeat an earlier mistake. The objective is the smallest durable change at the right layer, rather than exclusive commitment to any one update mechanism.

At the model layer, the available methods change different aspects of training:

Feizi groups DPO, GRPO, and RLVR in the broader post-training discussion. Their training procedures differ: DPO’s original formulation uses a direct classification loss over preferences rather than online sampling during fine-tuning. LoRA freezes pretrained weights and trains low-rank adaptation matrices. Feizi describes this restricted update as cheaper and safer; reducing trainable parameters does not itself establish freedom from regressions.

For the learning loop described here, model updates need explicit evaluation infrastructure. A production log and a critique do not yet provide the replayable tasks and evaluators needed to determine whether training improved the relevant behavior. That is the role of the reconstructed learning environment.

At the harness layer, trace-to-harness starts directly from a log and feedback: ask a coding agent to analyze the failure and modify the agent. The difficulty is verification. Without a replayable test, even an apparently sensible patch may not fix the original case, and its effects on previously successful cases remain unknown.

GEPA and scored prompt search make candidate selection explicit. They mutate prompts, score candidates, and retain promising versions through search methods such as evolutionary algorithms. That makes improvement testable, but only when benchmarks and evaluators exist. The distinction is not whether a coding agent can produce a change; it is whether the system can compare that change against alternatives and detect hidden regressions.

At the memory layer, systems such as Letta and Mem0 can store facts and corrections. Skill distillation goes further by compressing a successful trajectory into a reusable procedure. Feizi characterizes memory as the cheapest and fastest update path. It can operate directly on logs and feedback, but writing a correction is not evidence that the correction will be retrieved, resolve the failure, or preserve other behavior. Verification is still necessary.

Feizi calls the resulting discipline verifiable continual learning, or VCL. Every repair must demonstrate improvement while preserving what already worked. Operationally, that promise is scoped to the evaluated environments: passing prior tests supports preservation of those behaviors, not an exhaustive guarantee about every future production interaction.

The verification contract has three parts:

1. Executable test: Turn the failure into a task that can be replayed and graded.
2. Measured delta: Score the agent before and after the proposed update.
3. Regression test: Check that prior tests still pass after the change.

The first practical principle is replayability. A one-off failure must become a repeatable test. The aim is not merely to preserve the original transcript, but to simulate and evaluate similar scenarios and patterns. This is what allows the learning system to move from an observation about one session to evidence about a repair.

The second principle, holisticness, concerns diagnosis. Suppose a support agent cites a stale policy and skips a required escalation. The visible failure does not uniquely identify the component that caused it.

These are alternative diagnoses, not instructions to change every layer. A workflow gate may be the right repair when escalation must precede a refund; correcting memory may suffice when an obsolete fact caused the failure. Root-cause analysis should route the update to the smallest durable change that explains the behavior.

The third principle is lifelongness. Suppose an agent has already been optimized on K learning environments. A new failure becomes environment Eₖ₊₁. Optimizing only for that new environment can improve the latest case while undoing earlier successes.

Regression-aware learning makes preservation a constraint during optimization, rather than treating regression testing only as a final inspection. A compact mathematical statement of that objective is:

Here A is the existing agent, A′ is a candidate update, and S is an environment’s evaluation score. Candidate selection must account for both the recent failure and the behavior already learned.

The growing history creates a computational problem. Feizi calls for an approach whose cost does not grow even linearly with K, because repeatedly revisiting an expanding collection of environments can become prohibitive. This is an efficiency requirement in the talk, not a supplied algorithm or demonstrated complexity bound.

That leads to the fourth principle, efficiency. The learning loop must run frequently, so cost matters both in the repair and in the optimization process. Memory writes may be cheap, prompt or harness changes moderately expensive, and model-weight updates much more expensive. Regression-aware candidate evaluation adds its own cost. Replayability, holisticness, lifelongness, and efficiency therefore constrain the same system: it must generate useful tests, choose the right repair, preserve earlier behavior, and do so often enough to keep learning.

RELAI’s learning loop maps these principles onto an update workflow. Its inputs can be logs, feedback, instructions, or prompts. It first converts those signals into replayable learning environments, then performs root-cause analysis and routes proposed repairs to memory, model, or harness. Regression-aware optimization searches for improvements while preserving prior evaluated behavior. The output is a reviewable version update explaining what changed and why.

Integration begins with a one-time learning-harness setup. Feizi describes support for the user’s own LLM and agents built with major agent frameworks. After setup, the recurring workflow has two operations: create learning environments from the available signals, then invoke the optimizer:

relai optimize

The optimizer’s output is an updated agent delivered as a pull request for review. The demonstrated handoff is a proposed version change, not an automatic deployment.

The demonstration uses a fictional, tool-using support agent in a reproducible continual-learning benchmark. A single source of truth defines interacting policies, and deterministic evaluators grade behavior. The benchmark deliberately includes regression traps: a change that overfits the newest failure can break a task the agent previously handled correctly.

The first scenario starts without a production log. Instead, the input is an instruction to examine how the agent behaves when a caller is rude and adversarial. An interactive environment-creation command generates the simulation’s personas, intent, real or mock tools, and evaluators defining success. The instruction becomes a task the agent can actually face and be graded on.

The initial simulation scores the agent at 87%, with two evaluators showing particularly low scores. Feizi then runs relai optimize with a rollout budget. In the fictional support-agent demonstration, Feizi reports that one optimization loop raises the score from 87% to 97%, a gain of 10 percentage points. The talk does not specify the rollout count, name the two weak evaluators, or explain the averaging behind the reported improvement.

The next example begins with an undesirable production session and targeted feedback: “Keep fast eligible refunds, but do not generalize generosity beyond refund thresholds.” The correction contains both a change request and a preservation requirement. The agent should stop extending generosity beyond policy limits while retaining fast handling of eligible refunds.

The workflow remains the same: convert the session log and feedback into a replayable learning environment, then run relai optimize to address the new issue while preserving performance on earlier environments. Feizi presents repeated, tested updates as a compounding process: each new failure adds an opportunity to improve, and prior learning remains part of the optimization’s constraints.

This changes what it means to say that an agent has learned. A useful update may live in a prompt, workflow, or memory rather than model weights. A production log becomes actionable only when it supports replayable simulation and evaluation. And a new repair counts as durable when the system verifies the old cases alongside the new one. Feizi closes by making VCL available through RELAI, with replayability, holisticness, lifelongness, and efficiency as the requirements for that continuing loop.