Improving Agents is a Data Mining Problem

You run an agent. It takes a series of actions and makes some mistakes. Now you have the execution data—but what should you actually change? That is the starting problem for Vivek Trivedy, who leads applied research at LangChain: turning evidence of agent behavior into a repeatable improvement process, drawing on work with customers’ large collections of traces.

The process starts with a working agent, not a perfect one:

1. Ship and run it. Real environments produce feedback that an agent definition alone cannot supply.
2. Collect traces. Store tool calls, output messages, API calls, and CLI interactions so the execution remains available for inspection.
3. Mine and curate the data. As deployments grow, the material may amount to gigabytes or terabytes. Identify the interactions worth learning from.
4. Run experiments. Use those examples to test whether a different prompt, tool, orchestration scheme, or agent loop actually improves behavior.

The experiment closes the loop: a proposed improvement must answer to the evidence collected from previous runs.

Observability and continual learning depend on the same evidence. An agent acts in an environment, its traces record what happened, and those observations can inform changes to its knowledge or instructions. Collecting traces makes that feedback available; integrating it back into the agent is the learning step. Trivedy punctuates the point with a Will Hunting meme, included on a promise to Hamel: look at the data before deciding how to improve the system.

With a conventional Python program, you can read the functions, follow their calls, and often reason through the result. An agent adds prompts, tools, skills, hooks, middleware, and sometimes other agents arranged into a swarm. Reading that configuration does not reliably tell you how a prompt edit will change behavior across many runs. The task domain matters too: the same kind of edit can have different consequences in medicine and law.

This is the practical consequence of trading determinism for autonomy. The agent has more freedom to choose its actions, so understanding it requires tools that expose what it actually did. Traces provide an execution record against which expectations about its behavior can be checked.

At LangChain, traces are centralized in projects, either for an individual agent or across several agents. Other agents then read that material with particular questions in mind. A useful first investigation is to find good and bad interactions: where did a user become upset, and where did the interaction leave them happy? This connects internal execution details to the experience the system is supposed to deliver.

Long runs raise a different question. If an agent works through millions of tokens, does its behavior deteriorate after its first context compaction? What about its second? Or does it preserve the information it needs throughout? The trace is where those questions become inspectable, rather than assumptions about what compaction ought to do.

Traces also support comparisons between model choices. A task run with GPT-5.5 can motivate a counterfactual experiment with GLM-5.2: how would the alternative behave on the same task? Aggregate metrics help compare outcomes, while the traces expose finer differences in the actions and responses users actually see. The model swap is a question to test, not an improvement established merely by selecting another model.

Trivedy expects agent activity to produce a rapidly expanding volume of data. His forecast is that agents will eventually eclipse the data humans have produced over a lifetime on progressively shorter timescales: years, then six months, three months, and perhaps a day. Whether or when that crossover happens, the engineering problem already has two parts: paying to inspect the data and making it accessible to the inspecting agent.

The first limit is input cost. Reading millions of traces becomes expensive when each trace may itself contain millions of tokens. For a single full reading pass, the basic calculation is:

Here, N is the number of traces, T̄ is the average tokens per trace, and P is the price per million input tokens. The same calculation can be made directly in Python:

def trace_reading_cost(
    trace_count: int,
    average_tokens_per_trace: int,
    price_per_million_input_tokens: float,
) -> float:
    total_tokens = trace_count * average_tokens_per_trace
    return total_tokens / 1_000_000 * price_per_million_input_tokens

This estimates input-token charges; generating analyses or repeatedly revisiting traces adds work beyond that single pass.

The second limit is context capacity. A long Claude Code, Codex, or Deep Agents interaction may not fit inside the reviewing agent’s context window at all. Treat the trace as an external object that the agent can query, rather than requiring the whole execution to be loaded into context. Efficient mining therefore needs both an economical reader and a way for that reader to access relevant portions of large traces.

The practical model-selection sequence begins with capability. Trivedy’s team starts with a frontier model such as Opus or GPT-5.5 to determine whether a task is feasible. Once that establishes a working baseline, they inspect the traces and ask whether an open model can perform the same job. The target becomes the minimum capability sufficient for this particular task, rather than the strongest available model for every workload.

Trace judging provides one example. In work with Harvey on its Legal Agent Benchmark (LAB), Trivedy reports roughly matching Opus’s trace-judging capability with open models at one or two orders of magnitude lower cost. The comparison concerns this legal judging task; the talk does not specify the exact model pair, agreement metric, or pricing basis for that range.

Reaching that result involves trying models and engineering their harnesses. Reading how Opus approaches a judgment can reveal guidance a cheaper model needs explicitly in its prompt. Some differences reflect model capability, but some can be addressed by making the expected reasoning process clearer. Trace inspection thus informs the harness changes used to make the less expensive judge useful.

Harness engineering offers a fast feedback loop: change the prompt or surrounding execution system, run evaluations, and inspect the result. Eventually, however, further prompt edits may yield little improvement. That is the point at which Trivedy considers domain-specific fine-tuning.

Customers usually care about a narrower set of tasks than a general-purpose model must cover. Trivedy reports that fine-tuning base models on narrow customer tasks can match or exceed frontier-model performance on those tasks. That is an experience-based claim about specialization, without a single measurement protocol supplied for all such workloads.

The economics can change along with the model. Per-token inference makes each additional reading pass a metered expense; running an inference cluster makes hardware capacity and utilization the relevant costs.

For sufficiently high inference workloads, Trivedy finds the cluster approach cheaper. His description of unlimited inference means avoiding a separate charge for each token while the cluster runs—not unlimited throughput or cost-free computation. The cluster can be spun down when the work is finished.

LangSmith Engine is introduced as a product for automating this trace-mining workflow. Given trace data and something to investigate, it searches the executions, finds issues, and prepares datasets, evaluations, or feedback for human readers. The useful output is not simply an account of what happened: it is material that supports a subsequent improvement step.

One destination is distillation through supervised fine-tuning, or SFT. Suppose GLM-5.2 already performs a task well, but a smaller 9B or 13B model could make it cheaper to serve. Curate the good traces and examples from GLM-5.2, assemble them into a training dataset, and fine-tune the smaller model to imitate that behavior. This is the proposed distillation workflow; the talk does not present a measured outcome for those model sizes.

Another destination is evaluations and environments. Observed interactions supply tasks against which future versions can be tested. Trivedy argues that seeing an agent’s evaluation suite gives a rough picture of its intended behavior: developers repeatedly modify the system to make those tests pass. Choosing the examples is therefore part of choosing what the agent will be optimized to do.

A third destination is human review. Someone still needs to understand the agent’s actions and whether customers are happy, but that person rarely has time to read every trace. Preparing trace-derived content makes oversight manageable, especially in legal and medical work where human review remains necessary. The mining system must produce evidence people can process, as well as data models can learn from.

Trivedy connects this process to scikit-learn and his PhD work on adding learning algorithms to it. At an abstract level, scikit-learn supplies helpers for fitting learning systems to data. Agents change the components and algorithms, but the underlying machine-learning discipline remains recognizable: assemble the data, select a learning procedure, and fit a system to the tasks that matter.

He calls the agent version model–harness–task fit. The model and harness must work together on the target data; choosing either in isolation misses their interaction. The engineering work has two connected responsibilities:

• Find good fit functions. Explore improvement procedures such as autoresearch, reinforcement learning, OPD, OPSD, and SFT.
• Find good data. Supply examples and feedback that represent what the system needs to do.

Those two choices define the practical research problem for a team improving its agents.

Autoresearch makes the feedback loop explicit: perform an action, inspect the result and its traces, then update the system. If there is a score to improve, agents can often search for changes that increase it. But an increasing score is not sufficient evidence of progress: agents may exploit the scoring procedure, so the loop needs checks on how the improvement was obtained.

The Terminal-Bench example turns this into a concrete experiment: have an agent read its own traces, propose experiments, and attempt fixes. The important ingredient is dense feedback. A final pass/fail result tells the agent whether it succeeded, but gives little guidance about what to change after a failure.

The execution trace supplies the missing detail. Instead of receiving only a terminal score, the improving agent can inspect the sequence that produced it and use that evidence to choose a subsequent experiment.

Densifying feedback means making more of the evidence behind the outcome available for diagnosis. The trace is the record that holds it.

When should a team keep engineering the harness, and when should it fine-tune? Trivedy starts with feedback speed. Gather evidence as quickly as possible, whether through human labeling or additional agent runs. Trivedy estimates that a harness-engineering experiment can provide feedback in roughly two minutes. The estimate illustrates a fast iteration cycle, rather than a fixed runtime for every evaluation suite.

The resulting sequence is a sandwich:

1. Engineer the harness first. Use prompt and execution changes to address observed failures. Many teams satisfy their customer use case here.
2. Fine-tune if progress plateaus. Introduce model training when the remaining gap justifies it.
3. Revisit the harness. After tuning, make further harness changes if they help the updated model perform the task.

There is no requirement to advance through every stage. The sequence preserves a fast learning loop while leaving room to change the model when harness edits are no longer enough.

The longer-term question is how an agent incorporates its experience. After performing tasks, it might update its prompts to work more efficiently. If users repeatedly ask for a particular kind of search, it might notify its creator that this need keeps appearing. In both cases, actions in the environment produce information that should change the system’s future behavior. Trivedy leaves the precise form of continual learning open, while identifying three areas that need to evolve.

• Training data. Collect observations from agents taking actions so experience can inform later learning.
• Harness updates. Evolve the execution system with the model and its tasks. Codex, Claude Code, and LangChain’s harnesses reflect both the environments in which models are trained and the work they perform in practice.
• Memory. Maintain useful knowledge over time, rather than treating the agent’s history as an append-only log.

Memory becomes especially consequential when an agent works with someone for years, decades, or a lifetime. A continually growing file plus search does not by itself provide the revision and maintenance that long-lived memory needs.

Sleep-time compute and dreaming describe a possible way to do that maintenance: revisit traces across the agent’s life cycle, revise stored information, and update agent state. The purpose is to turn accumulated activity into more useful memory, not merely to preserve an ever-larger archive.

The immediate starting point is small: enable tracing and give an agent a concrete question to investigate in those traces. Trivedy also points to LangChain’s open-model fine-tuning assistance and model services as ways to support the improvement loop. The larger research challenge remains learning how to manage the volume of data agents produce—and how to integrate those observations back into agent state so the next run benefits from the last one.