How We Solved Context Management in Agents — Sally-Ann DeLucia

What should an agent remember, and what can it safely forget? For Alyx, an AI harness for building AI applications, that question became a practical constraint on the product. Sally-Ann DeLucia introduces herself as Arize’s head of product and a hands-on Alyx contributor, describing nearly a year of work on the agent. At the time of the talk, she describes Alyx as having advanced planning and more than forty skills, spanning prompt optimization, data generation, augmentation, and annotation.

The shift from prompt engineering toward context engineering, illustrated by an Andrej Karpathy post in the presentation, changes the engineering question. Writing good instructions is only part of the job; the system must also choose which information accompanies those instructions. Context management means selecting what the model sees, not merely filling its token window. A useful strategy preserves what the agent needs and removes what it does not.

Alyx operates on Arize’s observability data. A single trace already contains user input, prompts, and metadata; the user’s interaction with Alyx adds still more context. Asking for patterns across traces multiplies that burden. The product therefore needs both selective visibility and a way to work over data too large to show at once. Missing the right evidence produces bad answers, making context selection a product and UX problem as well as an engineering problem.

The team built Alyx using Alyx: if the agent could help them develop their own application, they would have evidence that it could help their users. But analyzing its own trace and span data created a feedback loop:

1. Alyx analyzed a trace containing its previous activity.
2. Growing spans pushed the analysis beyond the context limit.
3. Alyx failed, and the failure became additional recorded data.
4. A retry encountered the larger record and failed again.

The system meant to explain the data was constrained by the size of that same data. Retrying did not remove the cause; it added to it.

Escaping the loop required three related changes: control the active context, separate that context from memory while designing both together, and move heavy work out of the main agent. The first experiments addressed what to keep in the window.

The first approach kept the first 100 characters and discarded the rest. It appeared adequate for simple interactions, but follow-ups exposed what had disappeared. Alyx might answer a question about the most common inputs, then fail to understand a request for more detail about input B. The follow-up depended on a reference established earlier; without that information, it looked like a new conversation. The problem was not simply a shorter answer or less detail. Over-truncation had removed information needed for reasoning across turns.

Next came LLM summarization: compress the context into fewer tokens and pass the summary forward. That sounded like the natural solution, but the team found the retention of important information inconsistent. The summarizer decided what mattered without enough control from the surrounding system.

These were failures of the team’s particular strategies. Their summarization experiment did not establish that summarization is unsuitable for every agent; it showed that uncontrolled compression was not reliable enough for this workload.

The deployed approach combined truncation with retrievable memory. In DeLucia’s description, Alyx retains the first 100 and last 100 characters, removes the middle from active context, and stores the omitted material. These are the excerpt boundaries described in the talk, not a universal input budget. The crucial change is that removal from the model’s current view no longer means permanent loss.

A small TypeScript function makes that boundary operation concrete. It returns the visible excerpt separately from the middle that must be stored; short inputs remain intact.

type ContextExcerpt = {
  visible: string;
  omitted: string | null;
};

function splitContext(text: string): ContextExcerpt {
  const characters = Array.from(text);
  const boundary = 100;

  if (characters.length <= boundary * 2) {
    return { visible: text, omitted: null };
  }

  const head = characters.slice(0, boundary).join("");
  const middle = characters.slice(boundary, -boundary).join("");
  const tail = characters.slice(-boundary).join("");

  return {
    visible: `${head}\n[Middle stored for retrieval]\n${tail}`,
    omitted: middle,
  };
}

This separation expresses the core contract: visible is the material for active context, while omitted is material to preserve outside it. Retrieval must remain possible before the full text can be removed from the active conversation.

The surrounding policy matters as much as the slicing operation. Alyx handles duplicate messages and lengthy tool output by retaining the latest result, preserves the system prompt, and can retrieve earlier tool results or conversation messages when they become relevant. DeLucia reports that this combination had worked well enough to remain unchanged for several months, although the team was beginning to revisit it. Her distinction is precise: “context decides what the model sees, memory decides what survives.” The model does not need every retained fact in every request, but it needs a path back to the facts it later discovers it requires.

Users often keep one chat open while moving between pages in the application. That behavior exposed a weakness that short interactions had hidden: smart truncation could appear successful early in a session, yet Alyx would forget something much later. The team initially discovered these failures through user reports or manual inspection of conversation data.

Their long-session evaluation procedure loads ten conversation turns and tests the eleventh. This turns late-session forgetting into something the team can deliberately exercise instead of waiting for a complaint. The preload establishes the conversation state; the next turn probes whether the agent can still use the context it needs. It is an evaluation procedure, not a claim that eleven turns is the system’s limit or that every such test succeeds. Evaluation checks the consequences of context selection; it does not replace the selection mechanism.

Even with truncation, memory, and evaluations, some tasks still produced too much data for one agent. Search over Arize data made the problem especially clear. A trace can contain hundreds of spans, and locating the relevant evidence may require multiple queries, large results, and intermediate reasoning. Those steps help perform the search, but they do not all need to remain in the user’s main conversation.

The team moved that heavy work into sub-agents. Previously, chat history and data search occupied one agent’s context. After the split, the main agent kept the conversation and light context, delegated the search, and received its result. The search sub-agent held the heavy task data.

The result handoff is the boundary: the main conversation receives what it needs to continue without inheriting the entire search process.

Memory retrieval remains available when more detail is needed. This makes delegation compatible with later follow-ups: the result can be concise without making supporting context permanently inaccessible. DeLucia describes the split as a major improvement and reports that the team subsequently rolled out many sub-agents for data-intensive operations.

Truncation plus memory worked better for this team than its summarization attempt, but very large prompts and inputs could still hit provider limits. Alyx has an unusual source of pressure: other agents’ system prompts, user messages, and conversation histories are themselves the data its users want analyzed. As those applications accumulate more context, Alyx inherits a larger analysis problem. The team keeps returning to sub-agent decomposition, while continuing to investigate how far that strategy can take them.

Longer workflows also raised a separate memory problem. DeLucia observed conversations growing from fewer than ten turns to twenty-plus as users moved across application pages and asked more questions. This is a usage observation, not a measured success rate. At talk time, Alyx’s memory store preserved retrievable conversation context, but it did not provide the long-term memory needed to recall previously discussed issues in a new chat. That capability was still being developed.

Selection within a session remained heuristic too. Keeping the first and last hundred characters does not establish that those are the most useful characters. The team did not yet have a principled context budget or clear metrics for context quality; it used task evaluations as an indirect signal and was researching more sophisticated selection. DeLucia also reports seeing a similar truncation and compression strategy in publicly exposed Claude Code code. That comparison was her observation, not a demonstrated equivalence between the systems.

The resulting engineering practice is iterative: choose context, preserve recoverable memory, and evaluate whether the agent can still do its work. DeLucia closes by emphasizing all three and describing the team’s shift from concentrating on prompts to concentrating on context. Her claim that agents fail because of context captures what repeatedly broke Alyx; it should be read as the lesson of these failures, not an exclusive explanation for every agent failure.

The audience’s final question introduces another constraint: can context management preserve the prompt cache? The questioner points to effort reportedly visible in the exposed Claude Code implementation to avoid invalidating cached context. DeLucia does not claim Alyx has solved that problem. The team had done relatively little cache work and was prioritizing long-term memory.

Her answer does make the existing retrieval mechanism more concrete. Stored material lives in a database with IDs. Alyx has a tool that exposes those IDs, where the material sits in the conversation, message-count information, and a preview. These give the agent enough information to locate and request earlier context without carrying all of it in the active window. They explain retrieval, not a cache-preservation strategy.

More sophisticated handling remained on the roadmap, but current retrieval was working. Long-term memory was receiving priority because that was where DeLucia was hearing the most complaints. The next investment therefore followed the product’s observed failure: users wanted continuity across conversations, beyond the ability to recover omitted material within one.