We Kept Rebuilding Our AI Agent. Here's What Survived - Fuad Ali, Arize AI
Read the talk
We Kept Rebuilding Our AI Agent. Here’s What Survived
Fuad Ali follows Arize’s Alyx from a late-2023 prototype to one agent with on-demand skills, explaining why planning, stopping, context management and regression testing became problems for the harness to solve.
From a talk by Fuad Ali
At a glance
Ideas worth remembering
Harness-owned plan state reduced redundant to-do bookkeeping. Alyx presents the plan on every model call and was testing removal of the to-do tool entirely.
Completion needs an explicit check: Alyx can reject a finish call while plan items remain, warn before the iteration limit, and force a bounded stop.
Oversized tool results, growing history and cross-session memory need different mechanisms: selective file queries, replacement checkpoints, and a per-user Markdown file capped at 8,000 characters.
Alyx replaced ten page-specific specialists with one agent that loads skills and tools on demand. Narrow features such as filter generation may need only one model call.
Production sessions supply regression fixtures because an okay status can hide repeated calls or excessive latency. Human labels calibrate the judges, and people still choose the speed–quality tradeoff.
From a useful-but-unreliable prototype to planned work
Arize’s first prototype of its in-product agent was, in one staff engineer’s recollection, about half hallucination and half something useful. Fuad Ali, a product manager at Arize, uses that late-2023 starting point to explain the rebuilds behind Alyx. The estimate is an informal description of the prototype, not a measured accuracy result.
The setting is appropriately recursive: Alyx helps debug Alyx, while Signal, a separate trace-reading agent, examines the traces Alyx produces. “Agents all the way down.” Keeping those roles separate matters: Alyx is the user-facing AI engineering agent; Signal helps find problems in its execution.
Inside Arize AX, Alyx can read execution traces, turn findings into evaluations and datasets, run experiments, generate synthetic data from traces, and carry out multistep tasks. That range makes it more than a chat window over documentation. It needs access to product data and tools, a way to keep track of unfinished work, and enough context to choose what to do next.
Prompt optimization supplies a concrete before-and-after example. The earlier product could retrieve prompts stored in Arize and accept a request to optimize them. In the later interface, Ali asks Alyx to optimize its own system prompt, and Alyx creates a plan that includes checking resources around the application. What changes between the two interfaces? The important distinction is the visible move from retrieving a prompt to organizing an investigation around it—not evidence that this particular request produced a better final prompt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The rebuilds happened around the model
One early correction was to move math out of the model and into code. The late-2023 models were unreliable at computation, so the application’s harness—the code that runs the model, tools and surrounding state—took responsibility for it. That division survived the subsequent rebuilds: code computes; the model interprets and explains.
The first official product, then called Copilot, routed requests through a decision tree over a set of skills. An August 2025 rewrite introduced iterative tool calling and a to-do tool, allowing work to continue across multiple model calls. Alyx 2.0 relaunched in February 2026 with a new default model. In June 2026, the team merged specialist agents into one Alyx that loads skills on demand. Long-term user memory followed, written in the background.
These changes did not involve training the underlying models. Arize wrote its own agent loop, plan handling, stopping rule and memory management rather than using an agent framework. Better models opened up new options, but the team still had to decide how to use them. Owning the harness gave it control over those decisions—and responsibility for discovering where its assumptions broke.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move plan bookkeeping out of the model’s way
Planning began with an apparently reasonable tool: give the model a to-do list and let it check items off. Ali compares the choice to assigning chores. You can hand someone the whole list and ask them to maintain it, or hold the list yourself and keep supplying the next task. Alyx initially used the first arrangement.
The model became too enthusiastic about maintaining the list. Calls went into checking off tasks rather than doing useful work, and after several tool calls it could lose track of its focus. More than 20 iterations on the to-do tools eventually pushed the design toward harness-owned state.
The correction happened in stages. First, an in progress status made the current task explicit. Then the harness began rendering the plan into a message on every model call, with the rules beside the state they governed. The next call no longer depended on the model reconstructing its position from a growing history of tool interactions.
Once the plan was kept in ranked order, the extra status became redundant: the next unfinished item already identified the next task. Maintaining in progress was creating two calls for bookkeeping that the ordering could express, so the team removed it. At the time of the talk, it was testing the further step of removing the to-do tool entirely, keeping plan state and completion checks in the harness and presenting them inline in messages.
The result is a different division of labor. The harness remembers what remains and checks completion; the model consumes that state and does the next piece of work. A plan can help an agent stay focused, but making the model continually administer that plan can consume the calls the plan was meant to organize.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A text response does not necessarily mean “done”
The original stopping rule treated a model response without tool calls as the end of the turn. That assumption failed when a model announced its next action in one response and made the tool call in the next. The harness could stop after the announcement, before the promised work happened.
This was more than a model-upgrade nuisance. Alyx lets customers choose their own provider, including customers committed exclusively to Anthropic or OpenAI. Arize benchmarks models and runs regressions, but cannot guarantee that every customer-selected model has been tested. A stopping rule based on one model’s conversational habits was too fragile for that flexibility.
Finishing therefore became an explicit call that the harness checks. Three controls address different failure modes:
- Premature completion: If plan items remain open, the harness can bounce a finish call back up to three times.
- Approaching the budget: A warning arrives one step before the iteration limit.
- Failure to stop: A forced finish provides a final bound on execution.
The completion check gives unfinished work another chance; the iteration limit prevents that chance from turning into an unbounded loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Treat a huge result like a file
Context management contains several different problems. Ali’s analogy is a desk covered in books: one book can be too long to read, and enough individually manageable books can still fill the desk. For Alyx, a single tool result can overwhelm the context window, while the accumulated history of its own work can exhaust the budget over time.
Trace analysis makes the first problem concrete. Alyx may be reading another agent’s trace, containing a large JSON result and a great deal of nested activity. Dumping that blob into context can leave little room for useful work. The replacement is the book-on-a-shelf arrangement: keep the large result available elsewhere and leave a compact indication of its contents and location in the working context.
For JSON, the harness truncates within the result and lets the agent use code, including jq, to retrieve what it needs from the full data. The model’s ability to write queries becomes a way to inspect a large result selectively. How can Alyx inspect the full trace without reading it all at once? The two paths below separate what enters context immediately from what remains available for a later query.
The pointer preserves access; truncation reduces the immediate context load. This is the practical meaning of “treat a huge result like a file and just grep it”: make the data searchable rather than assuming it must all be present in the model’s current input.
Growing execution history needs a separate correction. When Alyx’s trajectory exceeds a window, the harness makes the model write a checkpoint, then drops the history that checkpoint replaces. A fixed token budget limits retained history. Selective file access handles oversized tool data; checkpointing handles accumulated work. Neither is the same as remembering a user across sessions.
A large trace has an immediate path and an on-demand path
Read the diagram as text
- Large trace result. A JSON blob can overwhelm the model’s context window.
- Full result kept available. on-demand — The book stays on the shelf rather than on the desk.
- Compact result in context. immediate — Truncation and a reference reduce what the model reads immediately.
- Alyx model call. Uses the compact result and requests more data when needed.
- Large trace result → Full result kept available: Retain for later access.
- Large trace result → Compact result in context: Truncate for context.
- Compact result in context → Alyx model call: Supply compact information.
- Alyx model call → Full result kept available: Query with code such as jq.
- Full result kept available → Alyx model call: Return requested data.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One 8,000-character file per user
For cross-session memory, Arize tried vector retrieval, episodic and semantic stores, and temporal knowledge graphs. It ultimately chose one Markdown file per user, capped at 8,000 characters. Ali describes the small file as beating the more elaborate alternatives for Alyx; the comparison is a product-design judgment here, without a reported benchmark or numerical quality margin.
The read and write schedules are deliberately different. Every full request loads the file. After a session finishes, a background job edits it. Memory maintenance therefore does not require the interactive agent to run a retrieval-and-update process at every step, and the size cap bounds how much persistent memory each request introduces into context.
The small artifact also makes review practical. A human labeler can inspect a user’s file, decide whether the right information was retained, and adjust the instructions used by the background writer. Concentrating writes in one end-of-session job gives the team a focused place to enforce user separation, including checking that another user’s information does not enter the file.
The appeal is specific: a bounded amount of user history, loaded consistently, with an inspectable writer and an artifact people can audit. It also expresses a limit. Alyx’s persistent memory must fit into that small file; this design is not an attempt to preserve every detail of every session. For this product, the team preferred that constraint to the machinery of its earlier memory systems.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ten specialists became one agent—and some tasks need no loop
The agent topology followed another expansion and rollback. Alyx began with one trace-analysis agent in 2025, grew to ten specialists, and assigned different specialists to different product pages. On the dataset page, for example, a specialist could inspect datasets and generate synthetic data.
The replacement was one agent that loads skill instructions and tools on demand, then decides which tools to use. The dataset capability remains available without requiring a separate dataset agent as the organizing unit. Ali credits both improving model capability and a desire to simplify; the rollback does not establish that specialists are always the wrong architecture.
Model selection follows the same task-specific reasoning. Arize generally defaults to “fast and good enough,” investing heavily in the harness rather than automatically switching to the newest model. Leading a general benchmark and being the best default for Alyx are different questions. The team does upgrade defaults, but treats that as a product decision to test.
At the time of the talk, the team was exploring routing models by task and training or post-training small models for narrow features. The “filter with AI” feature illustrates why: generating a filter can be simple enough for a very small model and a single call. Giving that feature the entire planning-and-tool loop may add machinery it does not need. Simplification can mean consolidating agents, but it can also mean declining to use an agent loop at all.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Production sessions become living regression fixtures
The debugging example returns to the recursive setup: Alyx’s own requests produce traces inside Arize, in a project still named Copilot Prod. Where does a user interaction become inspectable engineering evidence? In the live trace view, the in-product request and the record of the agent’s execution meet. The agent helping investigate a system is itself a system to investigate.
Those production records supply regression cases. Useful user queries and newly discovered failures enter datasets that grow as behavior changes. Session datasets can retain an entire session and let the team replay a particular turn through the agent. That provides a way to test a decision made partway through an interaction, rather than reducing every failure to a fresh opening prompt.
Arize Experiments then supports A/B comparisons of harness changes. The reason to retain old cases is blunt: “the fix can also be the next bug.” A change that repairs today’s failure may break behavior that previously worked, so the growing dataset becomes a record of obligations the next harness version must still satisfy.
A successful status alone is a weak definition of agent health. Two examples show what can hide behind it:
- Repeated work: A turn with 43 identical tool calls may report an okay status while plainly wasting execution.
- Excessive waiting: A response taking 227 seconds can be unacceptable even without an explicit error. Ali regards that wait as too long; the appropriate latency threshold still depends on the use case.
These are behavioral failures that require looking inside the trajectory, not merely checking whether a request returned.
Signal automates more of this trace review, reducing the time staff engineers spend reading executions by hand. Automation does not settle whether its judgments are right. Arize still aligns evaluators with human assessments, checking that judges reflect what its engineers consider good and bad behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
People still define the baseline and choose the tradeoff
Human work remains concentrated in defining the target. Gold labels and golden datasets for judges come from people. An evaluator needs examples of the judgments it is supposed to reproduce; an automated verdict cannot supply its own independent standard of correctness.
Speed versus quality is another decision the metrics inform without resolving. A model change might increase latency by one amount and improve response quality by another. Measuring both makes the choice clearer, but does not determine how much waiting a particular workflow should tolerate or which quality losses are acceptable. Those remain human product judgments.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What survived every rebuild
The ending distinguishes the machinery that changed from six commitments the team kept. Some have a detailed mechanism behind them; others remain concise design requirements:
- Trace everything into the team’s own product. Alyx’s execution stays available for investigation rather than disappearing behind its final response.
- Keep math and data work in code. Code computes aggregations, exposes them through tools, and lets the model read and explain the results.
- Keep golden datasets in CI. Preserve known expectations as part of checking subsequent changes.
- Return errors with an obvious fix. Make recoverable problems actionable.
- Keep streaming and persistence. These remain requirements across the rewrites.
- Keep skills. Names such as routers and tools changed, but the lasting purpose was to give the model a capability around an external system.
The surviving questions are concrete enough to use in a design review: Who holds the plan? What permits a turn to end? Where does an oversized result live? Who writes persistent memory? Does the task need several agents, one agent, or one model call? Which model is fast enough and good enough? And who checks the agent’s own fixes? Alyx’s answers changed as the team encountered failures; tracing, code-based computation, regression cases and human judgment made those changes inspectable.
“Every day we use Alyx to build Alyx” is the closing tagline. Its practical substance is that the product’s own sessions become material for debugging and testing the next version. The agent can help investigate its failures, while people still supply the standards by which an apparent improvement earns its place.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
A practical starting point for using the in-product AI engineering agent discussed in the recording.
Ali and coauthors explain how to inspect tool calls, retries, latency and outputs, then compare workflow changes using collected traces.
The open-source tracing project linked from the companion article, useful for applying trace-based investigation to coding-agent workflows.
Related talks
- Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI
Develops the verification side with a broken shopping-assistant price filter, trace-and-code investigation, replayed failures and human review of proposed fixes.
- Break It 'Til You Make It: Building the Self-Improving Stack for AI Agents
Expands the production-evaluation approach to tool calls, trajectories and multi-turn context, including the need to improve evaluators alongside agents.
Read the complete timestamped transcript
- 0:12
Uh, what's up, guys? Thank you guys for coming to this talk. I know we're getting to the end of the day. You had lunch, you're getting tired. So really appreciate the patience. You've been hearing a lot from all of us at Arize. You just heard Sophia's talk, maybe if you were in the room earlier. So thank you again. My name is Fuad. Uh, did a talk yesterday, if you weren't here. Uh, I work on product at Arize, so I'm what they call an AI PM, which I'll get into a little bit about, you know, kind of how that relates to shipping agents. Um, today I wanna talk about, uh, two years of shipping our
- 0:42
in-product production agent, uh, and some of the lessons we've learned. Hopefully, there are good takeaways for you guys as you are kinda building your own agents in the wild. Cool. So before I start, we're gonna get a little bit meta here. Uh, and hopefully, you guys are all kinda following along. But, you know, this slide was made with an agent. Yes, it was. Um, we have coding agents, um, that kind of operate with Alyx, with Signal, with other parts of our product that are themselves agents. Uh, we
- 1:12
have Signal, which is kind of the trace reading agent that reads our agent's traces, right? So we have an agent reading the agent traces. We have Alyx, which is the in-product agent. You can think of it like kinda like Cursor for AI engineering. That's the agent I'm talking about today. Uh, we actually use Alyx to debug Alyx, um, so it gets a little bit meta. Um, and then hopefully you get, you know, kind of an idea of how Alyx could help you evaluate your agent and some of the design patterns to be taken from Alyx and how that could be translated to your agent.
- 1:42
So anyways, turtles all the way down, agents all the way down. Uh, and if you are confused about which agent we're talking about, I will make sure to make that clear. So, um, what we're gonna cover is kind of like how Alyx got here. I've got some nice throwback screenshots from 2023, which feels like ages ago, about, uh, of how Alyx used to look, uh, where it used to be just a router over skills, uh, tool calling loop, um, and kind of how we've changed that. Uh, some of the hardest
- 2:11
decisions we made along the way, so specifically who owns the plan, how to end a turn, uh, how we dealt with memory across the agent, uh, should we use one agent, should we use many sub-agents, uh, wha-what default model to choose, and sort of how we test it as well. Um, then, like I mentioned, it gets a bit meta, but we debug Alyx with Alyx. So, uh, and now we have Signal as well, so how Signal is finding Alyx's retry loops and some of the real issues we found with our agent using Signal. Uh, and
- 2:41
then i-importantly, what, what still needs a person as well, and so where the human value kind of adds judgment and the AI PM world kinda comes in, uh, and some of the decisions we kept, um, that have lasted throughout the architecture of Alyx. Um, Alyx is the agent inside Arize AX. Alyx can read your traces, turn findings into tests, so building your evals for you, datasets, experiments. It can plan work. It can, uh, run multi-step, step tasks. It can create synthetic
- 3:11
data for you based on your traces. Um, and it can do a lot of different things. I would say, yeah, this is a quote from one of our staff engineers. Uh, on the first prototype back in late 2023 with like 3.5-era models, it was about fifty percent hallucination, fifty percent something useful. Proud to say we're getting a bit better, um, as models have gotten better as well. Uh, this is just a little before and after of the, uh, visuals, just to give you an idea. So you can give it, you know, prompts that you've stored in Arize. You can ask it to
- 3:41
optimize those prompts. Um, and obviously it had, you know, kind of tools to read prompts from our database, uh, and things like that. Now, uh, we have, you know, much more in-depth flow with planning, uh, with much more beautiful UI. Alyx kind of explains what it's doing. Uh, it'll reason through tasks, uh, much more strongly. So here I have like a request to optimize Alyx's own system prompt, and it actually makes a plan to check, you know, kind of resources around the application as well. Um, so yeah, this window shows up in Arize. You can
- 4:11
use it to debug your traces. Um, how did we get here? So quick kinda like timeline. We started the prototype in late 2023. Um, this was like with 3.5-era models. Um, we then, um, you know, very quickly decided to move a lot of math out of the model and into the harness code because if you remember asking agents to do math back then, I mean, nowadays they're solving every Stokes level problems, but, uh, back then, they were not very good at math.
- 4:41
Um, then we kind of had the official launch. We called it the Copilot back then when things were still called Copilots. Uh, a router over a set of skills, a decision tree. We had some rails. Uh, we rewrote a lot of it in August 2025. Um, iterative tool calling and a to-do tool come in, um, so the agent can actually kinda plan what it's doing. Uh, Alyx 2.0 was our big relaunch in February 2026. Uh, Sonnet 4.5 became the default model, and we'll talk through some of how we
- 5:11
test that model as well. Um, one Alyx was just released in June. So instead of specialist, uh, agents, we merged it into kind of like one agent that just loads skills on demand. Uh, and finally, memory. Um, we now have long-term memory, so as the user kind of interacts with Alyx throughout the course of their session, uh, they are kind of continuously, um, uh, the agent continuously knows, uh, sort of the history of that user, uh, and that's written in the background. Um, and
- 5:41
obviously we didn't train these models. Everything we rebuilt was harness code, which I think, um, actually gives me a bit of faith in the world of engineering that there's still very interesting problems to solve here, and it's not all up to the model, and you can't just run a frontier model and expect it to do everything for you. Uh, models got better on their own schedule for sure, and we made some decisions around that, but Pretty much everything we rebuilt was that harness code. Uh, we don't use an agent framework. Um, the loop, the plan, the stopping rule, and the memory are all
- 6:11
ours. Um, what that means is we have a lot of control over it, um, and, you know, we get to learn a lot along the way as well.
- 6:19
Uh, so first one, planning and to-do lists. Um, the analogy, I, I have a bunch of stuff on the slides, but the analogy I kind of think about around, you know, planning is like, uh, let's say you... I don't, I don't have kids. Maybe some of you guys have kids, and you can kind of relate to this analogy. But let's say you want, you know, your kid to do some chores. Um, you can give it a list of chores and ask it to, uh, or ask them to check off the lists of chores as they go. Um, you can check in with
- 6:49
them continuously and hold the list yourself and not tell them about the list except for the next task, right? So that's one paradigm. Um, and there are trade-offs to kind of like each, uh, scenario. We initially just gave the model a to-do list, and what we saw was it spent most of its time checking off the to-do list. Uh, so a lot of LLM calls for, um, pretty much nothing, like no useful unit of work other than updating a to-do list. It als- we also realized
- 7:19
it tended to lose focus after multiple tool calls. Um, and yeah, 20, 20-plus iterations on the to-do tools themselves. So we added this as an extra tool in its arsenal. And, uh, we eventually moved the plan into the harness one piece at a time. So what I mean by that is, first thing we did was we added an in progress so the, uh, the model wouldn't lose track of where it was. Um, then we decided to put the plan as a message, so the harness renders the plan to the model call every
- 7:49
time, um, the model gets a new call and the rules are next to the data, so it's a little bit easier for the model to interpret where it's at. Uh, we actually ended up dropping the in progress identifier because we realized we're just making two calls for effectively, you know, anything that isn't finished is in, in a stack ranked order is the next thing to do, so we removed that. Um, and we're actually testing now without a to-do tool at all, uh, just keeping everything in line in the messages, uh, completely. So the harness is keeping the state and the completion check,
- 8:19
and the model just consumes that and goes and does the next thing. Um, so maybe some lessons for you guys as you're building to-dos, uh, models can be very eager with, uh, lists. Um, then kind of like what ends the turn, uh, finishing became a gate the harness enforces. Um, so what we at first tried was, you know, kinda text without tool calls, uh, meant done. That broke on models that announce a step in one turn and
- 8:49
make the call in the next, which, um, you know, as you change models, you'll, you'll quickly find some of these breaking things. And one of the things we do in Alyx is we actually allow users to choose their own provider, uh, so we can't necessarily guarantee that we've tested every single model someone will try with Alyx. So a lot of, um, you know, some of the decisions we made here are to guard against those. And yes, we run regression tests, and we check against a bunch of different models, and we benchmark different models on Alyx. We have an opinion on a default, but
- 9:19
you know, obviously, some of our customers are exclusive Anthropic customers or exclusive OpenAI customers. So to actually empower them to use their own models, we kinda had to design against everything that was available, right? Um, so now we have an explicit finish call that bounces back, uh, when plan items are open up to three times, uh, a warning set one step before the iteration limit and then a force finish as well. Um, memory, this is a good one. Um, there are actually a number of problems here, and I'm-- I, I like analogies, so I'm gonna use
- 9:49
another analogy here. It's like, you know, you're, you're working at your desk, um, and a big book lands on your desk, and maybe one big book is fine. You can open the book, kind of search through the book. Uh, if the book gets too long, then it's kinda useless now 'cause you're gonna spend your entire time reading that book. That's one issue. Uh, and that's sort of like, you know, one big tool result comes back, and it's too big for you to read, right? Um, but you can imagine there's actually multiple books on your desk, right? So eventually, not
- 10:19
just are the books too long for you to read on your desk, but there's no space left on your desk, right? You just have too many books, right? So there's a, you know, you know, in this analogy, um, there's a lot of ways you can approach this, right? You can decide to store that book and just have a pointer to the book, right? So that's kind of what we did with the tool results where, okay, I now have a sticky note on my desk that says where the book is and what, you know, kind of high level the book contains. And so if I need to go
- 10:49
read that book, I'll just reference my sticky note and then go find the book on the shelf. My desk clears up a little bit, right? So that's kind of what we do with tool calls, uh, and tool results. For example, like Alyx's, uh, you know, can help you with trace analysis. Some of these traces are other agents' traces, and they can have a shit ton of stuff in the trace, right? So, um, if it's a big JSON blob, for example, that just blows up our context window, and then you can pretty much do no useful work. So instead, what we do is we truncate within, uh, the JSON, and we just let the
- 11:19
agent code to go find it if it needs the full context with jq. Um, one thing the agents are very, very good at is writing code and using code-like tools to, uh, kind of infer things. And so that's kind of what we try to leverage with that. Um, the agent's own trajectory also going past a, a certain, uh, window. Um, so what we now have is the harness makes the model write a checkpoint, uh, and then drops what it replaced. Uh, we have fixed token budget for
- 11:48
history, um, and across sessions, so I kinda got to like the long-term memory piece. Uh, basically, we just ask them a background job to edit one small file per user. I'll dive a little bit more into like why we decided to do this with, with long-term memory, uh, later in the presentation, but essentially that's how it works. Um- Kind of talked through this. So treat a huge result like a file and just grep it. Uh, don't dump it all into context. Don't worry about doing anything fancy.
- 12:19
Uh, for long-term memory, a small file actually beat a knowledge graph, and we, we tried things like vector retrieval, episodic and semantic stores, uh, temporal knowledge graphs, et cetera, et cetera. What we eventually landed on was we just have one file per user. It's a markdown file. It's capped at eight thousand characters. Um, and, you know, I, I think there, there is a lesson here that, you know, as engineers, um, we like to build things, and it's really fun to build things, and sometimes we get in our own way, and we
- 12:48
over-engineer. Um, and this file is just loading in on every full request. Uh, it's written into in a background job after the session finishes. It's small, it's exact, it's easy to load in, it doesn't blow up context, and it's very easy for a human to audit as well. Uh, so you can actually kind of have a human labeler go look at these memory files and, and determine whether the right things are being stored and kind of edit instructions for the background job, so. Um, and then you also only have one
- 13:18
job at the very end, which, uh, you can make very safe as well. Uh, so you're, you're making sure you're not writing memory from other users into the file. Um, so yeah, very good lesson on not over-engineering, uh, your problems. Um, briefly on specialists, um, you know, we had one trace agent that initially just did trace analysis back in twenty twenty-five. Then we had ten specialists, so every page kinda had its own specialist. If you're on the dataset page, there's, you know, a special sub-agent that can
- 13:48
generate synthetic data for you and look, look and introspect on all, all your datasets. Uh, we ended up kind of rolling that back as well and just using one agent where the agent can load skill instructions on demand and tools on demand, and we just let the agent decide what tools to do, right? So it is a bit of an evolution also in, in model capability, but I think simplification as well. Um, we-- O-on model capability, we also kinda stand by that the newest
- 14:18
model isn't always the best by default. Um, there are times where we upgrade the model by default, right? And so we have a perspective on that. Um, I think we have, um, you know, uh, what is the best model is a very subjective question. Uh, what is the best model on a generalized frontier is very different from what may be best for you guys in what you're building or what we're building. Uh, we generally just default to fast and good enough and spend a lot of time on the harness instead. Um, we are now
- 14:48
starting to explore auto-routing, um, so choosing models per task and training our own models or small post-training SLMs, things like that, on very narrow surfaces. Um, and some of these services actually don't even need the loop at all. You can just make a single model call. So an example of that is-- I wonder if I can pull it up. We have this, like, you know, filter with AI. Um, we've come a long way from SQL generation in the, in the past, and it's actually very, very easy to generate filters with very, very small models now. So,
- 15:18
um, that's an example where maybe we don't need the entire loop and we're just over-engineering again. Um,
- 15:26
cool. So that's a good takeaway. Um, production sessions becoming our test. So this is where I'll talk a little bit more about Arize as a platform. Uh, tracing everything is really important, right? So we trace-- Actually, if you look at this, these are the traces from this agent. Uh, so it's very meta here. Um, but every time you ask Alyx a quest-question here, the, the traces, these are the traces from Alyx. We, we still call it Copilot Prod. Um, so those are our live traces coming in from the tool. Um, this
- 15:56
makes it very easy to add, you know, regression datasets and, and keep track of different user queries that you wanna go test against and iterate against. Um, and so those production sessions become our test cases and become our living fixtures over time. They grow, they change, um, as we see new error cases. Um, and we also now have the ability to do session datasets as well. So you can store an entire session as a dataset and kind of play a specific turn through the agent if you need. Um, finally, we use Arize Experiments
- 16:26
throughout all of this as well to make sure that we're, you know, kinda A/B testing, uh, harness changes as we go through this. Uh, one takeaway I want you guys to, to kind of keep in mind also is that in agents, bugs rarely show up as explicit errors. Um, in the old world software, this was a lot. I, I don't know if it was easier, but it was different. Um, now it's more of like a qualitative, okay, the status says okay, but if there are forty-three identical tool calls in the same turn, that's
- 16:56
probably something we wanna fix. Um, softer metrics like latency, right? So I think we can all agree two hundred and twenty-seven seconds is way too long to wait for a response, but what actually is too r- too long, it really depends on your use case. Um, so we use Signal a lot now to kind of automate this review. We, we use less and less of the human reading the trace. Um, our staff engineers are very grateful for that. Uh, but we still do use kind of that human expertise, which I'll get to in a bit. Um,
- 17:27
the fix can also be the next bug, so these regression datasets are, are really important, and I would encourage you guys to keep them in mind. Judges need that human baseline, so that's where we kind of impart the human task. We align our evaluators, and we make sure that our judges actually accurately reflect what, you know, our engineers think as well. Uh, so that's very important. Um, I think we, we still keep a lot of things with the human. Those gold labels, those golden datasets, uh, for judges come from people. Um,
- 17:57
the speed versus quality is a human judgment call. I don't think there's any single answer across the stack on when you should be a little bit faster, but it's okay to degrade quality. It is still, you know, kind of like a human judgment call, even if you have the metrics around it, like, okay, ch-changing models will increase latency by X and decrease, uh, or increase quality response by Y. If we just use Astra everywhere, would that be the right decision? Still a human judgment call, I think. We have the metrics to support that, but, uh, we still have to make
- 18:27
those decisions. Um,
- 18:31
and these are kind of the, the six things that survived all these rebuilds. Uh, tracing into our own products, so we s- of course, still trace everything that Alyx does. Uh, math and data in code, so code computes the aggregations, the model reads them and explains. Uh, and then exposing those as tools to the, to the model. So we're not giving the model plain text math questions. We're actually asking it to use code to go do that. Uh, golden datasets in CI, errors with an obvious fix, streaming and persistent, and skills. Um,
- 19:02
we've changed what we call skills. I think, you know, routers, tools, et cetera, but really, it's give the model a capability around, um, an external system. Uh, so a lot of decisions you'll face, uh, and here's some of what we chose. So feel free to take a picture of the slide. Uh, who owns the plan? What ends a turn? Um, if your tool output is too big for your context, what do you do? Who should write the long-term memory? Do you-- Should you have one? Should you have multiple sub-agents? What model is good enough
- 19:32
and fast enough? Um, and who checks the agent's own fixes? Um, so yeah, that's me. Uh, on the bottom right of the screen there, my LinkedIn and X. Uh, the tagline is, "Every day we use Alyx to build Alyx." We have a bunch of blog posts on sort of like technical deep dives into how we built this. Um, and yeah, encourage you guys to all reach, reach out, and a big shout out to the Alyx team and our engineers for building this. Thank
- 19:58
you.