# We Kept Rebuilding Our AI Agent. Here's What Survived - Fuad Ali, Arize AI

Fuad Ali, Arize AI

[Talk page](https://ai.engineer/talks/YIhvrvHtonk-we-kept-rebuilding-our-ai-agent-heres) · [Video](https://www.youtube.com/watch?v=YIhvrvHtonk)

## We Kept Rebuilding Our AI Agent. Here’s What Survived

Fuad Ali follows Arize’s Alyx from a late-2023 prototype to one agent with on-demand skills, explaining why planning, stopping, context management and regression testing became problems for the harness to solve.

### From a useful-but-unreliable prototype to planned work

[0:12](https://www.youtube.com/watch?v=YIhvrvHtonk&t=12s)

Arize’s first prototype of its in-product agent was, in one staff engineer’s recollection, about half hallucination and half something useful. Fuad Ali, a product manager at Arize, uses that late-2023 starting point to explain the rebuilds behind **Alyx**. The estimate is an informal description of the prototype, not a measured accuracy result.

The setting is appropriately recursive: Alyx helps debug Alyx, while **Signal**, a separate trace-reading agent, examines the traces Alyx produces. “Agents all the way down.” Keeping those roles separate matters: Alyx is the user-facing AI engineering agent; Signal helps find problems in its execution.

Inside Arize AX, [Alyx](https://arize.com/docs/ax/alyx) can read execution traces, turn findings into evaluations and datasets, run experiments, generate synthetic data from traces, and carry out multistep tasks. That range makes it more than a chat window over documentation. It needs access to product data and tools, a way to keep track of unfinished work, and enough context to choose what to do next.

![Three cards describe Alyx reading traces, turning findings into evaluators, datasets and experiments, and planning multi-step work while asking before creating or changing anything. A quotation below describes the late-2023 prototype as “50% hallucinations, 50% something useful.”](https://ai.engineer/api/talk-essay-media/c08cbc9b0716f049e862d4d846dc86ea8b75f07da2c1e5ea8753bbd8d233ed52)

Inside Arize AX, Alyx reads execution traces, turns findings into tests and plans work that spans multiple steps.

Prompt optimization supplies a concrete before-and-after example. The earlier product could retrieve prompts stored in Arize and accept a request to optimize them. In the later interface, Ali asks Alyx to optimize its own system prompt, and Alyx creates a plan that includes checking resources around the application. What changes between the two interfaces? The important distinction is the visible move from retrieving a prompt to organizing an investigation around it—not evidence that this particular request produced a better final prompt.

### The rebuilds happened around the model

[4:11](https://www.youtube.com/watch?v=YIhvrvHtonk&t=251s)

One early correction was to move math out of the model and into code. The late-2023 models were unreliable at computation, so the application’s harness—the code that runs the model, tools and surrounding state—took responsibility for it. That division survived the subsequent rebuilds: code computes; the model interprets and explains.

The first official product, then called Copilot, routed requests through a decision tree over a set of skills. An August 2025 rewrite introduced iterative tool calling and a to-do tool, allowing work to continue across multiple model calls. Alyx 2.0 relaunched in February 2026 with a new default model. In June 2026, the team merged specialist agents into one Alyx that loads skills on demand. Long-term user memory followed, written in the background.

![A six-column timeline shows a late-2023 prototype, the July 2024 Copilot launch, an August 2025 rewrite introducing iterative tool calling, February 2026 planning in Alyx 2.0, June 2026 consolidation into one agent that loads skills on demand, and September 2026 long-term user memory.](https://ai.engineer/api/talk-essay-media/a467ab4d9747b7667be8ceaa3b0762530392000a3313127ab89a04319972fa87)

The timeline shows when the team replaced its router with iterative tool calls, added planning and folded specialist agents into one agent that loads skills on demand.

These changes did not involve training the underlying models. Arize wrote its own agent loop, plan handling, stopping rule and memory management rather than using an agent framework. Better models opened up new options, but the team still had to decide how to use them. Owning the harness gave it control over those decisions—and responsibility for discovering where its assumptions broke.

### Move plan bookkeeping out of the model’s way

[6:19](https://www.youtube.com/watch?v=YIhvrvHtonk&t=379s)

Planning began with an apparently reasonable tool: give the model a to-do list and let it check items off. Ali compares the choice to assigning chores. You can hand someone the whole list and ask them to maintain it, or hold the list yourself and keep supplying the next task. Alyx initially used the first arrangement.

The model became too enthusiastic about maintaining the list. Calls went into checking off tasks rather than doing useful work, and after several tool calls it could lose track of its focus. More than 20 iterations on the to-do tools eventually pushed the design toward harness-owned state.

The correction happened in stages. First, an `in progress` status made the current task explicit. Then the harness began rendering the plan into a message on every model call, with the rules beside the state they governed. The next call no longer depended on the model reconstructing its position from a growing history of tool interactions.

Once the plan was kept in ranked order, the extra status became redundant: the next unfinished item already identified the next task. Maintaining `in progress` was creating two calls for bookkeeping that the ordering could express, so the team removed it. At the time of the talk, it was testing the further step of removing the to-do tool entirely, keeping plan state and completion checks in the harness and presenting them inline in messages.

The result is a different division of labor. The harness remembers what remains and checks completion; the model consumes that state and does the next piece of work. A plan can help an agent stay focused, but making the model continually administer that plan can consume the calls the plan was meant to organize.

### A text response does not necessarily mean “done”

[8:19](https://www.youtube.com/watch?v=YIhvrvHtonk&t=499s)

The original stopping rule treated a model response without tool calls as the end of the turn. That assumption failed when a model announced its next action in one response and made the tool call in the next. The harness could stop after the announcement, before the promised work happened.

This was more than a model-upgrade nuisance. Alyx lets customers choose their own provider, including customers committed exclusively to Anthropic or OpenAI. Arize benchmarks models and runs regressions, but cannot guarantee that every customer-selected model has been tested. A stopping rule based on one model’s conversational habits was too fragile for that flexibility.

Finishing therefore became an explicit call that the harness checks. Three controls address different failure modes:

- **Premature completion:** If plan items remain open, the harness can bounce a finish call back up to three times.
- **Approaching the budget:** A warning arrives one step before the iteration limit.
- **Failure to stop:** A forced finish provides a final bound on execution.

The completion check gives unfinished work another chance; the iteration limit prevents that chance from turning into an unbounded loop.

### Treat a huge result like a file

[9:49](https://www.youtube.com/watch?v=YIhvrvHtonk&t=589s)

Context management contains several different problems. Ali’s analogy is a desk covered in books: one book can be too long to read, and enough individually manageable books can still fill the desk. For Alyx, a single tool result can overwhelm the context window, while the accumulated history of its own work can exhaust the budget over time.

Trace analysis makes the first problem concrete. Alyx may be reading another agent’s trace, containing a large JSON result and a great deal of nested activity. Dumping that blob into context can leave little room for useful work. The replacement is the book-on-a-shelf arrangement: keep the large result available elsewhere and leave a compact indication of its contents and location in the working context.

For JSON, the harness truncates within the result and lets the agent use code, including `jq`, to retrieve what it needs from the full data. The model’s ability to write queries becomes a way to inspect a large result selectively.

The pointer preserves access; truncation reduces the immediate context load. This is the practical meaning of “treat a huge result like a file and just grep it”: make the data searchable rather than assuming it must all be present in the model’s current input.

Growing execution history needs a separate correction. When Alyx’s trajectory exceeds a window, the harness makes the model write a checkpoint, then drops the history that checkpoint replaces. A fixed token budget limits retained history. Selective file access handles oversized tool data; checkpointing handles accumulated work. Neither is the same as remembering a user across sessions.

A large trace has an immediate path and an on-demand path
The compact result enters working context. The full result stays available for code-based inspection, so a later query can return needed data without loading the entire trace.
- Large trace result — A JSON blob can overwhelm the model’s context window.
- Full result kept available — The book stays on the shelf rather than on the desk.
- Compact result in context — Truncation and a reference reduce what the model reads immediately.
- Alyx model call — Uses the compact result and requests more data when needed.
- Large trace result → Full result kept available (Retain for later access)
- Large trace result → Compact result in context (Truncate for context)
- Compact result in context → Alyx model call (Supply compact information)
- Alyx model call → Full result kept available (Query with code such as jq)
- Full result kept available → Alyx model call (Return requested data)

### One 8,000-character file per user

[12:19](https://www.youtube.com/watch?v=YIhvrvHtonk&t=739s)

For cross-session memory, Arize tried vector retrieval, episodic and semantic stores, and temporal knowledge graphs. It ultimately chose one Markdown file per user, capped at **8,000 characters**. Ali describes the small file as beating the more elaborate alternatives for Alyx; the comparison is a product-design judgment here, without a reported benchmark or numerical quality margin.

The read and write schedules are deliberately different. Every full request loads the file. After a session finishes, a background job edits it. Memory maintenance therefore does not require the interactive agent to run a retrieval-and-update process at every step, and the size cap bounds how much persistent memory each request introduces into context.

The small artifact also makes review practical. A human labeler can inspect a user’s file, decide whether the right information was retained, and adjust the instructions used by the background writer. Concentrating writes in one end-of-session job gives the team a focused place to enforce user separation, including checking that another user’s information does not enter the file.

The appeal is specific: a bounded amount of user history, loaded consistently, with an inspectable writer and an artifact people can audit. It also expresses a limit. Alyx’s persistent memory must fit into that small file; this design is not an attempt to preserve every detail of every session. For this product, the team preferred that constraint to the machinery of its earlier memory systems.

### Ten specialists became one agent—and some tasks need no loop

[13:18](https://www.youtube.com/watch?v=YIhvrvHtonk&t=798s)

The agent topology followed another expansion and rollback. Alyx began with one trace-analysis agent in 2025, grew to ten specialists, and assigned different specialists to different product pages. On the dataset page, for example, a specialist could inspect datasets and generate synthetic data.

The replacement was one agent that loads skill instructions and tools on demand, then decides which tools to use. The dataset capability remains available without requiring a separate dataset agent as the organizing unit. Ali credits both improving model capability and a desire to simplify; the rollback does not establish that specialists are always the wrong architecture.

Model selection follows the same task-specific reasoning. Arize generally defaults to “fast and good enough,” investing heavily in the harness rather than automatically switching to the newest model. Leading a general benchmark and being the best default for Alyx are different questions. The team does upgrade defaults, but treats that as a product decision to test.

At the time of the talk, the team was exploring routing models by task and training or post-training small models for narrow features. The “filter with AI” feature illustrates why: generating a filter can be simple enough for a very small model and a single call. Giving that feature the entire planning-and-tool loop may add machinery it does not need. Simplification can mean consolidating agents, but it can also mean declining to use an agent loop at all.

### Production sessions become living regression fixtures

[15:26](https://www.youtube.com/watch?v=YIhvrvHtonk&t=926s)

The debugging example returns to the recursive setup: Alyx’s own requests produce traces inside Arize, in a project still named **Copilot Prod**. Where does a user interaction become inspectable engineering evidence? In the live trace view, the in-product request and the record of the agent’s execution meet. The agent helping investigate a system is itself a system to investigate.

Those production records supply regression cases. Useful user queries and newly discovered failures enter datasets that grow as behavior changes. Session datasets can retain an entire session and let the team replay a particular turn through the agent. That provides a way to test a decision made partway through an interaction, rather than reducing every failure to a fresh opening prompt.

Arize Experiments then supports A/B comparisons of harness changes. The reason to retain old cases is blunt: “the fix can also be the next bug.” A change that repairs today’s failure may break behavior that previously worked, so the growing dataset becomes a record of obligations the next harness version must still satisfy.

A successful status alone is a weak definition of agent health. Two examples show what can hide behind it:

- **Repeated work:** A turn with 43 identical tool calls may report an okay status while plainly wasting execution.
- **Excessive waiting:** A response taking 227 seconds can be unacceptable even without an explicit error. Ali regards that wait as too long; the appropriate latency threshold still depends on the use case.

These are behavioral failures that require looking inside the trajectory, not merely checking whether a request returned.

Signal automates more of this trace review, reducing the time staff engineers spend reading executions by hand. Automation does not settle whether its judgments are right. Arize still aligns evaluators with human assessments, checking that judges reflect what its engineers consider good and bad behavior.

### People still define the baseline and choose the tradeoff

[17:27](https://www.youtube.com/watch?v=YIhvrvHtonk&t=1047s)

Human work remains concentrated in defining the target. Gold labels and golden datasets for judges come from people. An evaluator needs examples of the judgments it is supposed to reproduce; an automated verdict cannot supply its own independent standard of correctness.

Speed versus quality is another decision the metrics inform without resolving. A model change might increase latency by one amount and improve response quality by another. Measuring both makes the choice clearer, but does not determine how much waiting a particular workflow should tolerate or which quality losses are acceptable. Those remain human product judgments.

### What survived every rebuild

[18:31](https://www.youtube.com/watch?v=YIhvrvHtonk&t=1111s)

The ending distinguishes the machinery that changed from six commitments the team kept. Some have a detailed mechanism behind them; others remain concise design requirements:

- **Trace everything into the team’s own product.** Alyx’s execution stays available for investigation rather than disappearing behind its final response.
- **Keep math and data work in code.** Code computes aggregations, exposes them through tools, and lets the model read and explain the results.
- **Keep golden datasets in CI.** Preserve known expectations as part of checking subsequent changes.
- **Return errors with an obvious fix.** Make recoverable problems actionable.
- **Keep streaming and persistence.** These remain requirements across the rewrites.
- **Keep skills.** Names such as routers and tools changed, but the lasting purpose was to give the model a capability around an external system.

The surviving questions are concrete enough to use in a design review: Who holds the plan? What permits a turn to end? Where does an oversized result live? Who writes persistent memory? Does the task need several agents, one agent, or one model call? Which model is fast enough and good enough? And who checks the agent’s own fixes? Alyx’s answers changed as the team encountered failures; tracing, code-based computation, regression cases and human judgment made those changes inspectable.

“Every day we use Alyx to build Alyx” is the closing tagline. Its practical substance is that the product’s own sessions become material for debugging and testing the next version. The agent can help investigate its failures, while people still supply the standards by which an apparent improvement earns its place.

## Key takeaways

- Harness-owned plan state reduced redundant to-do bookkeeping. Alyx presents the plan on every model call and was testing removal of the to-do tool entirely.
- Completion needs an explicit check: Alyx can reject a finish call while plan items remain, warn before the iteration limit, and force a bounded stop.
- Oversized tool results, growing history and cross-session memory need different mechanisms: selective file queries, replacement checkpoints, and a per-user Markdown file capped at 8,000 characters.
- Alyx replaced ten page-specific specialists with one agent that loads skills and tools on demand. Narrow features such as filter generation may need only one model call.
- Production sessions supply regression fixtures because an okay status can hide repeated calls or excessive latency. Human labels calibrate the judges, and people still choose the speed–quality tradeoff.

## Resources

- [Alyx documentation](https://arize.com/docs/ax/alyx) — A practical starting point for using the in-product AI engineering agent discussed in the recording.
- [Coding agent tracing and evaluation](https://arize.com/blog/open-source-coding-agent-tracing/) — Related reading for the discussion of repeated tool calls and response latency.
- [Arize harness tracing](https://github.com/Arize-ai/arize-harness-tracing) — A related repository for readers interested in tracing agent execution.

## Related talks

- [Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI](https://ai.engineer/talks/fnuNVFgLjHA-building-self-learning-loops-your-agent-fuad) — Ali’s talk on self-learning loops relates to his question about who checks an agent’s own fixes.
- [Break It 'Til You Make It: Building the Self-Improving Stack for AI Agents](https://ai.engineer/talks/Qvp9vw4jJQ8-break-it-til-you-make-it-building) — Related to the use of production failures to test the next version of an agent.

## Slides

### 2 years of shipping a production agent

Fuad Ali introduces lessons from two years of shipping Arize’s in-product production agent, focusing on what the team rebuilt and what it kept.

[0:11](https://www.youtube.com/watch?v=YIhvrvHtonk&t=11s) [Title slide reading '2 years of shipping a production agent' and 'What we rebuilt and what we kept,' credited to Fuad Ali, Product, Arize. A pink abstract pattern fills the right side. Conference branding, a speaker inset and an event footer surround the slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/da948bdaf4f897860d4b5c546bd19c79d189399238381dcf7db2aec07ef82deb.webp)

2 years of shipping a production agent

What we rebuilt and what we kept

AI ENGINEER NYC · OCTOBER 2026

Fuad Ali · Product, Arize

arize

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### We're gonna get a bit meta here

The turtle stack maps a debugging chain: Alyx debugs your agent, Signal reads Alyx's traces, and a coding agent fixes what Signal finds. This slide sits at the top, also made with an agent.

[0:52](https://www.youtube.com/watch?v=YIhvrvHtonk&t=52s) [Slide reading 'We're gonna get a bit meta here' and 'It's agents all the way down.' Five stacked turtles are labeled from bottom to top: Your agent, the one users talk to; Alyx, which debugs your agent; Signal, which reads Alyx's traces; a coding agent, which fixes what Signal finds; and this slide, also made with an agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/f7cc90825631b4d45f7959496fd9dcdd77685c4c2c85bb8634871bdd97e151d0.webp)

FAIR WARNING

We're gonna get a bit meta here

It's agents all the way down.

...

This slide
yes, also made with an agent

A coding agent
fixes what Signal finds

Signal
reads Alyx's traces

Alyx
debugs your agent

Your agent
the one your users talk to

arize
02

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
Engineering the future of AI

[1:18](https://www.youtube.com/watch?v=YIhvrvHtonk&t=78s) [Five colorful turtles are stacked beneath an ellipsis. From bottom to top, their labels identify the user's agent, Alyx debugging that agent, Signal reading Alyx's traces, a coding agent fixing Signal's findings, and this slide made with an agent. The subtitle reads, “It's agents all the way down.”](https://assets.ai.engineer/sites/aiecode2025/sha256/7a4e71fe89883ea2686a03b1f63c157080ad3dae6402698e5a53322118403dab.webp)

FAIR WARNING

We're gonna get a bit meta here

It's agents all the way down.

...

This slide
yes, also made with an agent

A coding agent
fixes what Signal finds

Signal
reads Alyx's traces

Alyx
debugs your agent

Your agent
the one your users talk to

arize
02

Interface text:
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### Two years of one agent, in four parts

The four-part agenda moves from Alyx’s evolution and six harness decisions to Signal finding retry loops and work that still needs a person, then to choices that survived rebuilds. Real examples include decisions the team reversed.

[1:49](https://www.youtube.com/watch?v=YIhvrvHtonk&t=109s) [Four numbered agenda cards cover how Alyx evolved, six harness decisions, debugging the agent with another agent, and decisions retained through rebuilds. A line below says every example is real, including decisions that were reversed.](https://assets.ai.engineer/sites/aiecode2025/sha256/cbb31f1c5eee38a9c9659ceb79817c7d36456774c21f35c0bdb4a53a8962aba4.webp)

WHAT WE'LL COVER
Two years of one agent, in four parts

01
How Alyx got here
From a router over skills, to a tool-calling loop, to one agent

02
Six harness decisions
Who owns the plan, what ends a turn, memory, one agent or many, the default model, how we test it

03
Debugging the agent with the agent
How Signal found Alyx's retry loops, and what still needs a person

04
What we kept
What survived every rebuild, and the decisions you'll face

Every example is real, including the ones we reversed.

arize
03

Interface text:
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### Alyx is the agent inside Arize AX

Alyx now turns trace findings into tests and plans multi-step work, asking before it creates or changes anything. Jack Zhou called its first prototype (late 2023, GPT-3.5-era models) “50% hallucinations, 50% something useful.”

[2:54](https://www.youtube.com/watch?v=YIhvrvHtonk&t=174s) [Three cards describe Alyx reading traces, turning findings into evaluators, datasets and experiments, and planning multi-step work while asking before creating or changing anything. A quotation below describes the late-2023 prototype as “50% hallucinations, 50% something useful.”](https://assets.ai.engineer/sites/aiecode2025/sha256/c08cbc9b0716f049e862d4d846dc86ea8b75f07da2c1e5ea8753bbd8d233ed52.webp)

THE AGENT
Alyx is the agent inside Arize AX

Reads your traces
Finds where an agent went wrong across sessions with hundreds of spans.

Turns findings into tests
Builds evaluators, datasets and experiments from what it finds.

Plans long work
Runs multi-step tasks and asks before it creates or changes anything.

“50% hallucinations, 50% something useful.”
Jack Zhou, on the first prototype, late 2023, GPT-3.5-era models

arize
04

OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### From a router, to a loop, to one agent

Alyx replaced its decision tree with iterative tool calls in August 2025. In 2026, it gained planning, merged specialists into one agent that loads skills on demand, and added long-term memory for every user in the background.

[4:13](https://www.youtube.com/watch?v=YIhvrvHtonk&t=253s) [A six-column timeline shows a late-2023 prototype, the July 2024 Copilot launch, an August 2025 rewrite introducing iterative tool calling, February 2026 planning in Alyx 2.0, June 2026 consolidation into one agent that loads skills on demand, and September 2026 long-term user memory.](https://assets.ai.engineer/sites/aiecode2025/sha256/a467ab4d9747b7667be8ceaa3b0762530392000a3313127ab89a04319972fa87.webp)

HOW IT GOT HERE
From a router, to a loop, to one agent

Late 2023
Prototype
GPT-3.5-era models. The math moves into code early.

Jul 2024
Copilot launches
A router over a set of skills. A decision tree, on rails.

Aug 2025
The rewrite
The decision tree goes. Iterative tool calling and a todo tool come in.

Feb 2026
Alyx 2.0
Planning ships. Claude Sonnet 4.5 becomes the default.

Jun 2026
One Alyx
Specialist agents merge into one that loads skills on demand.

Sep 2026
Memory
Long-term memory for every user, written in the background.

arize
06

AI Engineer

[4:20](https://www.youtube.com/watch?v=YIhvrvHtonk&t=260s) [Six-column timeline showing the late 2023 prototype, July 2024 Copilot launch, August 2025 rewrite with iterative tool calling, February 2026 Alyx 2.0 with planning, June 2026 consolidation into one agent, and September 2026 background memory. A conference banner appears below the slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/4d31377b16f71cb76d941ecbc871de29f9aac63c8a8f23553fd6a70fe27e99fa.webp)

HOW IT GOT HERE
From a router, to a loop, to one agent

Late 2023
Prototype
GPT-3.5-era models. The math moves into code early.

Jul 2024
Copilot launches
A router over a set of skills. A decision tree, on rails.

Aug 2025
The rewrite
The decision tree goes. Iterative tool calling and a todo tool come in.

Feb 2026
Alyx 2.0
Planning ships. Claude Sonnet 4.5 becomes the default.

Jun 2026
One Alyx
Specialist agents merge into one that loads skills on demand.

Sep 2026
Memory
Long-term memory for every user, written in the background.

arize
06

Engineering the future of AI

## Full transcript

[0:12](https://www.youtube.com/watch?v=YIhvrvHtonk&t=12s) Uh, what's up, guys? Thank you guys for coming to this talk. I know we're getting to the end of the day. You had lunch, you're getting tired. So really appreciate the patience. You've been hearing a lot from all of us at Arize. You just heard Sophia's talk, maybe if you were in the room earlier. So thank you again.  My name is Fuad. Uh, did a talk yesterday, if you weren't here. Uh, I work on product at Arize, so I'm what they call an AI PM, which I'll get into a little bit about, you know, kind of how that relates to shipping agents. Um, today I wanna talk about, uh, two years of shipping our

[0:42](https://www.youtube.com/watch?v=YIhvrvHtonk&t=42s) in-product production agent, uh, and some of the lessons we've learned. Hopefully, there are good takeaways for you guys as you are kinda building your own agents in the wild. Cool. So before I start, we're gonna get a little bit meta here. Uh, and hopefully, you guys are all kinda following along. But, you know, this slide was made with an agent. Yes, it was. Um, we have coding agents, um, that kind of operate with Alyx, with Signal, with other parts of our product that are themselves agents. Uh, we

[1:12](https://www.youtube.com/watch?v=YIhvrvHtonk&t=72s) have Signal, which is kind of the trace reading agent that reads our agent's traces, right? So we have an agent reading the agent traces. We have Alyx, which is the in-product agent. You can think of it like kinda like Cursor for AI engineering. That's the agent I'm talking about today. Uh, we actually use Alyx to debug Alyx, um, so it gets a little bit meta. Um, and then hopefully you get, you know, kind of an idea of how Alyx could help you evaluate your agent and some of the design patterns to be taken from Alyx and how that could be translated to your agent.

[1:42](https://www.youtube.com/watch?v=YIhvrvHtonk&t=102s) So anyways, turtles all the way down, agents all the way down. Uh, and if you are confused about which agent we're talking about, I will make sure to make that clear. So, um, what we're gonna cover is kind of like how Alyx got here. I've got some nice throwback screenshots from 2023, which feels like ages ago, about, uh, of how Alyx used to look, uh, where it used to be just a router over skills, uh, tool calling loop, um, and kind of how we've changed that. Uh, some of the hardest

[2:11](https://www.youtube.com/watch?v=YIhvrvHtonk&t=131s) decisions we made along the way, so specifically who owns the plan, how to end a turn, uh, how we dealt with memory across the agent, uh, should we use one agent, should we use many sub-agents, uh, wha-what default model to choose, and sort of how we test it as well. Um, then, like I mentioned, it gets a bit meta, but we debug Alyx with Alyx. So, uh, and now we have Signal as well, so how Signal is finding Alyx's retry loops and some of the real issues we found with our agent using Signal. Uh, and

[2:41](https://www.youtube.com/watch?v=YIhvrvHtonk&t=161s) then i-importantly, what, what still needs a person as well, and so where the human value kind of adds judgment and the AI PM world kinda comes in, uh, and some of the decisions we kept, um, that have lasted throughout the architecture of Alyx. Um, Alyx is the agent inside Arize AX. Alyx can read your traces, turn findings into tests, so building your evals for you, datasets, experiments. It can plan work. It can, uh, run multi-step, step tasks. It can create synthetic

[3:11](https://www.youtube.com/watch?v=YIhvrvHtonk&t=191s) data for you based on your traces. Um, and it can do a lot of different things. I would say, yeah, this is a quote from one of our staff engineers. Uh, on the first prototype back in late 2023 with like 3.5-era models, it was about fifty percent hallucination, fifty percent something useful. Proud to say we're getting a bit better, um, as models have gotten better as well. Uh, this is just a little before and after of the, uh, visuals, just to give you an idea. So you can give it, you know, prompts that you've stored in Arize. You can ask it to

[3:41](https://www.youtube.com/watch?v=YIhvrvHtonk&t=221s) optimize those prompts. Um, and obviously it had, you know, kind of tools to read prompts from our database, uh, and things like that. Now, uh, we have, you know, much more in-depth flow with planning, uh, with much more beautiful UI. Alyx kind of explains what it's doing. Uh, it'll reason through tasks, uh, much more strongly. So here I have like a request to optimize Alyx's own system prompt, and it actually makes a plan to check, you know, kind of resources around the application as well. Um, so yeah, this window shows up in Arize. You can

[4:11](https://www.youtube.com/watch?v=YIhvrvHtonk&t=251s) use it to debug your traces. Um, how did we get here? So quick kinda like timeline. We started the prototype in late 2023. Um, this was like with 3.5-era models. Um, we then, um, you know, very quickly decided to move a lot of math out of the model and into the harness code because if you remember asking agents to do math back then, I mean, nowadays they're solving every Stokes level problems, but, uh, back then, they were not very good at math.

[4:41](https://www.youtube.com/watch?v=YIhvrvHtonk&t=281s) Um, then we kind of had the official launch. We called it the Copilot back then when things were still called Copilots. Uh, a router over a set of skills, a decision tree. We had some rails. Uh, we rewrote a lot of it in August 2025. Um, iterative tool calling and a to-do tool come in, um, so the agent can actually kinda plan what it's doing. Uh, Alyx 2.0 was our big relaunch in February 2026. Uh, Sonnet 4.5 became the default model, and we'll talk through some of how we

[5:11](https://www.youtube.com/watch?v=YIhvrvHtonk&t=311s) test that model as well. Um, one Alyx was just released in June. So instead of specialist, uh, agents, we merged it into kind of like one agent that just loads skills on demand. Uh, and finally, memory. Um, we now have long-term memory, so as the user kind of interacts with Alyx throughout the course of their session, uh, they are kind of continuously, um, uh, the agent continuously knows, uh, sort of the history of that user, uh, and that's written in the background. Um, and

[5:41](https://www.youtube.com/watch?v=YIhvrvHtonk&t=341s) obviously we didn't train these models. Everything we rebuilt was harness code, which I think, um, actually gives me a bit of faith in the world of engineering that there's still very interesting problems to solve here, and it's not all up to the model, and you can't just run a frontier model and expect it to do everything for you. Uh, models got better on their own schedule for sure, and we made some decisions around that, but Pretty much everything we rebuilt was that harness code. Uh, we don't use an agent framework. Um, the loop, the plan, the stopping rule, and the memory are all

[6:11](https://www.youtube.com/watch?v=YIhvrvHtonk&t=371s) ours. Um, what that means is we have a lot of control over it, um, and, you know, we get to learn a lot along the way as well.

[6:19](https://www.youtube.com/watch?v=YIhvrvHtonk&t=379s) Uh, so first one, planning and to-do lists. Um, the analogy, I, I have a bunch of stuff on the slides, but the analogy I kind of think about around, you know, planning is like, uh, let's say you... I don't, I don't have kids. Maybe some of you guys have kids, and you can kind of relate to this analogy. But let's say you want, you know, your kid to do some chores. Um, you can give it a list of chores and ask it to, uh, or ask them to check off the lists of chores as they go. Um, you can check in with

[6:49](https://www.youtube.com/watch?v=YIhvrvHtonk&t=409s) them continuously and hold the list yourself and not tell them about the list except for the next task, right? So that's one paradigm. Um, and there are trade-offs to kind of like each, uh, scenario. We initially just gave the model a to-do list, and what we saw was it spent most of its time checking off the to-do list. Uh, so a lot of LLM calls for, um, pretty much nothing, like no useful unit of work other than updating a to-do list. It als- we also realized

[7:19](https://www.youtube.com/watch?v=YIhvrvHtonk&t=439s) it tended to lose focus after multiple tool calls. Um, and yeah, 20, 20-plus iterations on the to-do tools themselves. So we added this as an extra tool in its arsenal. And, uh, we eventually moved the plan into the harness one piece at a time. So what I mean by that is, first thing we did was we added an in progress so the, uh, the model wouldn't lose track of where it was. Um, then we decided to put the plan as a message, so the harness renders the plan to the model call every

[7:49](https://www.youtube.com/watch?v=YIhvrvHtonk&t=469s) time, um, the model gets a new call and the rules are next to the data, so it's a little bit easier for the model to interpret where it's at. Uh, we actually ended up dropping the in progress identifier because we realized we're just making two calls for effectively, you know, anything that isn't finished is in, in a stack ranked order is the next thing to do, so we removed that. Um, and we're actually testing now without a to-do tool at all, uh, just keeping everything in line in the messages, uh, completely. So the harness is keeping the state and the completion check,

[8:19](https://www.youtube.com/watch?v=YIhvrvHtonk&t=499s) and the model just consumes that and goes and does the next thing. Um, so maybe some lessons for you guys as you're building to-dos, uh, models can be very eager with, uh, lists. Um, then kind of like what ends the turn, uh, finishing became a gate the harness enforces. Um, so what we at first tried was, you know, kinda text without tool calls, uh, meant done. That broke on models that announce a step in one turn and

[8:49](https://www.youtube.com/watch?v=YIhvrvHtonk&t=529s) make the call in the next, which, um, you know, as you change models, you'll, you'll quickly find some of these breaking things. And one of the things we do in Alyx is we actually allow users to choose their own provider, uh, so we can't necessarily guarantee that we've tested every single model someone will try with Alyx. So a lot of, um, you know, some of the decisions we made here are to guard against those. And yes, we run regression tests, and we check against a bunch of different models, and we benchmark different models on Alyx. We have an opinion on a default, but

[9:19](https://www.youtube.com/watch?v=YIhvrvHtonk&t=559s) you know, obviously, some of our customers are exclusive Anthropic customers or exclusive OpenAI customers. So to actually empower them to use their own models, we kinda had to design against everything that was available, right? Um, so now we have an explicit finish call that bounces back, uh, when plan items are open up to three times, uh, a warning set one step before the iteration limit and then a force finish as well. Um, memory, this is a good one. Um, there are actually a number of problems here, and I'm-- I, I like analogies, so I'm gonna use

[9:49](https://www.youtube.com/watch?v=YIhvrvHtonk&t=589s) another analogy here. It's like, you know, you're, you're working at your desk, um, and a big book lands on your desk, and maybe one big book is fine. You can open the book, kind of search through the book. Uh, if the book gets too long, then it's kinda useless now 'cause you're gonna spend your entire time reading that book. That's one issue. Uh, and that's sort of like, you know, one big tool result comes back, and it's too big for you to read, right? Um, but you can imagine there's actually multiple books on your desk, right? So eventually, not

[10:19](https://www.youtube.com/watch?v=YIhvrvHtonk&t=619s) just are the books too long for you to read on your desk, but there's no space left on your desk, right? You just have too many books, right? So there's a, you know, you know, in this analogy, um, there's a lot of ways you can approach this, right? You can decide to store that book and just have a pointer to the book, right? So that's kind of what we did with the tool results where, okay, I now have a sticky note on my desk that says where the book is and what, you know, kind of high level the book contains. And so if I need to go

[10:49](https://www.youtube.com/watch?v=YIhvrvHtonk&t=649s) read that book, I'll just reference my sticky note and then go find the book on the shelf. My desk clears up a little bit, right? So that's kind of what we do with tool calls, uh, and tool results. For example, like Alyx's, uh, you know, can help you with trace analysis. Some of these traces are other agents' traces, and they can have a shit ton of stuff in the trace, right? So, um, if it's a big JSON blob, for example, that just blows up our context window, and then you can pretty much do no useful work. So instead, what we do is we truncate within, uh, the JSON, and we just let the

[11:19](https://www.youtube.com/watch?v=YIhvrvHtonk&t=679s) agent code to go find it if it needs the full context with jq. Um, one thing the agents are very, very good at is writing code and using code-like tools to, uh, kind of infer things. And so that's kind of what we try to leverage with that. Um, the agent's own trajectory also going past a, a certain, uh, window. Um, so what we now have is the harness makes the model write a checkpoint, uh, and then drops what it replaced. Uh, we have fixed token budget for

[11:48](https://www.youtube.com/watch?v=YIhvrvHtonk&t=708s) history, um, and across sessions, so I kinda got to like the long-term memory piece. Uh, basically, we just ask them a background job to edit one small file per user. I'll dive a little bit more into like why we decided to do this with, with long-term memory, uh, later in the presentation, but essentially that's how it works. Um- Kind of talked through this. So treat a huge result like a file and just grep it. Uh, don't dump it all into context. Don't worry about doing anything fancy.

[12:19](https://www.youtube.com/watch?v=YIhvrvHtonk&t=739s) Uh, for long-term memory, a small file actually beat a knowledge graph, and we, we tried things like vector retrieval, episodic and semantic stores, uh, temporal knowledge graphs, et cetera, et cetera. What we eventually landed on was we just have one file per user. It's a markdown file. It's capped at eight thousand characters. Um, and, you know, I, I think there, there is a lesson here that, you know, as engineers, um, we like to build things, and it's really fun to build things, and sometimes we get in our own way, and we

[12:48](https://www.youtube.com/watch?v=YIhvrvHtonk&t=768s) over-engineer. Um, and this file is just loading in on every full request. Uh, it's written into in a background job after the session finishes. It's small, it's exact, it's easy to load in, it doesn't blow up context, and it's very easy for a human to audit as well. Uh, so you can actually kind of have a human labeler go look at these memory files and, and determine whether the right things are being stored and kind of edit instructions for the background job, so. Um, and then you also only have one

[13:18](https://www.youtube.com/watch?v=YIhvrvHtonk&t=798s) job at the very end, which, uh, you can make very safe as well. Uh, so you're, you're making sure you're not writing memory from other users into the file. Um, so yeah, very good lesson on not over-engineering, uh, your problems. Um, briefly on specialists, um, you know, we had one trace agent that initially just did trace analysis back in twenty twenty-five. Then we had ten specialists, so every page kinda had its own specialist. If you're on the dataset page, there's, you know, a special sub-agent that can

[13:48](https://www.youtube.com/watch?v=YIhvrvHtonk&t=828s) generate synthetic data for you and look, look and introspect on all, all your datasets. Uh, we ended up kind of rolling that back as well and just using one agent where the agent can load skill instructions on demand and tools on demand, and we just let the agent decide what tools to do, right? So it is a bit of an evolution also in, in model capability, but I think simplification as well. Um, we-- O-on model capability, we also kinda stand by that the newest

[14:18](https://www.youtube.com/watch?v=YIhvrvHtonk&t=858s) model isn't always the best by default. Um, there are times where we upgrade the model by default, right? And so we have a perspective on that. Um, I think we have, um, you know, uh, what is the best model is a very subjective question. Uh, what is the best model on a generalized frontier is very different from what may be best for you guys in what you're building or what we're building. Uh, we generally just default to fast and good enough and spend a lot of time on the harness instead. Um, we are now

[14:48](https://www.youtube.com/watch?v=YIhvrvHtonk&t=888s) starting to explore auto-routing, um, so choosing models per task and training our own models or small post-training SLMs, things like that, on very narrow surfaces. Um, and some of these services actually don't even need the loop at all. You can just make a single model call. So an example of that is-- I wonder if I can pull it up. We have this, like, you know, filter with AI. Um, we've come a long way from SQL generation in the, in the past, and it's actually very, very easy to generate filters with very, very small models now. So,

[15:18](https://www.youtube.com/watch?v=YIhvrvHtonk&t=918s) um, that's an example where maybe we don't need the entire loop and we're just over-engineering again. Um,

[15:26](https://www.youtube.com/watch?v=YIhvrvHtonk&t=926s) cool. So that's a good takeaway. Um, production sessions becoming our test. So this is where I'll talk a little bit more about Arize as a platform. Uh, tracing everything is really important, right? So we trace-- Actually, if you look at this, these are the traces from this agent. Uh, so it's very meta here. Um, but every time you ask Alyx a quest-question here, the, the traces, these are the traces from Alyx. We, we still call it Copilot Prod. Um, so those are our live traces coming in from the tool. Um, this

[15:56](https://www.youtube.com/watch?v=YIhvrvHtonk&t=956s) makes it very easy to add, you know, regression datasets and, and keep track of different user queries that you wanna go test against and iterate against. Um, and so those production sessions become our test cases and become our living fixtures over time. They grow, they change, um, as we see new error cases. Um, and we also now have the ability to do session datasets as well. So you can store an entire session as a dataset and kind of play a specific turn through the agent if you need. Um, finally, we use Arize Experiments

[16:26](https://www.youtube.com/watch?v=YIhvrvHtonk&t=986s) throughout all of this as well to make sure that we're, you know, kinda A/B testing, uh, harness changes as we go through this. Uh, one takeaway I want you guys to, to kind of keep in mind also is that in agents, bugs rarely show up as explicit errors. Um, in the old world software, this was a lot. I, I don't know if it was easier, but it was different. Um, now it's more of like a qualitative, okay, the status says okay, but if there are forty-three identical tool calls in the same turn, that's

[16:56](https://www.youtube.com/watch?v=YIhvrvHtonk&t=1016s) probably something we wanna fix. Um, softer metrics like latency, right? So I think we can all agree two hundred and twenty-seven seconds is way too long to wait for a response, but what actually is too r- too long, it really depends on your use case. Um, so we use Signal a lot now to kind of automate this review. We, we use less and less of the human reading the trace. Um, our staff engineers are very grateful for that. Uh, but we still do use kind of that human expertise, which I'll get to in a bit. Um,

[17:27](https://www.youtube.com/watch?v=YIhvrvHtonk&t=1047s) the fix can also be the next bug, so these regression datasets are, are really important, and I would encourage you guys to keep them in mind. Judges need that human baseline, so that's where we kind of impart the human task. We align our evaluators, and we make sure that our judges actually accurately reflect what, you know, our engineers think as well. Uh, so that's very important. Um, I think we, we still keep a lot of things with the human. Those gold labels, those golden datasets, uh, for judges come from people. Um,

[17:57](https://www.youtube.com/watch?v=YIhvrvHtonk&t=1077s) the speed versus quality is a human judgment call. I don't think there's any single answer across the stack on when you should be a little bit faster, but it's okay to degrade quality. It is still, you know, kind of like a human judgment call, even if you have the metrics around it, like, okay, ch-changing models will increase latency by X and decrease, uh, or increase quality response by Y. If we just use Astra everywhere, would that be the right decision? Still a human judgment call, I think. We have the metrics to support that, but, uh, we still have to make

[18:27](https://www.youtube.com/watch?v=YIhvrvHtonk&t=1107s) those decisions. Um,

[18:31](https://www.youtube.com/watch?v=YIhvrvHtonk&t=1111s) and these are kind of the, the six things that survived all these rebuilds. Uh, tracing into our own products, so we s- of course, still trace everything that Alyx does. Uh, math and data in code, so code computes the aggregations, the model reads them and explains. Uh, and then exposing those as tools to the, to the model. So we're not giving the model plain text math questions. We're actually asking it to use code to go do that. Uh, golden datasets in CI, errors with an obvious fix, streaming and persistent, and skills. Um,

[19:02](https://www.youtube.com/watch?v=YIhvrvHtonk&t=1142s) we've changed what we call skills. I think, you know, routers, tools, et cetera, but really, it's give the model a capability around, um, an external system. Uh, so a lot of decisions you'll face, uh, and here's some of what we chose. So feel free to take a picture of the slide. Uh, who owns the plan? What ends a turn? Um, if your tool output is too big for your context, what do you do? Who should write the long-term memory? Do you-- Should you have one? Should you have multiple sub-agents? What model is good enough

[19:32](https://www.youtube.com/watch?v=YIhvrvHtonk&t=1172s) and fast enough? Um, and who checks the agent's own fixes? Um, so yeah, that's me. Uh, on the bottom right of the screen there, my LinkedIn and X. Uh, the tagline is, "Every day we use Alyx to build Alyx." We have a bunch of blog posts on sort of like technical deep dives into how we built this. Um, and yeah, encourage you guys to all reach, reach out, and a big shout out to the Alyx team and our engineers for building this. Thank

[19:58](https://www.youtube.com/watch?v=YIhvrvHtonk&t=1198s) you.
