What Does an AI Engineer Do Now? - Aparna Dhinakaran, Arize AI
Read the talk
What Does an AI Engineer Do Now?
Aparna Dhinakaran explains the engineering behind Arize Signal: agents that investigate telemetry, preserve state across disposable runs, propose fixes and depend on evaluators calibrated against human judgment.
From a talk by Aparna Dhinakaran
At a glance
Ideas worth remembering
Signal investigates configurable batches of traces and remembers prior issues, allowing later evidence to change the significance of an earlier observation.
Disposable sandboxes and persistent external state serve different purposes: temporary access ends with a run, while findings and recovery state support future work.
Evaluators need their own validation: compare their decisions with human judgments on ground-truth datasets before relying on them to assess agent work.
Trace both production agents and the coding agents behind them. Online evaluations over those traces can supply the next investigation with behavioral evidence and judgments.
From writing features to building the system that improves them
Using an LLM occasionally and operating thousands of agents are very different engineering practices. Aparna Dhinakaran of Arize AI opens with that gap, drawing on a funnel she attributes to Andrej Karpathy: more than six billion people who may have encountered an LLM, one to two billion casual users, 20 million professionals using it to compress substantial work, and roughly 5,000 frontier users running thousands of agents. These are the opening framing’s estimates, rather than a measured adoption study presented in the talk. The question is how the rest of the field moves toward operating those systems.
The progression raises the level at which engineers work: from writing code in higher-level languages, through copying generated code and using local agents, to delegating work to cloud agents and eventually teams or “factories” of agents. Dhinakaran presents this as both a roadmap and a direction Arize has already begun pursuing internally. As more execution moves into agents, the engineer has to design how that execution gets checked and improved.
That changes several responsibilities together:
- Feature development: Build the loop that produces and improves a feature, extending the job beyond implementing its first version.
- Review: Set up systems and agents to review work, with evaluations that can assess tasks whose outputs vary between runs.
- Observability: Give agents access to telemetry so they can inspect behavior and find recurring patterns.
The familiar sequence—build, evaluate, ship, monitor, improve—remains. The new engineering work is making the system carry out those activities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Signal reads traces and carries an investigation across runs
Arize’s concrete implementation is Signal, an always-on agent that reads traces, identifies patterns in telemetry and can propose a fix or open a pull request when it has repository access. Its starting task is deliberately small: read a configurable volume of traces collected over a configurable interval and look for problems. The trace data supplies the behavior to investigate; repository access gives the investigation a possible route to a code change.
What must persist when the next batch of traces arrives? Signal needs to remember previously identified issues, rather than treating every batch as unrelated evidence. Dhinakaran gives a useful progression: something initially looks suspicious without becoming a captured issue; later traces make the same concern look more critical. Keeping that history lets a later run reconsider an earlier observation as evidence accumulates. It also gives the agent context about problems already found.
This makes agents primary consumers of observability data. Telemetry becomes input to an ongoing investigation, with memory connecting one run to the next. Designing that consumption path is a different task from putting metrics on a dashboard for a person to inspect: the agent must receive enough recorded behavior and prior issue history to decide what deserves attention.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A trigger starts a fresh sandbox with the context to investigate
An always-on service still needs a rule for when work begins. Signal supports several alternatives:
- Scheduled sweeps: Wake every three or six hours, for example, and pull the configured amount of recent trace data.
- Events: Start an investigation from a Slack message or a user complaint.
- Metrics: Start when a configured metric condition triggers the agent.
These are alternative ways to initiate a run. A scheduled sweep does not need to wait for a complaint, and an event need not wait for the next scheduled interval.
Once triggered, the run receives the material it needs to investigate: trace context, additional third-party telemetry such as APM logs and metrics, repository access, and any necessary keys. Signal starts a sandbox for that run and places the required context and access inside it. The sandbox is the working environment for the investigation.
How does information cross that environment’s boundary? In Dhinakaran’s architecture, a single proxy is the only route in or out of the sandbox. Everything happening inside is traced, and generated outputs—including identified issues—are saved to memory. This arrangement makes the investigator observable too: its actions produce records that can be inspected, while its findings survive beyond the process doing the work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the findings and recovery state; tear down the runtime
A sweep can finish with an issue or without one. At completion, Signal separates what future work needs from what belonged only to this execution. The useful distinction is between retained knowledge and temporary access:
- Retained: Issue outputs and history, traces of the agent’s calls and PR activity, and any learned skills saved to the repository.
- Removed: The sandbox, credentials and repository access associated with the finished run.
A later run can therefore begin with the investigation’s history while receiving a newly created working environment.
The same separation supports recovery during a run. If the sandbox or agent harness fails, Signal is designed to resume from the failed step rather than repeat the investigation from the beginning. Here, the harness is the software running the agent’s work. State lives outside the sandbox, so a replacement sandbox and harness can recover it. The disposable environment can fail without taking all progress with it.
How does persistence help an interrupted run differently from a later investigation? The diagram separates those paths: saved recovery state supplies a replacement execution with progress, while issue history informs future investigations. Its shared node groups retained information without specifying whether those records occupy the same store. The talk also leaves unspecified the guarantees around replaying an action that had already produced an external effect.
Persistent state supports two kinds of continuity
Read the diagram as text
- Current execution. temporary — Sandbox and harness investigate the telemetry.
- Retained information. persistent — Recovery state outside the sandbox and retained issue history, grouped by purpose without specifying storage layout.
- Replacement execution. temporary — New sandbox and harness pick up the interrupted run.
- Later investigation. temporary — Starts with the history of issues already found.
- Current execution → Retained information: Saves progress and findings.
- Current execution → Replacement execution: On sandbox or harness failure: relaunch.
- Retained information → Replacement execution: Restores progress to resume.
- Retained information → Later investigation: Supplies prior issue history.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make model and harness choices testable
Signal’s architecture also aims to avoid dependence on one model or harness. Dhinakaran says it began with Claude Code and, at recording time, also supported Cursor and Codex. Supporting other major harnesses, including open-source ones, is a goal. She describes broader multiple-model support as forthcoming while also describing model selection within the Anthropic family when using Claude Code.
That flexibility creates an evaluation question with a practical cost attached: can a cheaper configuration preserve the quality of detected issues and proposed PRs? More powerful models may produce better work, but their value needs to be assessed through the agent’s results. Dhinakaran presents cost and quality comparison as a reason for the design; she does not give a measured configuration comparison here.
Producing a PR then creates another piece of work to evaluate. Signal runs evaluations over the agent’s proposed change. Finding a plausible problem and writing a patch do not by themselves establish that the change solves the problem or preserves existing behavior. The evaluation stage supplies the check between a proposed improvement and confidence in its outcome.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Define good behavior, then test whether the judge recognizes it
An evaluation can begin with a dataset, a prompt and outputs checked for regressions as part of development. The harder responsibility is deciding what those checks should reward or reject. Dhinakaran recommends spending time with traces and agents, identifying common errors and translating human judgment into evaluations. This is where the engineer defines what good behavior means for the particular product.
Skills can assist with writing those evaluations. Dhinakaran mentions Arize’s Evaluator skill as help for both creating evals and aligning them with human judgment. Assistance with authoring still leaves a separate question: does the resulting evaluator actually make the judgments the team intends? A judge that misses the errors people care about cannot reliably check an automated repair.
Arize addresses that question by evaluating its evaluators. The team builds ground-truth datasets, runs the candidate evaluations and compares their decisions with human outputs. It then works on alignment before releasing a new evaluator. The human judgments provide reference decisions; agreement with them is something to check, rather than assume because an evaluator returns a confident verdict. The talk gives the procedure without specifying an agreement threshold.
Once aligned, evaluations can serve two settings:
- Offline testing: Check a proposed change during development, including whether it introduces regressions on the chosen examples.
- Online evaluation: Judge recorded production behavior as the application runs.
The same effort to define useful judgments can therefore support both release testing and continued observation after release.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Trace the coding agents as well as the product
After pre-release evaluations and deployment, the instruction is to “trace everything.” That includes the product’s agents and the coding agents that built them. Claude Code, Cursor and Codex runs are themselves observable work. Recording their behavior helps explain what happened inside the factory producing the application, extending investigation beyond the application’s final responses.
Production traces then receive online evaluations. Those results become additional data for agents investigating the system: the investigator reads telemetry together with evaluation results to find problems within traces. The causal connection matters. Instrumentation records execution; evaluations attach judgments to that behavior; agents consume both to locate gaps and propose improvements. An evaluation result becomes an input to subsequent engineering work.
This returns to the opening gap between frontier labs and ordinary businesses. Dhinakaran suggests that part of the difficulty is business data: it is messier than the constrained, testable work available in a lab. AI engineers have to make testing and improvement work with that operational reality. Her closing direction is concrete: trace the agents, learn to evaluate the products, and build systems that use collected telemetry to improve them. The engineering responsibility extends through the behavior of the system after it ships.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
A companion product overview covering continuous trace investigation, managed agents, proposed fixes and experiments that test complete agent behavior.
A practical next step for connecting coding agents to Phoenix traces, datasets and experiments.
Explains the instrumentation work behind useful traces, including streamed outputs, asynchronous spans, late-arriving evaluations and attribute limits.
Related talks
- The Death of the Code Review: What the Data Actually Says — Laurie Voss, Arize AI
Develops the verification problem behind software factories, including review harnesses, evaluating reviewers and monitoring production behavior.
- How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase
Adds a framework for deciding where agents can operate with less supervision, using bounded workflows, representative evaluations and human fallback.
Read the complete timestamped transcript
- 0:12
Awesome. Well, welcome to AI Engineer. Um, uh, thought given we're at AI Engineer New York, we'd kind of revisit the question of what does an AI engineer do. Feel like every year we have an idea of what an AI engineer does, and that definition kind of changes. Um, and as I was putting this talk together, it was super timely because Karpathy actually just tweeted something this weekend where he kind of, he kinda talks about this growing gap
- 0:42
between basically folks at the frontier who are running thousands of agents and swarms of always-on agents, and he, he really kind of drops this funnel where six billion plus users have maybe touched or interfaced with an LLM, one to two billion are maybe like casual free tier users, 20 million are professionals who have maybe started using it for more, you know, taking some piece of work that would've taken weeks down to, to days. But really
- 1:12
those at the frontier labs are, which is a super small percentage, it's like 5,000 users, uh, are the ones who are running thousands and thousands of agents at once. And as I look at this, it really is-- I mean, he's kind of dropped a bit of the funnel. I mean, the first question anyone looks at this is that, where am I on this? And y- you know, he's kind of drawn out a bit of a roadmap of how does everyone get to, in the next couple of years, one of those
- 1:42
5,000 users who are running thousands of agents. And this has been happening I'd say for, for a couple of decades now, where the abstraction layer has gotten higher and higher. I mean, definitely faster in the last couple of years, but we've gone from engineers writing higher level abstraction languages to, I mean, just copy-pasting code using local agents, to using cloud agents, to really the expectation now is how, how do all
- 2:12
of us start running a team of agents or factories of agents internally and build toward that? And what this has led to is essentially what is a then and now of what an AI engineer is. And this is, this is, this is a bit, uh, I, I think of a, of a roadmap for some, but I think internally this is what we've already started to kind of push towards. As an AI engineer, your job's no longer to just write the feature, write an AI feature. It's about how do you build the loop to write that feature.
- 2:42
It's no longer about, you know, just reading the diff and actually doing code review, it's actually how do you set up systems and agents to do the code reviews, and how do you write evals to actually do the non-deterministic k- um, to review those non-deterministic tasks. It's actually no longer even about just setting up metrics and, and software kind of telemetry, but it's actually how do you have agents go through and read the observability data and have agents actually consume that telemetry to find
- 3:12
patterns. And so this loop that we've kind of been talking about, which is the build, the eval, ship, you monitor, you improve it, well, the AI engineer is now actually responsible not just for doing each of those steps, but how do you build the loop that does the, the, that actually does each of those steps? And at Arize, one of the things we've rolled out in our product, and I'm gonna talk about a bit of what we've learned in building this, is we've built Signal to actually run this loop for AI engineers. What Signal
- 3:41
does is that Signal actually looks at all of the traces, it reads the telemetry, it identifies patterns in that telemetry, and it can actually put up a fix or a PR if it has access to the repo. And Signal, Signal's an always-on agent. So this is an example of how Signal starts, but it actually starts off by just reading the traces. Super simple example, but it goes through, reads, um, you know, all of this configurable, but
- 4:12
w- an, a certain volume of traces that came in in some cadence. It reads it, and then the goal is to actually identify issues from that telemetry data. It has to keep track of issues that it's previously seen, so we have to consider the memory of did it see something before that it thought was an issue? Did something seem like an issue but it wasn't actually captured, and now as it collects more data, it actually thinks it's much more of a critical, critical issue? And so Signal actually starts
- 4:42
off by reading the traces to identify problems. And so one of the first things that, that we realized was that the future of who's actually consuming telemetry data has really changed. It's no longer about humans consuming telemetry data. It's not even about humans consuming it and agents kind of taking on the task, but it really is how do you set up your telemetry data so that agents are the primary consumers of that data? And what that does is that that
- 5:12
observability data can become the trigger for actually running and triggering these always-on agents. And so the first thing that if you're building a team of a f- you know, a factory of really always-on agents is what triggers these agents to wake up or start? And there can be a variety of triggers. For Signal, for us it's, you, it's configurable to run on an ongoing cadence, so every three hours, six hours, go pull the last, you know, X amount of traces,
- 5:42
read them, identify problems. It can also be triggered by events, so Slack messages, um, if some user complains about something. It can be a metric-based trigger. But these are essentially triggers to kick up the agent actually starting. Once the agent actually starts, we have to spend some time figuring out, well, what does, what does an agent run actually look like? For us, we actually pull in all of the context of the traces,
- 6:12
um, any additional third-party data. So we allow collecting to APM logs, APM metrics, um, but all of the third-party telemetry data that we want to pull in. Um, access to the repo, um, access to the actual, um, y- you-- the, the access to any keys that are needed to actually run, run this agent. But all of this information is, um, put inside of a sandbox, and this is actually what's started up
- 6:42
every time the agent wakes up. And so the agent's triggered, we build a sandbox, we put everything that we need inside of that sandbox. There's a proxy. The only way to get information in and out of the sandbox is by this single proxy. And everything that happens inside of that sandbox is actually already getting traced. And every output, so the issues that are generated and created, are all getting saved to some kind of memory. And so this happens every single time the agent actually runs. Every time the agent
- 7:12
finishes running, so it sweep through all of the traces, um, it either found something or it didn't, there's what actually stays and what actually gets thrown away. What stays is that if it found any issues, all of the output is actually saved off in some kind of memory so that the next time an agent gets spun up, it doesn't have to start from ground zero. It has a history of what are the issues that it actually found. Um, the agents are traced, so all of the telemetry
- 7:42
data of what the agent called, um, you know, if it actually put up a PR, all of those things are actually traced. And any skills that it may have learned while actually doing a run can also get saved to the actual repo. But the things that go away is the sandbox gets torn down. Any credentials, um, you know, obviously get, get removed. Um, the repo access is removed, and so all of the stuff that doesn't need to be saved kinda gets torn down after the run of Signal.
- 8:13
And we've designed Signal in a way to actually keep this agent configu-- uh, durable. So that means that either if the sand-- if the sandbox fails or if the harness fails, then it shouldn't have to start everything from scratch in the middle of the run. It should actually just be able to pick up from the step that it failed and all of the state, so-called, is kinda saved off in kind of a, um, a memory outside of the sandbox. And so we've designed this to
- 8:42
actually be a very durable agent, so that if it fails, it can spin it back up. Um, you can launch a new sandbox, launch a new harness to kind of pick back up where it started. So all of this is essentially the underpinnings of how we built Signal to run every single run. Agent gets triggered, agent gets started, loaded with all the context, the evidence. Uh, it sweeps through the telemetry data. It either finds something or it doesn't. All of that gets saved to memory. If something fails in the middle, the agent can get
- 9:12
re-set up. And we built Signal to also be i- you know, agnostic of the model or the harness. So today, Signal supports not just Claude Code, which is what we started with, but it also supports Cursor and Codex. And the goal is to actually support all of the, the major, even the open source harnesses out there. Um, and Signal actually also supports, um, multiple, uh-- it w- will support kind of multiple models. So even if you're using Claude Code, uh, you can pick any
- 9:42
of the models inside the Anthropic family. Um, and this is really important to us because, one, it's an awesome way for us to evaluate the quality of the agent, um, every time we spin it up. It helps us control costs, so we can see, you know, as we use more powerful models, the better quality maybe the PRs are. But can we reduce the quality of the, the-- you know, can we reduce the cost and actually still maintain the rigor, um, and maintain the quality of
- 10:12
issues that are actually detected by Signal? So this is something we deeply thought about from, from first principles when we were designing Signal. And this is actually a pretty good segue into what does an AI engineer need to do. Well, an AI engineer really needs to kind of learn how to write evals. And so we actually-- every time Signal runs, Signal finds an issue, it puts up a PR. Well, there's evals that are run on top of the agent to test that PR. Um, and an eval can be as simple
- 10:42
as, um, you know, testing something as part of your development workflow. You have a dataset. You can be a simple prompt, um, it-- with, with outputs that you're testing to make sure that there's no regressions. Um, but these evals can also-- n- they need to be kind of aligned with human judgment. And I'd argue that this is actually probably where an AI engineer should be spending a lot of their time as they're building the system to actually run the loop. They should be actually
- 11:12
spending time looking at the traces with their agents, identifying what the common errors are. Um, and there's a lot of skills. I think I was hearing Lance yesterday talk about the Claude skill to build evals. We have a skill with Arize, um, Evaluator, to actually help you build and write evals, as well as align those evals with the human judgment. But this is probably where the shift to defining what is good becomes really, really important for an AI engineer to actually do.
- 11:42
And for a lot of teams I, I think we speak with and we work with, their question becomes: Well, how do I trust the evaluator, and do I, do I really trust that the eval is, um, y- you know, is gonna capture, y- y- you know-- How, how do I trust that it's as good as my judgment? And I'd say what we spend a lot of our own time doing for our own products is that we actually evaluate the evaluators Um, we spend a decent amount of time on actually building
- 12:12
these data sets, golden truth data sets. We run evals, verify them against the human output, and then try to align to make sure that the judge actually agrees with the human, um, and, and they're in agreement before we actually ship out, uh, a new evaluator. And these evals can then be used both in an offline testing as well as in an online testing manner. Um, and this, this is probably what I would say as an AI engineer
- 12:42
is more and more of what the job is becoming. Um, and so what I'm gonna leave you guys with is, as an AI engineer, you've built this agent. You've built this agent to actually run, be always on. You run evals before you actually ship it into production. You can ship it, and then once it goes into production, the goal is to actually trace everything. And tracing everything now means not just tracing the agents, but also
- 13:11
tracing the coding agents behind the agents that built this. So you can trace, um, Claude Code, you can trace Cursor, you can trace Codex, but all of these coding agents can be traced so you can understand what happened, um, in the factory of agents that you all are building. And then these agents, the, all the traces can then also be evaluated online. So there's an offline evaluation when you're actually testing what you're, what a new change is,
- 13:42
but as it goes into production, these become your online evals that you're running on top of the telemetry data. This, for us, ends up becoming the new data that then gets pulled back into the agents to read the telemetry data, the evals, and surface up kind of problems inside of the traces. So the, the system really becomes, how do you go from collect data, eval that data, run agents on top of it so
- 14:12
that you can find problems and figure out where the gaps are to actually improve, improve your systems? So if I go back to kind of the question that we started this whole thing with, it's what does an AI engineer do? I mean, the gap between-- I think there's a pretty big gap between what's happening in the labs today and what's happening with a lot of businesses. And some of it could just be the data inside of businesses are a lot more messy than what is very constrained and testable inside of a lab. But AI
- 14:41
engineers are kind of the, the role to close that gap, and I think it really starts with tracing everything, tracing the agents you're actually building, running evals, so being able to learn how to test these pro- uh, test these products that we're really building, and then how do you build a system so that agents can actually improve, um, from the telemetry that's actually being collected, um, so that you're no longer just building individual features, but you're actually building the system to improve, um, to
- 15:11
improve your products. Um, if you wanna learn more, come check out. We have a booth. Um, you can check out Signal, learn more. But thank you so much for listening to us.