The Agentic AI Engineer

How do you keep improving agents when every change still needs someone to inspect the results? Building an agent already involves an offline loop: implement, test, evaluate, improve. Deployment adds an online loop: monitor production traces, diagnose failures, and feed those findings into the next version. Benedikt Sanftl and Burak Cemil Özafşar introduce this problem from their work at Mutagent, where they introduce themselves as CEO and co-founder, and CTO, respectively.

The familiar workflow starts with an issue. An engineer implements a change, perhaps with a coding agent, generates samples, and then reads the outputs and traces. Shipping and A/B testing may follow, but interpreting the feedback remains manual. When an organization plans to roll out hundreds of agents, that review work becomes a scaling constraint. Automating implementation alone leaves much of the development cycle untouched.

The opportunity is to fit more complete improvement cycles into the same time window. The proposed Agentic AI Engineer extends agentic work beyond writing code to the evaluation, diagnosis, and optimization that determine whether a change is worth keeping.

The lifecycle begins by separating the responsibilities that a single coding session can otherwise blur together:

1. Specify. Define the agent’s responsibilities, functions, and decisions under particular conditions.
2. Build. Realize that specification in a harness or framework, including a coding environment such as Claude Code or Codex.
3. Evaluate. Check whether the implementation meets functional requirements, much as unit tests check software behavior.
4. Ship. Update the code, an agent platform, or agents running in a local harness.
5. Monitor. Observe production executions and trigger diagnostics, for example after a volume of traces accumulates or on a daily or weekly schedule.
6. Diagnose. Collect failures and categorize their root causes.
7. Optimize. Make targeted changes for the identified failure modes, then evaluate again before redeploying.

There are two entry paths into this lifecycle:

An existing agent does not need to be rebuilt from scratch to benefit from the loop. Its current behavior supplies the starting point for optimization.

A useful specification captures the requirements and success criteria of the process the agent will handle. Outside coding, those processes often differ substantially between companies. The agent needs the right context for its particular environment, rather than only a generic instruction to complete a task.

The specification should make four things explicit:

• Context: What information must be available to complete the work?
• Integrations and tools: Which systems and operations does the agent need?
• Responsibilities: Which jobs does it own, and which jobs are outside its remit?
• Constraints: What boundaries must it respect while making decisions?

Together, these become the blueprint against which future implementations are judged.

The coding agent then translates that blueprint into an implementation for the chosen platform. Keeping the specification separate from implementation details matters because the runtime may change while the business requirements remain. Changing frameworks a year later is one possible scenario; the specification should still explain what the replacement must accomplish.

Özafşar grounds this flexibility argument in the team’s experience building agents over three years. A framework may lack a required capability, or a bottleneck may leave the team waiting for an upstream fix. The practical aim is to choose a harness that fulfills the requirements without making those requirements dependent on that harness. Sanftl points to the movement toward agent-loop runtimes, naming Hermes and Deep Agents as examples.

Once an initial implementation exists, evaluation-driven development supplies a stopping condition. An autonomous worker needs a way to determine whether the agent is good enough to move forward. The evaluation suite has two parts: the metrics or criteria used to judge behavior, and the dataset of cases the agent must satisfy.

Domain experts can define the initial criteria, but they cannot anticipate the entire suite. Historical examples provide a starting dataset; synthesized cases based on known samples provide another. Over time, user feedback and production failures expose missing criteria, edge cases, and difficult inputs. The evaluation suite is a product of discovery. It becomes more useful as it captures the actual situations in which the agent fails.

Özafşar illustrates the review burden with a hypothetical dataset of 200 items. Without automated evaluations, someone has to inspect results in observability dashboards and logs. Repeating that work across many features or parallel experiments stretches the evaluation stage until the reviewer becomes the bottleneck. An evaluator agent can take on the trace-sifting work in the background, but the engineer still has to design a clear evaluation or termination gate.

An agent receives a task or intent, then acts under a system prompt or decision structure. Evaluating that execution means inspecting the trajectory, not merely reading the final response. Was the necessary context available? Did each tool return usable information? An incorrect intermediate tool result can propagate into the final answer, even when later steps appear reasonable.

The harness also affects behavior, so it belongs inside the evaluation boundary and can itself become a target for optimization. Assessing only an isolated prompt or model misses the interactions that determine how the agent performs the task. For tasks that change external state, the resulting state also needs checking: a plausible trace alone does not establish successful completion.

Evaluation must then produce feedback that supports a change. A score from an LLM judge can be useful, but a poorly defined rubric leaves the engineer guessing what to fix. Özafşar prefers binary criteria when they identify a specific violation and a corresponding corrective action.

The distinction is about the quality of the feedback. A pass/fail label still needs a meaningful criterion behind it.

Judge calibration is part of the experimental setup. Different judges can disagree, and the same judge can assess the same case differently across runs. That variance must be controlled well enough to compare versions; otherwise an apparent improvement may reflect scoring noise. Consistent judgments are necessary for a useful comparison, but consistency alone does not establish that a judge is correct.

After deployment, executions supply evidence of recurring failure modes. Diagnostics groups those occurrences by root cause and origin: a problematic prompt section, a missing tool, or a malfunctioning tool, for example. Once the failures are categorized, the next step is to generate evaluations that detect them, followed by remedies intended to address them. Over time, the agent accumulates a history of failure modes against which later versions can be checked.

The first diagnosis may require deep reading of LLM traces. Once a failure is understood, however, it may have code-checkable indicators: particular content or a recognizable sequence of tool calls. Those indicators allow subsequent diagnostics to locate known problems without interpreting every trace from scratch.

Özafşar warns that, at millions of traces, reading every trace can cost more than executing the agent itself. The talk supplies no workload or cost measurement for that comparison. The proposed response is representative sampling: segment the trace population and use learned indicators to guide which traces receive deeper inspection. The accompanying diagram shows the aggregation step—clustering, categorizing, and ranking failing traces into a shorter list of causes.

With an evaluation suite in place, an optimization agent can vary parts of a feature and run experiments toward target scores, including auto-research-style experimentation. The speakers envision automatic shipping once those targets are met and the evaluation suite is green. Production then supplies the outer loop: diagnostics discovers another issue, proposes a remedy, and sends it into the next round of optimization. This is the talk’s proposed autonomous release loop; current Helix documentation instead specifies approval before changes land.

The specialist agents performing these stages collectively form the Agentic AI Engineer. The offline loop improves performance against the current test dataset. The online loop brings in real user feedback and live traces, clusters failures, and derives new evaluation criteria. Those criteria feed back into both the specification and the agent, so the system’s definition of acceptable behavior grows with production experience. This is the mechanism behind the speakers’ expectation of continued improvement, rather than a demonstrated accuracy result.

In the product configuration described during the talk, Mutagent runs in the customer’s environment, locally or in the cloud; a managed service is presented as future work. Two agents are in research preview:

• Evaluator agent: Helps build evaluation sets and datasets that support the optimization loop.
• Diagnostics agent: Analyzes existing production traces, reducing the time engineers spend reading them manually.

An orchestrator connects these agents inside the coding environment.

Source connectors bring in traces and incident reports. Those sources can include trace stores, ticketing systems, and Slack messages where users report failures. The output can target several engineering surfaces: a GitHub pull request, edits to agent Markdown files, framework-specific changes, or deployment to managed services. The orchestrator dispatches the specialist work while those connectors connect it to the systems the team already uses.

The demonstration uses Claude Code as the harness. On startup, the orchestrator shows a dashboard of lifecycle stages, agents found in the codebase, and prior configurations. Slash commands trigger individual workflows; choosing diagnose starts root-cause analysis. The request can target an entire agent or all invocations of a particular skill.

Diagnostics retrieves traces from the configured source, which may be Langfuse, local Claude transcripts, or a JSONL export from an observability platform. Because the diagnostic run takes time, Özafşar switches to a pregenerated result to demonstrate the output. The report walkthrough therefore shows the review workflow, rather than a diagnostic run completing live.

The result is an HTML report that identifies the agent and its tools. When code access is available, it can also identify the harness or framework. The report surfaces primary signals or failure modes according to how frequently they occur in the sampled traces.

The diagnostic process uses multiple tiers of filtering or segmentation to select a representative sample. An LLM first reads a portion of the traces for obvious problems; those findings guide which signals receive further attention. The displayed report places a signal census alongside coverage cards and a timeline heatmap, making the scope of the inspected evidence visible with the findings.

A diagnostic request can also start with a known issue. If a user has reported a particular problem, the engineer can supply it when launching the workflow. The agent then performs a guided search for related incidents, rather than relying entirely on open-ended discovery to decide what matters.

The overview shows the detected issues and their frequency within the selected timeframe. Each failure-mode tab explains the problem and follows a recursive why chain toward its origin, then offers possible remedies. This turns a list of symptoms into a proposed account of what caused them and what could change.

An assumptions block makes that account reviewable. Traces do not always include the code needed to establish why an agent behaved as it did, so the diagnostics agent may infer details incorrectly. The reviewer can inspect and correct those assumptions before selecting fixes. Remedy selection supports both the recommended choice and multiple remedies when several are needed.

After reviewing the individual problems, the user reaches a decisions page with a general feedback field. Özafşar mentions Wispr Flow as a way to dictate that feedback into text. The final output is a Markdown task definition that carries the reviewed decisions back to the coding agent in the terminal. A compact handoff can keep the proposed work and its verification together:

# Approved Remedies

## Failure mode
Describe the recurring problem and the evidence supporting it.

## Reviewed assumptions
Record corrected assumptions and any remaining unknowns.

## Selected remedies
List the changes approved for implementation.

## Verification
Add a regression case for the diagnosed failure.
Run the evaluation suite before proposing a release.

The demonstration ends with this handoff. The selected remedies are ready for the coding agent to apply; the report itself has not established that those changes were implemented or deployed. The engineer’s work is concentrated on reviewing the diagnosis and deciding what should change, while the agents handle the repeated inspection and preparation.