# The Agentic AI Engineer

Benedikt Sanftl · Burak Cemil Özafşar

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/pSto5YaNGUo-agentic-ai-engineer) · [Video](https://www.youtube.com/watch?v=pSto5YaNGUo)

## The Agentic AI Engineer: From Specifications to Reviewed Fixes

An agent-building loop needs more than automated code changes: it needs explicit requirements, actionable evaluations, production diagnostics, and a path from discovered failures to reviewed fixes.

Familiarity with tool-using agents, execution traces, and basic software testing will help you follow the lifecycle.

### When reviewing changes becomes the bottleneck

[0:01](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1s)

How do you keep improving agents when every change still needs someone to inspect the results? Building an agent already involves an **offline loop**: implement, test, evaluate, improve. Deployment adds an **online loop**: monitor production traces, diagnose failures, and feed those findings into the next version. Benedikt Sanftl and Burak Cemil Özafşar introduce this problem from their work at [Mutagent](https://mutagent.io/), where they introduce themselves as CEO and co-founder, and CTO, respectively.

The familiar workflow starts with an issue. An engineer implements a change, perhaps with a coding agent, generates samples, and then reads the outputs and traces. Shipping and A/B testing may follow, but interpreting the feedback remains manual. When an organization plans to roll out hundreds of agents, that review work becomes a scaling constraint. Automating implementation alone leaves much of the development cycle untouched.

![Slide showing implement change, generate samples, manual evaluation and hand tweaking around a loop, with a central clock and three limitation panels below.](https://ai.engineer/api/talk-essay-media/629a88faaf72573d255a6ff05f23f6356e9edd8c0dcc0b9855eb9497381e73da)

Manual evaluation is the bottleneck in a slow development loop.

**The opportunity is to fit more complete improvement cycles into the same time window.** The proposed Agentic AI Engineer extends agentic work beyond writing code to the evaluation, diagnosis, and optimization that determine whether a change is worth keeping.

### Give every stage an explicit job

[3:09](https://www.youtube.com/watch?v=pSto5YaNGUo&t=189s)

The lifecycle begins by separating the responsibilities that a single coding session can otherwise blur together:

1. **Specify.** Define the agent’s responsibilities, functions, and decisions under particular conditions.
2. **Build.** Realize that specification in a harness or framework, including a coding environment such as Claude Code or Codex.
3. **Evaluate.** Check whether the implementation meets functional requirements, much as unit tests check software behavior.
4. **Ship.** Update the code, an agent platform, or agents running in a local harness.
5. **Monitor.** Observe production executions and trigger diagnostics, for example after a volume of traces accumulates or on a daily or weekly schedule.
6. **Diagnose.** Collect failures and categorize their root causes.
7. **Optimize.** Make targeted changes for the identified failure modes, then evaluate again before redeploying.

There are two entry paths into this lifecycle:

| Starting point | What already exists | Where work begins |
| --- | --- | --- |
| Cold start | An intended capability | Conceptualization and specification |
| Existing feature | A running agent with some level of accuracy | Improvement of the existing implementation |

An existing agent does not need to be rebuilt from scratch to benefit from the loop. Its current behavior supplies the starting point for optimization.

![Flowchart connecting cold-start and existing-feature inputs to specification, build, evaluation, release, monitoring, diagnosis and optimization, with production outputs at right.](https://ai.engineer/api/talk-essay-media/1fec331b06855c02dd659ef4fc0b48f870be8bc0631cf8e56c1c359a70e04789)

New features and existing issues enter an orchestrated agent development lifecycle.

### Keep the specification independent of the runtime

[7:15](https://www.youtube.com/watch?v=pSto5YaNGUo&t=435s)

A useful **specification** captures the requirements and success criteria of the process the agent will handle. Outside coding, those processes often differ substantially between companies. The agent needs the right context for its particular environment, rather than only a generic instruction to complete a task.

The specification should make four things explicit:

- **Context:** What information must be available to complete the work?
- **Integrations and tools:** Which systems and operations does the agent need?
- **Responsibilities:** Which jobs does it own, and which jobs are outside its remit?
- **Constraints:** What boundaries must it respect while making decisions?

Together, these become the blueprint against which future implementations are judged.

The coding agent then translates that blueprint into an implementation for the chosen platform. Keeping the specification separate from implementation details matters because the runtime may change while the business requirements remain. Changing frameworks a year later is one possible scenario; the specification should still explain what the replacement must accomplish.

Özafşar grounds this flexibility argument in the team’s experience building agents over three years. A framework may lack a required capability, or a bottleneck may leave the team waiting for an upstream fix. The practical aim is to choose a harness that fulfills the requirements without making those requirements dependent on that harness. Sanftl points to the movement toward agent-loop runtimes, naming Hermes and [Deep Agents](https://github.com/langchain-ai/deepagents) as examples.

### Discover what good enough means

[11:16](https://www.youtube.com/watch?v=pSto5YaNGUo&t=676s)

Once an initial implementation exists, **evaluation-driven development** supplies a stopping condition. An autonomous worker needs a way to determine whether the agent is good enough to move forward. The evaluation suite has two parts: the metrics or criteria used to judge behavior, and the dataset of cases the agent must satisfy.

Domain experts can define the initial criteria, but they cannot anticipate the entire suite. Historical examples provide a starting dataset; synthesized cases based on known samples provide another. Over time, user feedback and production failures expose missing criteria, edge cases, and difficult inputs. **The evaluation suite is a product of discovery.** It becomes more useful as it captures the actual situations in which the agent fails.

Özafşar illustrates the review burden with a hypothetical dataset of 200 items. Without automated evaluations, someone has to inspect results in observability dashboards and logs. Repeating that work across many features or parallel experiments stretches the evaluation stage until the reviewer becomes the bottleneck. An evaluator agent can take on the trace-sifting work in the background, but the engineer still has to design a clear evaluation or termination gate.

### Evaluate the trajectory and make failures actionable

[15:23](https://www.youtube.com/watch?v=pSto5YaNGUo&t=923s)

An agent receives a task or intent, then acts under a system prompt or decision structure. Evaluating that execution means inspecting the **trajectory**, not merely reading the final response. Was the necessary context available? Did each tool return usable information? An incorrect intermediate tool result can propagate into the final answer, even when later steps appear reasonable.

The harness also affects behavior, so it belongs inside the evaluation boundary and can itself become a target for optimization. Assessing only an isolated prompt or model misses the interactions that determine how the agent performs the task. For tasks that change external state, the resulting state also needs checking: a plausible trace alone does not establish successful completion.

Evaluation must then produce feedback that supports a change. A score from an LLM judge can be useful, but a poorly defined rubric leaves the engineer guessing what to fix. Özafşar prefers binary criteria when they identify a specific violation and a corresponding corrective action.

| Evaluation form | What makes it useful |
| --- | --- |
| Score | A rubric that explains what each level means |
| Binary criterion | A precise condition whose failure identifies a problem |

The distinction is about the quality of the feedback. A pass/fail label still needs a meaningful criterion behind it.

**Judge calibration is part of the experimental setup.** Different judges can disagree, and the same judge can assess the same case differently across runs. That variance must be controlled well enough to compare versions; otherwise an apparent improvement may reflect scoring noise. Consistent judgments are necessary for a useful comparison, but consistency alone does not establish that a judge is correct.

### Turn recurring failures into cheaper checks

[18:57](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1137s)

After deployment, executions supply evidence of recurring failure modes. Diagnostics groups those occurrences by root cause and origin: a problematic prompt section, a missing tool, or a malfunctioning tool, for example. Once the failures are categorized, the next step is to generate evaluations that detect them, followed by remedies intended to address them. Over time, the agent accumulates a history of failure modes against which later versions can be checked.

The first diagnosis may require deep reading of LLM traces. Once a failure is understood, however, it may have **code-checkable indicators**: particular content or a recognizable sequence of tool calls. Those indicators allow subsequent diagnostics to locate known problems without interpreting every trace from scratch.

Özafşar warns that, at millions of traces, reading every trace can cost more than executing the agent itself. The talk supplies no workload or cost measurement for that comparison. The proposed response is representative sampling: segment the trace population and use learned indicators to guide which traces receive deeper inspection. The accompanying diagram shows the aggregation step—clustering, categorizing, and ranking failing traces into a shorter list of causes.

![A grid of failing traces leads to five horizontal bars for ranked causes, above panels labeled Cluster, Categorize and Rank.](https://ai.engineer/api/talk-essay-media/ea6ecaa9889fd7cf8b710a1e2d0c5fe5002d0c7f34a6994c3c1b69687e1c9ef3)

Failing traces are clustered, categorized and ranked into a short list of causes.

### Connect experiments back to production

[22:16](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1336s)

With an evaluation suite in place, an optimization agent can vary parts of a feature and run experiments toward target scores, including auto-research-style experimentation. The speakers envision automatic shipping once those targets are met and the evaluation suite is green. Production then supplies the outer loop: diagnostics discovers another issue, proposes a remedy, and sends it into the next round of optimization. This is the talk’s proposed autonomous release loop; current [Helix documentation](https://docs.mutagent.io/helix/what-is-helix) instead specifies approval before changes land.

The specialist agents performing these stages collectively form the Agentic AI Engineer. The offline loop improves performance against the current test dataset. The online loop brings in real user feedback and live traces, clusters failures, and derives new evaluation criteria. Those criteria feed back into both the specification and the agent, so the system’s definition of acceptable behavior grows with production experience. This is the mechanism behind the speakers’ expectation of continued improvement, rather than a demonstrated accuracy result.

![Seven numbered lifecycle steps beside an agent diagram linking specification, an evaluation system, offline evaluation and optimization, and online deployment, monitoring and diagnosis.](https://ai.engineer/api/talk-essay-media/b09e51d31ec309158c0cb712b1b5f8b389331c01bd4b960eccf1ff95d9b84a5b)

The lifecycle joins offline evaluation and optimization with online monitoring and diagnosis.

### Run specialist agents inside the coding environment

[25:06](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1506s)

In the product configuration described during the talk, Mutagent runs in the customer’s environment, locally or in the cloud; a managed service is presented as future work. Two agents are in research preview:

- **Evaluator agent:** Helps build evaluation sets and datasets that support the optimization loop.
- **Diagnostics agent:** Analyzes existing production traces, reducing the time engineers spend reading them manually.

An orchestrator connects these agents inside the coding environment.

Source connectors bring in traces and incident reports. Those sources can include trace stores, ticketing systems, and Slack messages where users report failures. The output can target several engineering surfaces: a GitHub pull request, edits to agent Markdown files, framework-specific changes, or deployment to managed services. The orchestrator dispatches the specialist work while those connectors connect it to the systems the team already uses.

### Start with a scoped diagnostic request

[27:05](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1625s)

The demonstration uses Claude Code as the harness. On startup, the orchestrator shows a dashboard of lifecycle stages, agents found in the codebase, and prior configurations. Slash commands trigger individual workflows; choosing `diagnose` starts root-cause analysis. The request can target an entire agent or all invocations of a particular skill.

Diagnostics retrieves traces from the configured source, which may be [Langfuse](https://langfuse.com/docs/evaluation/core-concepts), local Claude transcripts, or a `JSONL` export from an observability platform. Because the diagnostic run takes time, Özafşar switches to a pregenerated result to demonstrate the output. The report walkthrough therefore shows the review workflow, rather than a diagnostic run completing live.

### Inspect the evidence behind the diagnosis

[29:13](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1753s)

The result is an HTML report that identifies the agent and its tools. When code access is available, it can also identify the harness or framework. The report surfaces primary signals or failure modes according to how frequently they occur in the sampled traces.

The diagnostic process uses multiple tiers of filtering or segmentation to select a representative sample. An LLM first reads a portion of the traces for obvious problems; those findings guide which signals receive further attention. The displayed report places a signal census alongside coverage cards and a timeline heatmap, making the scope of the inspected evidence visible with the findings.

![Dark report showing a signal census table, coverage cards with 1,946 traces and a 102-trace representative sample, and a colored timeline heatmap.](https://ai.engineer/api/talk-essay-media/896b7469b7b44c7e4b6dbccb67d1ad6c849ec6ca563e6595a91dad240587747b)

The diagnostic report presents failure signals alongside scan and sample coverage.

A diagnostic request can also start with a known issue. If a user has reported a particular problem, the engineer can supply it when launching the workflow. The agent then performs a guided search for related incidents, rather than relying entirely on open-ended discovery to decide what matters.

### Review assumptions before handing off fixes

[31:39](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1899s)

The overview shows the detected issues and their frequency within the selected timeframe. Each failure-mode tab explains the problem and follows a recursive **why chain** toward its origin, then offers possible remedies. This turns a list of symptoms into a proposed account of what caused them and what could change.

An **assumptions block** makes that account reviewable. Traces do not always include the code needed to establish why an agent behaved as it did, so the diagnostics agent may infer details incorrectly. The reviewer can inspect and correct those assumptions before selecting fixes. Remedy selection supports both the recommended choice and multiple remedies when several are needed.

After reviewing the individual problems, the user reaches a decisions page with a general feedback field. Özafşar mentions [Wispr Flow](https://wisprflow.ai/) as a way to dictate that feedback into text. The final output is a Markdown task definition that carries the reviewed decisions back to the coding agent in the terminal. A compact handoff can keep the proposed work and its verification together:

```markdown
# Approved Remedies

## Failure mode
Describe the recurring problem and the evidence supporting it.

## Reviewed assumptions
Record corrected assumptions and any remaining unknowns.

## Selected remedies
List the changes approved for implementation.

## Verification
Add a regression case for the diagnosed failure.
Run the evaluation suite before proposing a release.
```

The demonstration ends with this handoff. The selected remedies are ready for the coding agent to apply; the report itself has not established that those changes were implemented or deployed. The engineer’s work is concentrated on reviewing the diagnosis and deciding what should change, while the agents handle the repeated inspection and preparation.

![Decisions page with a general feedback field, a live Markdown preview headed Approved Remedies, and a Copy decisions as markdown button.](https://ai.engineer/api/talk-essay-media/61883f3983e67e0e50c5fc5ce540801dde51277cf952c7d09518735f4552e4b7)

Reviewed remedies appear in a Markdown handoff on the decisions page.

## Resources

- [Mutagent](https://mutagent.io/) — Product overview of agents for specification, evaluation, trace diagnosis and optimization.
- [Helix lifecycle documentation](https://docs.mutagent.io/helix/what-is-helix) — Current research-preview documentation for running Mutagent's lifecycle inside a coding agent, with human approval of changes.
- [Deep Agents](https://github.com/langchain-ai/deepagents) — LangChain's extensible agent harness with subagents, filesystem tools and context management.
- [Hermes Agent](https://github.com/nousresearch/hermes-agent) — Nous Research's agent with persistent knowledge and a loop for creating and improving reusable skills.
- [Langfuse evaluation concepts](https://langfuse.com/docs/evaluation/core-concepts) — Guide to production-trace evaluation, datasets, experiments and feedback between online and offline testing.
- [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) — Practical guidance on agent trajectories, outcomes, grader calibration, repeated trials and regression suites.
- [Wispr Flow](https://wisprflow.ai/) — Voice dictation software for entering spoken feedback as text.

## Slides

### The Agentic AI Engineer

A coiled ring introduces agent development as a continuous loop, with the slide emphasizing the speed of that loop.

[0:00](https://www.youtube.com/watch?v=pSto5YaNGUo&t=0s) [Title slide introducing the agent development loop and its speed.](https://assets.ai.engineer/sites/aiecode2025/sha256/b9caceb11e3f3d84d7a8ab6b429d4cdadfe518aecabfc2a1f81b83f18fa62bd6.webp)

THE MECHANISM

The Agentic AI Engineer

An AI agent is never “done.” It lives in a development loop — and the speed of that loop is the whole game.

Interface text:
MUTAGENT
Bene
Burak
The Agentic [illegible]
02 / 16

### The agent development loop

The seven-stage loop connects offline building and evaluation with online monitoring and diagnosis. Production traces feed back into optimization for the next version of the agent.

[0:26](https://www.youtube.com/watch?v=pSto5YaNGUo&t=26s) [Circular diagram of seven stages from conceptualize through optimize, spanning offline and online work.](https://assets.ai.engineer/sites/aiecode2025/sha256/8d653b8072fb53046cd01100167c5ec899d130730061605503db9e69022954d6.webp)

THE LOOP

An agent is never “done.” It lives in a loop.

Seven stages, run over and over — each pass making the agent a little better. Offline you build & sharpen it; online it runs, and reality feeds the next pass.

Conceptualize
Build
Evaluate
Deploy
Monitor
Diagnose
Optimize

OFFLINE
ONLINE

one loop
offline build · online learn

MUTAGENT

Interface text:
Bene
Burak
The Agentic [illegible]
02 / 16

### Development cycle throughput

The slide compares 12 human cycles with 242 agent cycles in the same time window. Each square represents a development cycle; the denser agent grid illustrates how more cycles can fit into that window.

[2:12](https://www.youtube.com/watch?v=pSto5YaNGUo&t=132s) [Grids compare 12 human cycles with 242 agent cycles in the same time window.](https://assets.ai.engineer/sites/aiecode2025/sha256/cd627d81cfbdca7350723a3ac36b47f1111dea8e54a82aa51b294d0af7317099.webp)

THROUGHPUT

It’s throughput — how many cycles fit the same window.

Each square is one development cycle. Run the loop by hand and the grid stays nearly empty; run it agentically and the same window fills in — and every cycle can make the agent better.

HUMAN
weeks per cycle
12 cycles

AGENT
minutes per cycle
242 cycles

same window of time →
Legend: less → more

Same wall-clock window — the [illegible], and every one can compound on the last. Throughput is the moat.

Interface text:
MUTAGENT
Bene
Burak
The Agentic [illegible]
04 / 16

### Conceptualize: define and design

Business context, intent and constraints feed a signed spec. It sets acceptance criteria and an agent design that later build, evaluation and optimization stages use.

[7:14](https://www.youtube.com/watch?v=pSto5YaNGUo&t=434s) [Business context, intent and constraints feed a signed specification with success criteria.](https://assets.ai.engineer/sites/aiecode2025/sha256/f72b4e8a0146dc6c697c525d4f402c6f92cfdcd66e7fed447a80c8081eb70b76.webp)

PHASE 1 · CONCEPTUALIZE

Define why, design how — into one spec.

Capture the intent and, critically, what “good” means; then shape the agent that delivers it. The signed spec is what every later stage runs against.

DEFINE
Why we’re building it, the intent, and the bar for “good” — the acceptance criteria the agent will be judged on.

DESIGN
The agent’s shape — the routines, tools, and decision logic that turn that intent into behaviour.

business context
intent / goal
constraints

SPEC v1
what “good” means
signed ✓

[illegible] The signed spec is the [illegible] phase runs against — Build writes to it, Evaluate grades against its “good,” and Optimize has to beat it.

Interface text:
MUTAGENT
Bene
Burak
The Agentic [illegible]
06 / 16

### Make good measurable

Teams can define criteria from a spec or production traces, then generate or sample cases. The illustrated eval uses pass/fail checks and rolls results into a 61/100 success rate.

[15:18](https://www.youtube.com/watch?v=pSto5YaNGUo&t=918s) [Criteria and datasets are derived from examples and traces, alongside an example evaluation system with a 61 out of 100 success rate.](https://assets.ai.engineer/sites/aiecode2025/sha256/a7681c1254c655dc6ad2ea5ef6e79ea3a2c2b4dc362ab3df02b2ea1ff1006800.webp)

PHASE 3 · EVALUATE
Make “good” measurable.

Derive a dataset of cases and a set of criteria — each a binary check, so a fail points straight at the broken dimension. Roll them into one 0–100 success rate the loop can chase.

EVALS · THE CRITERIA

GUIDED — COLD START
Define criteria with the team from the spec and a few examples — no data needed yet.

DISCOVERED — FROM TRACES
Derive criteria from production traces — successes say what to keep, failures what to check for.

DATASETS · THE CASES

SYNTHESIZE — FROM GROUND TRUTH
Generate cases from the spec, historical exports, or known-good examples — before you have traffic.

DISTILL — FROM TRACES
Sample real production runs into a representative test set — the cases that actually happen.

EVAL SYSTEM
DATASET · PASS · FAIL PER CASE

CRITERIA — EACH A BINARY CHECK
Cites its source — PASS
No fabricated facts — PASS
Correct tool used — FAIL
Valid output format — PASS
Policy respected — PASS

SUCCESS RATE
61 / 100

Interface text:
MUTAGENT
Bene
Burak
The Agentic [illegible]
08 / 16

### Failures become new evaluations

Production failures are grouped by root cause, then turned into new evaluations. The examples add checks for missing tools, escalation rules, and output schemas to the eval system.

[19:03](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1143s) [Production failures are clustered by root cause and converted into evaluations.](https://assets.ai.engineer/sites/aiecode2025/sha256/785e8771bb02b6d04bb413faf60f98c4cc11d13aeae1d92261b6838e522b45e4.webp)

PHASE 4 · DIAGNOSE

Failures cluster into root causes — and become new evals.

Live, the agent is watched. Failing traces are grouped by root cause, each cause becomes a finding, and each finding is fed back as a new eval — so the system keeps growing itself.

live production traces
cluster →

root causes
missing tool / context  7
wrong escalation  4
format / schema  3

findings → new evals
+ eval: assert tool present
+ eval: escalation rule
+ eval: schema check

absorbed ↓
EVAL SYSTEM
grows every cycle

Interface text:
MUTAGENT
Bene
Burak
The Agentic [illegible]
11 / 16

### Diagnosing failures at scale

Diagnosis groups similar failing traces, labels each cluster, and ranks causes by impact. The speaker says learned indicators and representative samples can reduce the need to read every trace.

[20:24](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1224s) [Failing traces are clustered, categorized and ranked by cause.](https://assets.ai.engineer/sites/aiecode2025/sha256/edfe6db13f940a394d256fc5f0c8af00941cf7785043d4ae9912359fe48e8f7c.webp)

PHASE 4 · DIAGNOSE · AT SCALE

From a flood of failures to a short list of causes.

You can’t read a million failing traces. Diagnosis clusters them by similarity and categorizes each group — so you fix the biggest cause first, not the loudest symptom.

~MILLIONS OF FAILING TRACES
cluster · categorize

A FEW RANKED CAUSES
Missing context — 1.2k
Wrong tool use — 880
Off-path / looping — 610
Unsupported claim — 430
Format / policy miss — 210

Cluster
Group the failing traces that look alike into one bucket — no human reads them one by one.
[illegible] traces → one cluster

Categorize
Label each cluster by the kind of failure, so every bucket maps to a fixable cause.
missing context · wrong tool · off-path

Rank
Order the causes by impact — fix the biggest one first, not the loudest symptom.
1.2k cause → top of the queue

Interface text:
MUTAGENT
Bene
Burak
The Agentic [illegible]
12 / 16

### Signal census and scan coverage

The report selects latency spikes as the primary signal in 1,799 of 1,946 traces. It scans all 1,946 at code level and deep-reads a 102-trace sample; missing output, loops, and cost overshoot remain unconfirmed.

[30:29](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1829s) [A signal census marks latency spikes as primary and shows scan coverage for 1,946 traces and a 102-trace representative sample.](https://assets.ai.engineer/sites/aiecode2025/sha256/3c8b94064b181b0ecac511bb3566686cdcc25be23fbf217fef39854dccd89096.webp)

Signal census → primary selected by failure-validity gate + impact×prevalence + deep-read corroboration

latency-spike selected by impact×prevalence; impact×prevalence (no deep-read corroboration)

Signal / failure-mode | Present? | Measure | Decision
latency-spike | YES | 1799/1946 (92%) | ★ PRIMARY
missing-output | YES | seen in 5/1946 sampled | SECONDARY — discovered, unconfirmed (no mechanical evidence)
loop | YES | seen in 4/1946 sampled | SECONDARY — discovered, unconfirmed (no mechanical evidence)
cost-overshoot | YES | seen in 3/1946 sampled | SECONDARY — discovered, unconfirmed (no mechanical evidence)
wrong-output / hallucination | 0 | 0 errors | excluded — no signal
low-score | 0 | 0 feedback | excluded — no signal

Scan coverage

1,946 TRACES IN WINDOW
5 slices · window-based

1,946 CODE-LEVEL SCAN (TIER 0)
100% · no LLM

102 REPRESENTATIVE SAMPLE
5% of window · worst/med/best/rand

102 LLM DEEP-READ (ANALYZERS)
102 of 102 sampled (100% of sample) · coverage: medium

24h timeline heatmap (colour = avg latency, number = trace count)
[illegible] · 30d · primary signal: latency-spike
Timeline: [illegible]; 02 41; 03 22; 04 12; 05 13; [illegible]; 10 23; 11 29; 12 46; 13 40; 14 45; 15 52; 16 63; 17 57; 18 51; 19 338; 20 466; 21 141; 22 71; 23 60
avg latency: [illegible] <50s · 50–65s · 65–85s · [illegible] · hover a cell for detail
[illegible] at 20h, 19h, 07h; latency peaks [illegible]

[illegible] | Severity | WHAT/WHY/WHERE | [illegible] | One-liner
F- [illegible] | HIGH | latency-spike / tool-misuse / [illegible] | [illegible] | generateDraftReply — a nested LLM composition call serialized after the parent reasoning loop [illegible]

Interface text:
Overview · F-001 The draft tool is the latency si... · F-003 Repeated tool calls with no stop... · F-002 Terse contact replies are silentl... · F-004 Step-count and context tail comp... · Decisions
Selected: [illegible] · Copy decisions as markdown · Reset all
Bene · Burak

### Draft generation latency remedies

The report separates verified assumptions from a hypothesis requiring client code, then offers selectable remedies: cap and stream drafts, or remove the separate draft generation.

[32:32](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1952s) [The report lists assumptions and recommends limiting draft length and streaming draft generation.](https://assets.ai.engineer/sites/aiecode2025/sha256/18b2808cb5aabde2affcb2e5a5411c7133d8a45e94740fadb61b293bfc6c41a2.webp)

[illegible] per-call decode 19.5–61s; agent model call steps only 2–11s — The draft tool re-decodes context in series, double-paying for the same conversation.

The agent design routes message composition through a separate generation instead of producing the message in the reasoning step.
trace trace:673a73edaabd4131#obs:generateDraftReply=63s — Drafting is architecturally a second model round-trip per message.
← ORIGIN

Sampling coverage — representativeness proof
NOT COMPUTED

Per-finding representativeness was not computed for this finding (no per-finding sample proof attached). The run-level sampling methodology still applies; see the Methodology tab.

ASSUMPTIONS
• generateDraftReply is an LLM-backed sub-call (not a deterministic template fill). VERIFIED — Its per-call wall-clock (median 19.5–61s) and decode-bound scaling match an LLM generation, not a template; corroborated independently by the worst-latency and typical analyzers.
• The draft sub-call re-sends/re-decodes the parent conversation context rather than reusing it. HYPOTHESIS-PENDING — Consistent with the double-decode latency pattern, but the tool's internal prompt is not in the export (codeAccess=false) — needs the client's tool implementation to confirm.

Remedies

★ RECOMMENDED
RANK 1  R-001-b  Cap draft length and stream the draft generation
cost:low  correct:medium  local-agent  Atlas-Email-Agent generateDraftReply tool config (max_tokens / streaming)

WHY THIS REMEDY
A low-cost mitigation when the architectural change (R-001-a) is not immediately feasible; bounds the tail without touching the agent loop.

WHY THIS WORKS
[illegible] decode is output-token-bound [illegible] draft length and streaming reduces the serial decode time per draft call proportionally.

APPLY TARGET  Atlas-Email-Agent [illegible] (max_tokens / streaming)

SOURCE NOT FOUND — HYPOTHESIS
The apply target's current source is not accessible to the analyzer, so a Before/After cannot be cited. No diff is shown; apply this remedy as a hypothesis and verify against the live [cut off].

Interface text:
Overview · F-001 The draft tool is the latency si... · F-003 Repeated tool calls with no stop... · F-002 Terse contact replies are silentl... · F-004 Step-count and context tail comp... · Decisions
Bene · Burak
Selected: [illegible] · Copy decisions as markdown · Reset all

[33:12](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1992s) [Diagnostic report proposes producing the message in the reasoning step to remove a separate draft generation.](https://assets.ai.engineer/sites/aiecode2025/sha256/81f456aeef1caad7d26678c9ecd7fe957c4b5812fe293241a672d58047ae2afa.webp)

SOURCE NOT FOUND — HYPOTHESIS
The apply target's current source is not accessible to the analyzer, so a Before/After cannot be cited. No diff is shown; apply this remedy as a hypothesis and verify against the live target.

APPLY PLAN
No structured plan provided.

APPLY INSTRUCTIONS
1. Set a max_tokens ceiling on the draft generation appropriate to email replies.
2. Enable streaming so downstream steps can start before the full draft completes if the pipeline allows.

Feedback on this remedy — overrides, conditions, why-not, modifications. Merged verbatim into the master Copy decisions bundle.

RANK 2  R-001-a  Produce the message in the reasoning step instead of a separate draft generation
cost:medium  correct:medium  local-agent  Atlas-Email-Agent message-composition path (client tool definition for generateDraftReply)

WHY THIS REMEDY
Eliminating the nested generation removes the dominant 35–39% wall-clock term at the architectural root, rather than trimming around it. Preferred over caching/length tweaks because those only shave the secondary tail.

WHY THIS WORKS
If the agent emits the final message text directly in its reasoning step (which already holds the full context, cached at ~85%), the second full decode disappears — cutting one LLM round-trip per message.

APPLY TARGET  Atlas-Email-Agent mess[illegible] (client tool definition for generateDraftReply)

[illegible] NOT FOUND — HYPOTHESIS
[illegible] apply target's current source [illegible] a Before/After cannot be cited. No diff is shown; apply this remedy as a hypothesis and verify against the live [illegible].

APPLY PLAN
[cut off]

APPLY INSTRUCTIONS
1. Have the client confirm whether generateDraftReply re-sends the conversation [cut off]

Interface text:
Overview · F-001 The draft tool is the latency si... · F-003 Repeated tool calls with no stop... · F-002 Terse contact replies are silentl... · F-004 Step-count and context tail comp... · Decisions
Bene · Burak
Selected: [illegible] · Copy decisions as markdown · Reset all

### Decisions and markdown handoff

The decisions page combines selected remedies and general feedback into a live Markdown preview. Burak describes copying the resulting task definition for a coding agent to apply the fixes.

[33:28](https://www.youtube.com/watch?v=pSto5YaNGUo&t=2008s) [Decisions page with a general feedback box and a live preview of approved remedies.](https://assets.ai.engineer/sites/aiecode2025/sha256/4b7675763378e486ebc10b3a3f8aaba5253ec864c8ee01a45f8079171c52facd.webp)

⑦ Decisions — copy approvals

★ green = recommended (pre-checked). Apply-target tags show change-type per remedy.

GENERAL FEEDBACK (SPEECH-TO-TEXT FRIENDLY)
Voice-dump anything here — approve a remedy AND add conditions, reject with rationale, request a remedy not listed, scope overrides, questions, priorities… Folded into the Copy-decisions export verbatim.

LIVE PREVIEW — MASTER PLAN MARKDOWN
4 remedies · 4401 chars

# Approved Remedies (handoff)

Generated: 2026-06-21T10:33:42.679Z

---

## R-001-b — Cap draft length and stream the draft generation
**Finding:** F-001 — The draft tool is the [illegible]
**Severity:** high

[illegible] also has its own notes bo[illegible]ates as you select remedies and type feedback.

Interface text:
MUTAGENT
Atlas-Email-Agent — Diagnostic Report
generated 2026-06-08T22:31:58Z · source langfuse · 1,946 traces · 4 findings
Overview
F-001 The draft tool is the latency si...
F-003 Repeated tool calls with no stop...
F-002 Terse contact replies are silentl...
F-004 Step-count and context tail comp...
Decisions
Selected: [illegible]
Copy decisions as markdown
Reset all
Bene
Burak

### Stop debugging. Start evolving.

Mutagent frames agent development as a continuous loop, from conceptualizing and building through evaluation, deployment, monitoring, diagnosis and optimization.

[34:15](https://www.youtube.com/watch?v=pSto5YaNGUo&t=2055s) [Closing slide presents a continuous agent development loop beneath the words “Stop debugging. Start evolving.”](https://assets.ai.engineer/sites/aiecode2025/sha256/1737c0c728dbcdf93fdd59ae6cad1dfe7b83fc8044114c556e3e3b4249d006b6.webp)

Stop debugging. Start evolving.

Reliable agents come from one thing — a loop that never stops, run fast enough to compound.

conceptualize · build · evaluate · deploy · monitor · diagnose · optimize ↻

MUTAGENT

The Agentic [illegible]
16 / 16

Interface text:
Bene
Burak

## Full transcript

[0:01](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1s) Oh. Hi, everybody. Uh, welcome to our talk, The Agentic AI Engineer. I'm Bene, CEO and co-founder of, uh, Mutagent, and I'm here with my colleague.

[0:13](https://www.youtube.com/watch?v=pSto5YaNGUo&t=13s) Hi, I'm Burak. I'm the CTO of Mutagent, and today we're basically gonna talk about loops and how the Agentic AI Engineer works.

[0:28](https://www.youtube.com/watch?v=pSto5YaNGUo&t=28s) So as you're all aware of now, loops is the hot topic, how you build software, uh, in an agentic loop. And, uh, we apply the same loop to the building of AI agents.

[0:40](https://www.youtube.com/watch?v=pSto5YaNGUo&t=40s) And as you're all aware, there's, uh, two concepts here. One is the offline loop, where while you build, you iterate, um, on your agent, you test it, you evaluate it, you improve it, and you go on.

[0:54](https://www.youtube.com/watch?v=pSto5YaNGUo&t=54s) And then you have a second loop, which is-- we call the online loop, where once your agent is deployed to production, you monitor its traces, you diagnose this, and then you feed it back into your optimization loop, uh, yeah, to iterate and have multiple versions of your agents.

[1:13](https://www.youtube.com/watch?v=pSto5YaNGUo&t=73s) Yeah. Up to until now, um, what we u- did was, uh, this l- doing this loop manually. It's quite slow. Um, the life cycle is basically, um, you have an issue, um, you wanna change something to your agent.

[1:28](https://www.youtube.com/watch?v=pSto5YaNGUo&t=88s) Um, yeah, you implement the change. Um, you maybe vibe implement the change if you use coding agents for it. Um, yeah, you generate some samples for this new feature or issue to test it.

[1:39](https://www.youtube.com/watch?v=pSto5YaNGUo&t=99s) Um, yeah, then you look at the result, you look through the traces, how does the outcome look like. Then you maybe ship it, you do A/B testing, and all your feedback is kind of manually.

[1:50](https://www.youtube.com/watch?v=pSto5YaNGUo&t=110s) It takes very long. Um, yeah, and, uh, the, the bottleneck basically becomes the human, uh, review and the human, yeah, building time. And, uh, yeah, that you can't scale, especially if in your organization you're now planning to roll out hundreds of agents, uh, et cetera.

[2:13](https://www.youtube.com/watch?v=pSto5YaNGUo&t=133s) Yeah, and, uh, yeah, this is why we think the Agentic AI Engineer is the natural next step to build agents. And I'll have, uh, Burak deep dive into how we improve timing and, uh, the road to production reliability with the Agentic AI Engineer.

[2:33](https://www.youtube.com/watch?v=pSto5YaNGUo&t=153s) Uh, so yeah. The key thing here is basically once you reach a certain number of agents or AI-based features, the human performing this loop, again, cannot really scale in enough time.

[2:49](https://www.youtube.com/watch?v=pSto5YaNGUo&t=169s) So this is why doing this agentically is the key to increasing the throughput, because then you can fit many more cycles into the same time window. And now how that loop works is basically we have a few stages.

[3:09](https://www.youtube.com/watch?v=pSto5YaNGUo&t=189s) So this is when you are starting from scratch. Uh, like the current software development practices, you first create a spec for your agent or your skill in this case, and here you need to define all the responsibilities and the functions that agents needs to handle, the decisions that it has to make on

[3:34](https://www.youtube.com/watch?v=pSto5YaNGUo&t=214s) certain conditions. And here, again, this is only the definition stage. Once you defined your agent's requirements, then you can finally go on to the build, and build is where you then realize that spec in a specific harness or, uh, agent framework or in these days, you could even build it as a Claude Code or

[3:59](https://www.youtube.com/watch?v=pSto5YaNGUo&t=239s) a Codex agent. Then comes the next step. This is where you define clear evaluations to evaluate your agent's performance, because these are the key metrics then where you can say, "Hey, my agent is functional or not."

[4:17](https://www.youtube.com/watch?v=pSto5YaNGUo&t=257s) Uh, can think of essentially equivalent to unit tests for coding. This is how you verify, uh, your agent works. Then, uh, after evaluation, if everything looks fine, you usually have the ship, uh, basically where you deploy this agent to production.

[4:37](https://www.youtube.com/watch?v=pSto5YaNGUo&t=277s) Uh, again, this can be a code update, this can be a direct update on, uh, any agent platform or again, your local, uh, harness agents. Then comes the online part.

[4:51](https://www.youtube.com/watch?v=pSto5YaNGUo&t=291s) This is where then the agent is continuously monitored for issues and based on certain trigger conditions, then you can start automatic diagnostics. Again, this can be based on the volume of traces that your agent generates or, you know, weekly or daily jobs.

[5:10](https://www.youtube.com/watch?v=pSto5YaNGUo&t=310s) Uh, then we go into diagnosis stage. This is where you collect all the failures for your agents and do structured root cause analysis to then understand where the failures are coming from.

[5:25](https://www.youtube.com/watch?v=pSto5YaNGUo&t=325s) Once you understand and categorize the failures, then you can finally go on to the optimization stage. This is where then you create, let's say, very specific changes or mutations for your agents, uh, to deal with the found failure modes.

[5:43](https://www.youtube.com/watch?v=pSto5YaNGUo&t=343s) And then the whole cycle repeats again. You evaluate, and if everything looks good, then you can deploy again. Now, we will maybe do a deep dive on each stage, what that entails, uh, then-

[6:01](https://www.youtube.com/watch?v=pSto5YaNGUo&t=361s) So, uh, before we continue, Burak, um, we have two paths here. Um, like, uh, one is the cold start path and one is basically existing features. Um, should we dive deeper for like a half a minute on what the difference is here, what, uh, we see and why this is important?

[6:17](https://www.youtube.com/watch?v=pSto5YaNGUo&t=377s) Uh, yeah.

[6:19](https://www.youtube.com/watch?v=pSto5YaNGUo&t=379s) Yeah. So today, if you again sit down to build an agent, there are, you know, two options. Either you already have an agent, it's built, it's already running somewhere with, um, you know, certain accuracy.

[6:36](https://www.youtube.com/watch?v=pSto5YaNGUo&t=396s) Uh, however, um, the other option is, again, you are creating from scratch. So when you create from scratch, obviously you design with the spec and the conceptualization stage. If you have an existing feature, the most likely, again, the agent is there, but then you are optimizing over something that already exists.

[7:01](https://www.youtube.com/watch?v=pSto5YaNGUo&t=421s) Okay, cool. And, uh, yeah, let's dive deeper into each phase. I mean, uh, you just mentioned the speccing of new building agent, so this seems to be an important artifact, so let's dive into it.

[7:15](https://www.youtube.com/watch?v=pSto5YaNGUo&t=435s) Right. Uh, so with the spec-driven development, also prevalent for building software, uh, artifacts these days or any kind of coding agent workflows, basically the goal is to capture the requirements for the agent and especially the success criteria.

[7:38](https://www.youtube.com/watch?v=pSto5YaNGUo&t=458s) Uh, the reason being, apart from coding agents, the agents in other domains, uh, they handle specific processes and this, uh, differs from company to company, so it's very specialized to the environment where the agent is in.

[7:59](https://www.youtube.com/watch?v=pSto5YaNGUo&t=479s) And then the key point here is to clearly define which context requirements that the agent, uh, has, uh, again, which integrations and tools it then needs to have, what are the jobs to be done or the responsibilities that the agent will handle and what it will not, and then finally, again, the

[8:24](https://www.youtube.com/watch?v=pSto5YaNGUo&t=504s) constraints, uh, and in general, the boundaries for the agent. Um-

[8:32](https://www.youtube.com/watch?v=pSto5YaNGUo&t=512s) Okay

[8:32](https://www.youtube.com/watch?v=pSto5YaNGUo&t=512s) ... now, once we can define a clear spec, as I said, it becomes like a blueprint for any future development, which then the implementation is held against.

[8:46](https://www.youtube.com/watch?v=pSto5YaNGUo&t=526s) Uh-

[8:46](https://www.youtube.com/watch?v=pSto5YaNGUo&t=526s) Okay. Then let's dive into how we would then build an agent from the spec, right?

[8:53](https://www.youtube.com/watch?v=pSto5YaNGUo&t=533s) Right. So basically then the spec tells your coding agent, uh, what to build, and then here the target platform choice is entirely yours 'cause as, uh, I mentioned, the agent space is very...

[9:16](https://www.youtube.com/watch?v=pSto5YaNGUo&t=556s) changing very rapidly these days. So the framework you are using today, you might wanna change, you know, in a year or so. So essentially because of that, uh, spec is, um, isolated from the implementation detail.

[9:35](https://www.youtube.com/watch?v=pSto5YaNGUo&t=575s) So once you decide to build, you can pick any target. Here then your coding agent of your choice will take that spec and give you a, you know, initial version of that agent, uh, which is then basically customized to run on any platform that you see fit.

[9:55](https://www.youtube.com/watch?v=pSto5YaNGUo&t=595s) And-

[9:55](https://www.youtube.com/watch?v=pSto5YaNGUo&t=595s) Why would I want to change the platform, uh, in a year down the line? So what, what, what did we learn out of experience here? Um-

[10:05](https://www.youtube.com/watch?v=pSto5YaNGUo&t=605s) Uh, again, building agents for the last three years, sometimes the agent framework or the harness does not always have the capabilities or-

[10:18](https://www.youtube.com/watch?v=pSto5YaNGUo&t=618s) Mm-hmm

[10:19](https://www.youtube.com/watch?v=pSto5YaNGUo&t=619s) ... occasionally you hit, um, like a bottleneck or a roadblock, and then you have to rely on the underlying framework to kind of get rid of that, and this can sometimes take a while.

[10:34](https://www.youtube.com/watch?v=pSto5YaNGUo&t=634s) The key here is to be flexible because, yeah, essentially again, it-- you wanna pick the best harness or the framework that can, you know, fulfill your requirements.

[10:50](https://www.youtube.com/watch?v=pSto5YaNGUo&t=650s) Uh-

[10:50](https://www.youtube.com/watch?v=pSto5YaNGUo&t=650s) Mm-hmm. I mean, we've all seen it in the, in the last months, uh, we [chuckles] how new harnesses shipped and, uh, yeah, how everything went from like, uh, agents built in code towards, uh, yeah, defined as like an agent loop runtime.

[11:05](https://www.youtube.com/watch?v=pSto5YaNGUo&t=665s) I mean, we've seen Hermes coming up, uh, Deep Agents and all of these frameworks, right?

[11:10](https://www.youtube.com/watch?v=pSto5YaNGUo&t=670s) Yeah.

[11:12](https://www.youtube.com/watch?v=pSto5YaNGUo&t=672s) Okay, let's continue. Uh, what happens after build?

[11:16](https://www.youtube.com/watch?v=pSto5YaNGUo&t=676s) Now, after you build for agents, uh, you essentially go into the eval-driven development loop, which I would call, and this is kind of equivalent to test-driven development for building software with agents because then the agent needs a termination condition, right?

[11:41](https://www.youtube.com/watch?v=pSto5YaNGUo&t=701s) So when is an AI feature or an agent is good enough? So here again, there are two ways to kind of create your eval suite, which is composed of the evals, uh, like the metrics and the criteria you evaluate, and the data sets themselves, which usually contain the cases you have to satisfy.

[12:07](https://www.youtube.com/watch?v=pSto5YaNGUo&t=727s) Now, in the beginning, the original option is that you can sit with your domain experts, uh, and then try to write down

[12:21](https://www.youtube.com/watch?v=pSto5YaNGUo&t=741s) eval metrics and criteria that would cover the feature or the agent you want to build. But most teams working on that will already know that this is a bit difficult, as in you cannot pre-guess the entire evaluation suite from the beginning.

[12:41](https://www.youtube.com/watch?v=pSto5YaNGUo&t=761s) And secondly, you can always start with historical data or synthesized data from a known, uh, you know, sample of the data that you would like the agent to be tested on.

[12:57](https://www.youtube.com/watch?v=pSto5YaNGUo&t=777s) But essentially, the real and the complete eval suite is a product of discovery. What that means is over time, from user feedback, from production failures, you collect the metrics and criteria plus the additional dataset cases, uh, which is often representative for edge cases

[13:22](https://www.youtube.com/watch?v=pSto5YaNGUo&t=802s) or, you know, hard cases that the agent needs to deal with. And with that, then you finally have an evaluation suite where you can run the agent against and exactly know where it fails, uh.

[13:39](https://www.youtube.com/watch?v=pSto5YaNGUo&t=819s) Okay. Before we continue, why does the agentic AI engineer help us here so much? So, um, obviously we're talking about the evaluator agent, so

[13:49](https://www.youtube.com/watch?v=pSto5YaNGUo&t=829s) why is it so, why is it so good to use this kind of concept or thought process in the building?

[13:56](https://www.youtube.com/watch?v=pSto5YaNGUo&t=836s) Uh, I mean, one issue is, again, imagine you have a dataset of 200 items. Uh, here without automated evals, running this and evaluating by like human eyes takes quite a while.

[14:15](https://www.youtube.com/watch?v=pSto5YaNGUo&t=855s) Uh, you would have to, again, scroll through an observability dashboard and logs, uh, and this in turn increases your loop time per eval stage. So as soon as you have a lot of features that you need to evaluate and experiment on, then it suddenly becomes impossible to, you know, do this quickly or in

[14:40](https://www.youtube.com/watch?v=pSto5YaNGUo&t=880s) parallel. Then the human essentially becomes the bottleneck. Uh-

[14:45](https://www.youtube.com/watch?v=pSto5YaNGUo&t=885s) Okay. So we just have an agent do this work in, uh, sifting through traces.

[14:50](https://www.youtube.com/watch?v=pSto5YaNGUo&t=890s) Yes. As the current era says a bit, you know, the... You design loops for your agent, so then they can autonomously work as many of these things in the background.

[15:04](https://www.youtube.com/watch?v=pSto5YaNGUo&t=904s) Uh, and then your job becomes designing these loops with a clear eval or termination gate.

[15:12](https://www.youtube.com/watch?v=pSto5YaNGUo&t=912s) Okay. Understood. So let's look at the how a good eval, uh, supposed to be constructed and what it kind of evaluates.

[15:23](https://www.youtube.com/watch?v=pSto5YaNGUo&t=923s) Right. So in the context of agents, uh, in general, the mainly the trajectory is important because, uh, agents receive input or let's say intent or tasks, and then they have a specific system prompt or like a decision tree that they kind of operate

[15:48](https://www.youtube.com/watch?v=pSto5YaNGUo&t=948s) over. And when doing agent evaluations, again, in general, we check, hey, was the context complete? As in, did the agent have all the required context, uh, to perform the task end to end?

[16:03](https://www.youtube.com/watch?v=pSto5YaNGUo&t=963s) Then this includes also chain, uh, checking every, you know, tool output in the trajectory because every wrong tool output in, um, session can in the end lead to a wrong output as the final output.

[16:22](https://www.youtube.com/watch?v=pSto5YaNGUo&t=982s) Uh, and apart from that, again, there are different things you can evaluate on. So these days also the harness that the agent is op-operating on, uh, has, uh, quite drastic effects on the agent behavior.

[16:40](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1000s) And this is again another vector of optimization. But in the end, when you evaluate an agent, you would like to evaluate all of these things and not something in just isolation.

[16:55](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1015s) Mm-hmm.

[16:58](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1018s) Right. And in general here, what makes an eval useful? So you can always have, uh, you know, metrics or evals that are working in a LLM as a judge fashion and give you some score.

[17:19](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1039s) But here, in order to make an eval, you know, useful, it has to provide actionable feedback. Uh, as in, uh,

[17:31](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1051s) when you use score-based evals, unless your rubric is very well defined, then this does not exactly tell you what to fix. Uh, in such cases, using, uh, binary type of evals or criteria, uh, is preferred because there you have a kind of a call to action.

[17:55](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1075s) If an eval or a criteria fails, then you know exactly what happened and how to kind of deal with that problem. Then, uh, another point is your LLM as a judge solution should be calibrated so that you don't have the scoring noise between judges because- Since LLM

[18:20](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1100s) are, LLMs are undeterministic, what you will mostly encounter is the same judge can, uh, evaluate a problem different ways on each run. And then here you have to make sure your LLM-as-a-judge solution deals with this variance problem.

[18:40](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1120s) Otherwise, it's, uh, hard to run experiments and conclusively say, "Hey, my initial or my improved version is better than my initial version of my agent."

[18:54](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1134s) Mm-hmm.

[18:57](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1137s) Right. Uh, after then evaluation, basically we go live, and this is where then, uh, collecting, let's say, learned failures, failure modes, or primary signals, uh, from the executions takes place.

[19:20](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1160s) Um, here the idea is, again, when an agent encounters a problem over time, there will most likely be multiple occurrences of this. So it starts by identifying what failure mode, uh, that the agent has encountered, and then grouping essentially these failure

[19:45](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1185s) modes by the root causes and where they originate from. Again, this could be a section in the agent prompt. This could be missing tools or, let's say, malfunctioning tools.

[20:01](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1201s) But after your diagnostics results are categorized, then you can finally generate new evaluations, first to detect these problems, and then second, you can generate improvements and remedies based on these problems.

[20:24](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1224s) And yeah, with that, over time you have, um, a buildup of these learned failure modes. So then every agent over time, uh, gathers, uh, these historical data that it can always check against.

[20:44](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1244s) When you, uh, start diagnosing, there's a bit of a upfront cost in the beginning because you often need to deep read the LLM traces to find out what's going on.

[20:59](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1259s) Over time, you can, uh, collect code-checkable indicators, uh, per failure mode. What that means is there are maybe specific pieces of content or there's a specific tool call sequence where you know that the agent will encounter an issue, and then this later on helps you to diagnose problems in your traces without

[21:24](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1284s) actually having to read through all your traces. Now, why is that a problem? Because if you have, uh, let's say, millions of agent traces,

[21:37](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1297s) trying to read all of these, uh, it actually costs more than the execution itself. So it's not the most efficient way of diagnosing

[21:49](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1309s) the problems. Uh, what you want to aim for is, again, try to, uh, pick a representative sample, uh, from your, uh, whole traces, uh, by, you know, um, intelligent segmentation strategies and using, again, as I said, the learned indicators.

[22:13](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1333s) Okay. Cool. And-

[22:16](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1336s) And then with that, you can finally start building the autonomous optimization loop, because here then given that you have a eval suite, you can, you know,

[22:33](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1353s) vary your feature. Again, update certain sections or even run auto-research style experiments to see that whether you can reach the desired or the target scores for your evals. As long as you can reach that, then it's automatically shipped to production.

[22:52](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1372s) Then, uh, from production, you have your outer loop. Uh, again, every issue that's found then on diagnostics gives you a improvement or a remedy, which then you can optimize on, and then as long as the eval suite is green, then this again gets deployed to production.

[23:16](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1396s) Let's continue now with the whole life cycle. Um, yeah. As we learned now, the details of each, everything starts with a spec. Um, you def-define and design it. Uh, you build your agent and all of this can be done obviously agentic.

[23:32](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1412s) So you define agents that do this work, and then they all together become, uh, the Agentic AI Engineer. Um, you go through the offline loop where once you build your evaluation system, uh, yeah, you evaluate, you optimize, you test, you improve your accuracy on your test data set.

[23:51](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1431s) Once you feel ready, you deploy it to production, and that's when the online loop starts. Here you get real feedback from users. Um, yeah, uh, real test cases. You continuously monitor them.

[24:04](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1444s) You look at your live traces. You diagnose them, as we learned before with the diagnose agent. You can do re-root cause analysis. You can cluster your failure modes. Uh, you'll derive new eval criterias from it.

[24:16](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1456s) They become part of the spec. Uh, they become co-part of the agent. So they, they continuously grow with you while you use your agent in production. And, uh, yeah, this becomes the online loop, and the more use cases you collect, the more use production data you see, the better your agent becomes, and the better scoring your agent

[24:37](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1477s) becomes. And this all together now, um, becomes one end-to-end loop that you can run agentic and, uh, yeah. From here on, uh, yeah, you can now see the combination to software-driven development with coding agents.

[24:54](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1494s) You can transport these concepts also for AI engineering and building AI agents. And, uh, now as we at Mutagent work on this, we're gonna show you now how this l- looks like as a product.

[25:06](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1506s) Um, this is kind of how it looks like. Um, for now, everything runs in your environment and we-- in, as cloud and local. Um, we'll offer managed service down the line.

[25:23](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1523s) Um, so it's a set of, uh, agents. We have two in research preview, the ones we talk deep about this in our talk. First one is the evaluator agent that helps you build an eval set, a good data set, because this is the core of the optimization, the eval-driven development loop.

[25:39](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1539s) The second one we have is the diagnose, the diagnostics agent. It does you-- It helps you diagnose your traces you already have in productions because we learn from our users that reading through these traces took them a lot of time and having kind of an agent they can just spin off from their coding environment to analyze the

[25:59](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1559s) traces is, uh, very helpful to them. And, uh, as you can see here, um, both of these agents, um, are connected through an orchestrator, which runs in your coding environment.

[26:12](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1572s) And what you need basically is connectors to sources where you basically have all your traces, where you get your incidents from. This could also be like a ticketing system, um, Slack, where people report failures of your agents.

[26:27](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1587s) Then you connect them and then obviously, as we mentioned before, there's different target platforms. Um, this could become a PR that automatically gets raised in GitHub. Um, this could be just the adjustment of your agents in, uh, yeah, in MD files.

[26:44](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1604s) Uh, yeah, different frameworks we target and, uh, or being deployed to managed services. Um, yeah, kind of this is how our platform works. Um, we spin up different agents.

[26:54](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1614s) They're all connected to an orchestrator and, uh, I guess Burak, let's show, uh, our first agent in research preview, uh, to the audience.

[27:05](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1625s) Uh, in practice. Uh, as I said, you have, um, orchestrator, which kind of handles the dispatch for all the other sub-agents or the stages we have. Then, then you can use it in any coding, um, harness or agent you want.

[27:26](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1646s) In this case, I'm using Claude Code. But then what it will do is when you first boot up, it will show you kind of like a dashboard of which stages you have and also kind of the existing things in, uh, let's say your code base, your agents, and the prior configurations.

[27:46](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1666s) Now, here I have some slash commands. These are basically, let's say, trigger commands for the workflows or specific stages. And if I type diagnose here, this will basically start the diagnostic stage for the root cause analysis.

[28:07](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1687s) And here, again, you can point the diagnostics into a specific scope. This can be an agent, this can be a skill as well, so you could basically diagnose all invocations of a particular skill.

[28:24](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1704s) Here, uh, o- one-- when you start the diagnostics, it will, uh, retrieve traces from your configured source platform. In this case, again, it can be something like Langfuse, it can be your local Claude transcripts or, or another, you know, like a JSONL format that's, like,

[28:49](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1729s) exported from one of the observability platforms. Now, uh, the diagnostics run itself takes quite a while to finish, so I will show you instead a pre-generated, uh, kind of version of what it does and how that looks like in the end.

[29:08](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1748s) Okay, so what do we see here now?

[29:13](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1753s) So basically, once you run the diagnostics agent, as I said, on your agents, you will get, um,

[29:24](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1764s) a generated HTML artifact, uh, which basically shows you, you know, the details of your agent, which tools you have.

[29:35](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1775s) Uh, then in general, if you've, uh, if you have code access, it will also tell you which harness or the framework then the agent is using. And then here, in general, it will

[29:50](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1790s) basically go through your traces and then select, uh, primary signals, or let's say failure modes, depending on how frequent they occur in the trace samples.

[30:06](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1806s) Now, as I said, most of the time you don't want to read all of your traces because this is not cost efficient. So there the diagnostics use like a, a multi-tier filtering or segmentation to kind of pick a representative sample.

[30:27](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1827s) So- Uh, originally a few, you know, pieces of the traces or let's say a portion will be read by the LLM to see that if there are any, uh, detectable obvious problems.

[30:44](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1844s) And based on that, then the diagnostics then decides or to focus on a particular failure mode or a signal. Uh, the other option is then when trigger the diagnostics, you can also specify an issue that you have seen yourself or issue that you're looking for.

[31:08](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1868s) This could be something that the users reported. Uh, in this case, it's again more of a guided search, meaning, uh, hey, uh, if you tell the diagnostics agent you have a particular problem you're interested in, then it will try to find all incidents or occurrences regarding, uh, the same problem.

[31:32](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1892s) Okay.

[31:33](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1893s) Uh-

[31:33](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1893s) And what do we see here in, uh, in, uh, yeah, in full here?

[31:39](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1899s) Uh, so this is in general the overview, which tells you again, which, uh, issues kind of were detected and kind of shows you like the frequency, uh, giving the given timeframe.

[31:56](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1916s) Then for each tab you will have a failure mode and kind of like an explanation of the problem. And then here you can kind of see where that problem comes from or why it happens.

[32:12](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1932s) The agent will provide kind of like, um, a recursive why chain, let's say, to then tell you, "Hey, uh, this is where the issue is originating from." Now, in addition, it will offer you kind of remedies or fixes to deal with that.

[32:33](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1953s) Uh, and then here you will see an assumptions block. Uh, here this is important because, uh, when you are reading traces, sometimes you don't always have access to the code, so this helps us detect, uh, when the LLM or let's say the diagnostics agent makes certain assumptions that are also not correct, and we can

[32:58](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1978s) see and correct them here. Uh, but i- in the end, you will be presented by certain corrections or remedies which then kind of can fix your problem. You can, you know, pick either the recommended or, you know, as many as you need as it's multi-choice.

[33:18](https://www.youtube.com/watch?v=pSto5YaNGUo&t=1998s) And, uh, once you basically go through all your problems, uh, at the end, you kind of get to a decisions page. And here again, for those using, you know, uh, text-to-speech or speech-to-text AI features like Wispr Flow, there's always a general feedback

[33:43](https://www.youtube.com/watch?v=pSto5YaNGUo&t=2023s) box so that you can always talk into your mic. And finally, when you are happy with all the decisions, you get a markdown, uh, let's say, um, task definition for your coding agent, so all the fixes or the remedies can then be directly applied once you go back to your terminal, uh, and to your

[34:08](https://www.youtube.com/watch?v=pSto5YaNGUo&t=2048s) coding agent.

[34:11](https://www.youtube.com/watch?v=pSto5YaNGUo&t=2051s) Thanks for the demo, Burak. So last words on our product.

[34:20](https://www.youtube.com/watch?v=pSto5YaNGUo&t=2060s) Yeah. So, uh, thanks for listening in to our talk. Um, yeah, I hope we could show you some, uh, good insights on how we envision the future of agent building with the Agentic AI Engineer.

[34:33](https://www.youtube.com/watch?v=pSto5YaNGUo&t=2073s) Um, yeah, stop, uh, the debugging. Um, have agents do the tedious work and reach out to us if you have further questions and, uh, yeah, have a great, uh, conference.
