# Designing Agents (The Floor Is the Frontier)

Ben Hylak

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/jHMiYtjoJfA-designing-agents-floor-is-frontier) · [Video](https://www.youtube.com/watch?v=jHMiYtjoJfA)

## Designing Agents: The Floor Is the Frontier

Reliable agents need more than impressive capabilities: they need tests of the whole system, production issues tracked over time, and clear boundaries between detection and investigation.

Familiarity with tool-using agents, execution traces and basic software tests will help; the code example uses TypeScript.

### What actually makes a deployed agent better?

[0:15](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=15s)

Where does evaluation advice stop helping with the agent you are actually building? That is Ben Hylak’s opening question: which parts of the workflow remain annoying, unreliable or poorly served by the prevailing advice? He observes that continual learning is still uncommon in deployed products and labs, despite the attention it receives. There is no settled, one-size-fits-all method for improving agents, so he frames the session as a practical discussion and invites questions from builders.

The earlier chatbot problem was easier to specify. Ask for the capital of the United States, expect Washington, D.C., and check the answer. Hylak roughly estimates that developers already knew the answers to 80–90% of the questions their chatbot users would ask. That made fact checking a plausible center of an evaluation strategy.

Agents operating in finance, healthcare and defense introduce a different problem. Hylak initially disliked the term *agent*, but came to appreciate the distinction it draws: these systems use tools, encounter obstacles and improvise ways around them. An agent might decide to decompile something to get past a roadblock—a workaround its developer never anticipated. The same flexibility can produce an ingenious solution or a harmful action. Evaluating the final answer alone no longer captures everything that matters.

![Slide contrasting agents a year ago with chatbots becoming agents, with a smaller line mentioning finance, healthcare, and defense.](https://ai.engineer/api/talk-essay-media/95b198f89a2bfd9c20b7f0de6b632f4bd3aaf11237178fc91816c62b8c7cd82e)

“A year ago, agents barely existed. Now, every chatbot has become an agent.”

### When the harness changes, what does the test still mean?

[3:57](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=237s)

Much evaluation advice still assumes the chatbot setting: assemble a predefined dataset, then grade responses against expectations. Hylak uses a 1,000-example dataset as his example of that prescription. The problem appears when expectations encode the implementation rather than the behavior the product needs. A test might require a particular tool call whenever the user asks a particular question; replacing the model or harness can invalidate that expectation.

![White slide reading: “Meanwhile, agent evals are stuck in the chatbot era. Pre-defined, fragile test-cases, etc.”](https://ai.engineer/api/talk-essay-media/be64e61b6af11ba4dcc1f4f8a61b47d7429bd6ac799c921ab1bf3d9e1d729ac9)

Agent evals remain stuck in the chatbot era of predefined, fragile test cases.

In Hylak’s illustrative harness-switch example, moving to Claude Code CLI makes 80% of the existing tool-specific evals unhelpful. This is a hypothetical illustration, not a measured failure rate. The underlying concern is durability: if the agent has a different set of tools or a different way to solve the task, a test that insists on the old route may no longer tell you whether the new system works.

That maintenance burden becomes a product decision. After spending months constructing an eval set, would a team delay adopting a new model by two weeks to update it? Hylak suspects most would not. His criticism of what he calls “big eval” is directed at this gap between prescribed process and actual release decisions. **Evaluation should buy safety that teams will use**, rather than create an elaborate gate they bypass whenever it becomes inconvenient.

Hylak’s perspective comes from his work as Raindrop’s CTO and co-founder. He describes the company’s work as finding critical production issues, checking that fixes work without unexpected side effects, and simulating changes against past behavior before release. He names Vercel, Speak and Framer as customers and reports adoption among Fortune 100 companies. That exposure informs advice the company revises as models and effective practices change.

Its open-source tracing tool, [Workshop](https://www.raindrop.ai/workshop/), also provides a place to experiment with self-healing loops: when a needed capability is missing, an agent can extend the tool itself. Hylak reports that Workshop has thousands of users. He compares hosted Raindrop’s agent issue detection to Sentry and points to [How to evaluate AI agents](https://www.howtoeval.com/) as the company’s practical guide to these workflows.

### The product team’s responsibility

[7:10](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=430s)

The useful question is not simply which issues an agent has, but which changes will make it better with the tools available. Some problems cannot yet be solved economically. Hylak uses **[Fable](https://www.anthropic.com/claude/fable)** as an example: training a comparable model yourself may make less sense than waiting to integrate it. This leads to his distinction between a *benchmark maxer* and a *floor raiser*.

The word *eval* obscures that distinction because it can mean a lab benchmark, an application test or an online assessment. Borrowing a model lab’s language does not give a product team the same job or the same tools.

| Concern | Model lab | Application team |
|---|---|---|
| Main target | General-purpose capabilities | A particular product’s behavior |
| Knowledge to supply | Broad model competence | Company data and domain meaning |
| Integration problem | Model behavior at the API | Data shapes, access and tools |

Application teams must teach the system what their data means and how to access it. Their responsibility extends into the environment around the model.

User expertise changes that responsibility further. The guide’s quiz asks whether users are domain experts and whether the product augments or replaces their work. A developer can delete an incorrect Copilot autocomplete or Cursor tab completion. Engineers using Claude Code CLI or Codex can also recognize mistakes and repair their environment; a failure might originate in something installed incorrectly on the user’s computer. Products such as Devin shift more work onto the system. An AI doctor raises a different question again: how much can the user reasonably be expected to detect and correct?

The **ceiling** is the most impressive capability the agent can exhibit. The **floor** is its worst behavior: recommending a competitor, deleting data, or sending an unwanted AI-written email to a customer because it happened to have email access. These failures damage trust even when the system is otherwise highly capable. Hylak cites **GPT-4o** sycophancy as an example of a floor problem rather than a missing capability. Raising the floor means attending to the behaviors that make users unwilling to trust the product.

### Test the whole agent, locally

[11:28](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=688s)

Offline evaluation needs to follow the shape of the system being evaluated. A `string.includes()` check on an answer, or a playground for editing a cloud-managed prompt, captures only a narrow slice of an agent. Hylak says he knows only one or two companies still using managed cloud prompts. In the systems he sees, the effective prompt includes the code, the harness and everything connected to it—not just an instruction string.

**Keep evals as code and make them look like software tests.** Unit tests can exercise individual components; end-to-end tests can exercise the agent through its harness. Hylak cites Sentry’s [Vitest Evals](https://github.com/getsentry/vitest-evals), which he describes as Vitest with syntactic sugar, and invokes OpenAI’s term *macro evals*. His practical recommendation is straightforward: run tests on the agent locally, alongside the code that determines its behavior.

![Slide with guidance to make evals code-aware and think in terms of software testing, alongside a dark code example and a note to keep evaluation local.](https://ai.engineer/api/talk-essay-media/7b1edf25b0a9ba96dfedebfacd3e52d2a78bf06f9c60efbb0eb0dc99e9f9daad)

Evaluate the agent, not an LLM call.

### Turn strange behavior into an actionable issue

[13:08](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=788s)

Production improvement begins with discovering failures you do not already know to test for. But a stream of reports that something weird happened does not establish what deserves attention. For each issue, Hylak wants two facts: **when it started** and **how many users it affects**.

Those facts support a practical investigation:

1. **Establish onset.** If an issue began yesterday or several days ago, inspect what changed around that time.
2. **Identify candidate changes.** A model replacement or a downstream change gives the investigation a concrete starting point.
3. **Measure reach.** Count affected users and express that count as a percentage of users, so an anecdote can be placed in context.

Hylak contrasts an issue affecting three users with one affecting 100,000 users to illustrate why reach changes prioritization. Agents can produce an effectively unbounded variety of problems; onset and reach help a team choose where to spend its effort.

The right evaluation method also depends on the deployment’s scale. Hylak describes Raindrop customers ranging from five users to millions. A small internal enterprise application can still provide critical information, so a small audience does not imply low stakes.

At illustrative volumes of 10, 20 or 100 million messages per day, Hylak recommends experiments on a very small sample of free-tier traffic. For products with only five or ten users, he advises against experiments or A/B tests. These are different operating conditions, not different levels of commitment to quality: the available traffic changes which methods can produce useful evidence.

### A cluster is not an operational issue

[15:31](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=931s)

The first of Hylak’s three tactical lessons from Raindrop is that clustering traces does not, by itself, define useful issues. Clustering can help with one-off exploration: a mass of logs becomes a set of patterns worth inspecting. He connects this exploratory work to Hamel Husain’s error analysis, but the useful activity is understanding failures, not merely obtaining clusters.

Ongoing operations need something more stable. As with ordinary telemetry, a team needs to know when an issue began and how much it has grown. Hylak points to the difficulty of tracking clusters reliably over time, including through temporal clustering. Cluster boundaries also need not match the product’s own definition of an issue. Two companies may reasonably group the same behaviors differently because their products and responsibilities differ.

Consider a wrong price quote and an incorrectly calculated refund. Both might fall into a cluster called *price issues*, but that category can hide separate root causes.

| Observed behavior | Broad cluster | Useful distinction to preserve |
|---|---|---|
| Wrong price quoted | Price issues | Quote failure |
| Wrong refund calculated | Price issues | Refund calculation failure |

A common subject does not establish a common cause. If the grouping erases that distinction, it becomes harder to tell which failure started recently, which is growing, and which fix should address it.

### Express trace classifiers as code

[18:11](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1091s)

The second lesson applies **code mode**, familiar from discussions of MCP, to production traces. Instead of treating every classification as an open-ended reasoning task, write classifiers that can run in a sandbox. Hylak reports that this approach scales to production use. The slide gives concrete categories such as repeated tool calls, missed extraction and tool failures.

![Slide showing a classifier code example checking repeated tool calls, missed extraction, and tool failures.](https://ai.engineer/api/talk-essay-media/2e56237498d62a855b29a656c534675832e99d81edcc519048d5f430e3b5184a)

Many classifiers can be written as code.

For example, a TypeScript classifier can select explicit tool failures while preserving the trace and call identifiers needed for investigation. In this small example, `quote_price` succeeds and `calculate_refund` fails; the returned finding points to the refund call rather than grouping the entire trace under price issues.

```typescript
type ToolCall = {
  id: string;
  name: string;
  status: "success" | "error";
};

type Trace = {
  id: string;
  toolCalls: ToolCall[];
};

type Finding = {
  traceId: string;
  callId: string;
  toolName: string;
  category: "tool-failure";
};

function classifyToolFailures(trace: Trace): Finding[] {
  return trace.toolCalls
    .filter(call => call.status === "error")
    .map(call => ({
      traceId: trace.id,
      callId: call.id,
      toolName: call.name,
      category: "tool-failure" as const,
    }));
}

const trace: Trace = {
  id: "trace-1",
  toolCalls: [
    { id: "call-1", name: "quote_price", status: "success" },
    { id: "call-2", name: "calculate_refund", status: "error" },
  ],
};

const findings = classifyToolFailures(trace);
```

The classifier detects an explicit error status; it does not establish whether a successful tool call returned a correct price or refund. That requires a different check. Hylak’s recommendation is to execute such classifiers in a sandbox; the function above supplies the classification logic, not the sandbox.

Raindrop has a feature to assist this workflow, but Hylak emphasizes that teams can implement it themselves. The operational idea is to make a useful distinction executable and apply it across traces.

### Detect a signal, then ask an agent to investigate

[18:38](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1118s)

The final lesson separates **anomaly detection** from **anomaly investigation**. In Hylak’s experience, agents are poor at the former. Rather than asking an agent to search a large body of activity for something anomalous, give it an anomaly that has already been detected.

Extract deterministic signals wherever possible. Keyword frequency is his example: count occurrences, identify a spike, and make that spike the target of an agent’s investigation. A spike does not prove there is a product issue. It supplies a tangible change the agent can examine, explain and connect to the surrounding traces. The agent’s task becomes investigating a specific signal instead of deciding, from an undifferentiated mass of activity, what deserves attention.

## Resources

- [How to evaluate AI agents](https://www.howtoeval.com/) — Ben Hylak's practical guide to floor raising, code-aware evals and learning from production failures.
- [Workshop by Raindrop](https://www.raindrop.ai/workshop/) — A free, open-source local tool for debugging agents and working with agent evals.
- [Sentry vitest-evals](https://github.com/getsentry/vitest-evals) — A Vitest extension with examples of agent harnesses, evaluation cases and judges.
- [Evals Skills for Coding Agents](https://hamel.dev/blog/posts/evals-skills/) — Hamel Husain explains skills for reviewing traces, categorizing failures and building product-specific evaluations.

## Slides

### How to Raise the Floor

Ben Hylak introduces a practical look at how people improve agents and what works in the real world.

[0:20](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=20s) [Title slide: How to Raise the Floor, practical evaluation for frontier agents, presented by Ben Hylak of Raindrop.](https://assets.ai.engineer/sites/aiecode2025/sha256/44f2f19c4419415fd87309d71d2b44ad61fb25e6f6d6f5275bb1162ddc3d7fb2.webp)

HOWTOEVAL.COM

How to Raise
the Floor

Practical evaluation for frontier agents

raindrop | Ben Hylak

### An Invitation to Q&A

Ben Hylak invites agent builders to bring their real-world evaluation challenges to an extended Q&A, saying there is no one-size-fits-all solution.

[0:35](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=35s) [Slide says the speaker would rather be grilled than just say things and will leave time for Q&A.](https://assets.ai.engineer/sites/aiecode2025/sha256/3868b7c77447d162c9b3711b73fc7f91e022d72e02cdc92b2026ad007a538245.webp)

i could just say things.
i’d rather let you grill me

i’ll keep a lot of time for Q/A after.

### The Rise of Agents

The speaker recalls a mostly chatbot landscape a year earlier, when limited flexibility made fact-checking easier. Agents are now being deployed in finance, healthcare, and defense.

[1:50](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=110s) [Slide contrasts the recent emergence of agents with their deployment in finance, healthcare, and defense.](https://assets.ai.engineer/sites/aiecode2025/sha256/f149b0ee8db53e967e39b37c5e40a915f7e1f0b7dcf0dd5d3294986af3887054.webp)

A year ago, agents barely existed.
Now, every chatbot has become an agent.

Agents are being deployed in finance, healthcare, and even defense.

### Agent Behavior and Harm

Agents use tools to navigate their environments and find creative ways around roadblocks. The speaker argues that those unexpected solutions can be helpful or harmful.

[3:00](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=180s) [Slide describes agents navigating environments, using tools, and finding creative solutions that may help or cause harm.](https://assets.ai.engineer/sites/aiecode2025/sha256/1f662af06d5f0ffcbb0faafff7fd900c6d08242155aedf83e0bc1a9e41d4331d.webp)

Agents are an almost self-aware entity.

Navigating their environment, using the tools at their disposal.
Finding creative solutions to problems their developers couldn't have imagined.
Sometimes those solutions are helpful. Sometimes they cause real harm.

We've come a long way from next token prediction.

### Agent Evals in the Chatbot Era

The speaker argues that predefined agent evals are fragile: tests tied to specific tool calls can break when teams change models or harnesses.

[3:56](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=236s) [Slide says agent evaluations remain stuck in the chatbot era, with predefined, fragile test cases.](https://assets.ai.engineer/sites/aiecode2025/sha256/e30b4449ab9dc0dda1d67f24a42d83c7c9ec0f698b203e4de5be741ed9babf8e.webp)

Meanwhile, agent evals are stuck in the chatbot era. Pre-defined, fragile test-cases, etc.

### Raindrop Logo

The speaker introduces Raindrop, which finds critical issues in production agents.

[5:23](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=323s) [A black Raindrop logo centered on a light background.](https://assets.ai.engineer/sites/aiecode2025/sha256/126b7326b3e20fff22be2f90a68af8486d77db55e1e8dd9d60804cebc0a5da6b.webp)

### Simulating production changes

Raindrop finds issues in production agents, checks that fixes work without unexpected side effects, and simulates changes using past behavior before deployment.

[5:29](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=329s) [Three statements describe finding issues, verifying fixes, and simulating changes; the simulation statement is emphasized.](https://assets.ai.engineer/sites/aiecode2025/sha256/06379bf4b3c40ba64886ae78516b60427907ff6fbf45d40fa49c80828d171bb3.webp)

raindrop finds hidden issues in production agents.
verifies that changes actually work - without side effects.
simulates changes before they land in production (based on past behavior)

### Companies using Raindrop

Raindrop lists Vercel, Speak and Framer among its users. Its co-founder says these customers give the team a view of what works in the real world.

[5:41](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=341s) [A headline says the product is used by AI companies and Fortune 100s, above a grid of company logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/c6bc952d9dc7fa5eef7877944dbf52e6e8ca4ad90c95f137b166ca2860301656.webp)

TRUSTED BY

Used by the best AI companies
and Fortune 100s

Vercel
Speak
Framer
clay
Browserbase
DOORDASH
Applied Intuition
Poke
Tolan
GCAI

### Workshop and Raindrop

Workshop is an open-source local agent debugger for traces and tool calls. Raindrop is the hosted offering for detecting issues in agent behavior.

[6:15](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=375s) [A split slide presents Workshop as a local agent debugger and Raindrop as a production visibility product, with interface screenshots.](https://assets.ai.engineer/sites/aiecode2025/sha256/aabfd646f4b7d6b93c13872ade70e6c60da1eb2bd2e210d3be916413a8cc1584.webp)

Workshop
The missing local agent debugger. See traces, tool calls and sub-agents... and let Claude/Codex test your agent.
OPEN SOURCE
raindrop.ai/workshop

906 GITHUB STARS
56 FORKS
OSS FREE LOCAL DEBUGGER
curl -fsSL https://raindrop.sh/install | bash

Raindrop
Production visibility for agent behavior, issues, and user impact.

Interface text:
Task CLI reports success for no-op done and remove commands
[illegible]
Events: 333
Users: 15
Last seen: 1h ago
Detected: 10d ago
Details
Activity
Users / Events
User Impact
Repeat 'done' calls on already-completed tasks and 'remove' calls for nonexistent task IDs return success output and exit zero [illegible] instead of reporting a no-op failure.

Remove 404 prints success and rewrites tasks.json even though no task existed. Return a concise stderr error with a nonzero exit, while preserving successful removal behavior.

Considering CLI modification
[illegible]

### HowToEval community resource

The speaker’s team publishes HowToEval as a practical guide to evaluating AI agents. The slide previews its 2026 edition.

[6:50](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=410s) [A slide introduces HowToEval as a resource for evaluating AI agents, alongside a cropped website preview.](https://assets.ai.engineer/sites/aiecode2025/sha256/fc280f7344278baa4380e2d9ca7cd978531233a585cccde2efc475e7a68f17f5.webp)

COMMUNITY RESOURCE

HowToEval.com

We publish one of the most popular resources on how to evaluate AI agents.

https://www.howtoeval.com/

2026 EDITION

How to evaluate AI ag[illegible]

From code-aware evals to self-healing loops: what actu[illegible] works for evaluating AI agents in production.

By Ben Hylak / Published May 2026 / 14,330 views

CONTENTS
Foreword
Benchmark-maxxer or floor-raiser?

Foreword
A year ago, agents barely existed. Now they [illegible]

Interface text:
howtoeval.com

### How do I make my agent better?

The speaker frames the goal as improving an agent with the tools available: some issues may be impossible or not worth solving.

[7:08](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=428s) [Centered text asks how to make an agent better.](https://assets.ai.engineer/sites/aiecode2025/sha256/4aa69b9e7f13eaea9d7c664330cd448da4cf69e9ca7d0d9bf4110b22e1ee58f6.webp)

How do I make my agent better?

### Benchmark-maxxer or floor-raiser?

Companies building on models have different tools and responsibilities from labs. Copying lab benchmarks can send them after a goal that was never made for them.

[7:54](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=474s) [A slide asks whether the audience is a benchmark-maxxer or floor-raiser and says teams often copy benchmarks made for different goals.](https://assets.ai.engineer/sites/aiecode2025/sha256/161addcd59c52652f09dbfdf12a71d0151334ad560a87980304696b5d030e106.webp)

Are you a benchmark-maxxer or floor-raiser?

Most teams never make this choice explicitly. They borrow the language of evals from labs, copy the shapes of benchmarks, and end up chasing a goal that was never made for them.

### Take the quiz

The quiz weighs irreversible actions, user expertise, and the cost of mistakes. An engineer can delete a bad autocomplete suggestion; other users may not recognize or easily fix an agent’s error.

[9:22](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=562s) [Three quiz questions ask about irreversible actions, user expertise, and the consequences when an agent is wrong.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b05d5ffcd95424dbf2f3b1fbaa6c9c32939fd03998a68669858124337dd8520.webp)

TAKE THE QUIZ

TAKE THE QUIZ
QUESTION 2 OF 7
Does your agent touch anything irreversible?
Sends emails, moves money, makes decisions that stick.
Yes
No. Everything it does can be undone.

TAKE THE QUIZ
QUESTION 3 OF 7
Are your users primarily engineers?
Or experts in the domain the agent operates in.
No. Most users wouldn’t spot a wrong answer.
Yes. They know when the agent gets it wrong.

TAKE THE QUIZ
QUESTION 4 OF 7
Your agent is wrong. What happens next?
It could be really bad.
The user sighs, fixes it, and moves on

### Ceiling vs Floor

The ceiling is an agent’s best capability; the floor is its worst behavior. Deleting data or sending an unintended email can break user trust, and frontier labs may not fix those failures for you.

[10:23](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=623s) [Two panels contrast an agent’s ceiling, its best performance, with its floor, its worst behavior.](https://assets.ai.engineer/sites/aiecode2025/sha256/6e979d4ac36245186ec325357a96c0cd5adc4a2f9b96da301b37228a9a0c8f59.webp)

THE MENTAL MODEL

Ceiling vs Floor

CEILING
The best your agent can do.
Usually the job of frontier labs.

FLOOR
The worst your agent does.
You can't wait for frontier labs, and they might not help anyway.

### Hitting the floor hurts

A house outline places capabilities at the ceiling and issues at the floor. The speaker argues that problems at the floor can break user trust.

[11:23](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=683s) [A house outline illustrates capabilities as the ceiling and issues as the floor.](https://assets.ai.engineer/sites/aiecode2025/sha256/c31f5f9ca0a71a376df73d3a5cdb0ad094cc5a0e94fc0badeb8a5e04cd86cccb.webp)

WHAT USERS FEEL

Hitting the floor hurts.

The ceiling is defined by capabilities.
The floor is defined by issues.

CEILING
capabilities

FLOOR
issues

### Floor-raising

The talk turns to floor-raising, starting with offline evals.

[11:26](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=686s) [Section divider reading “Floor-raising.”](https://assets.ai.engineer/sites/aiecode2025/sha256/b63882a838e1abb1c27b30d195bc7602dc0e5c7f46e7eeeab22802d28df5c0cc.webp)

Floor-raising

### Evaluate the agent, not an LLM call

Run offline evals like software tests: exercise the agent’s real path and check its outputs and tool calls. The refund example checks both approval status and the sequence of tools used.

[11:32](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=692s) [Guidance to make agent evaluations code-aware and test the full system, alongside an example test.](https://assets.ai.engineer/sites/aiecode2025/sha256/844a8395a15ca12ccf0cf2825320c11af0840b6bc72bf6df4053ad21bbb21969.webp)

OFFLINE EVALS

Evaluate the agent,
not an LLM call.

Once agents are entangled with code, tools, retrieval, permissions, and product state, prompt scoring stops making sense.

Make evals code-aware.
The behavior lives in the whole system. A good offline eval runs the real agent path and asserts on output, tool calls, files changed, structured data, or final state.

Think software testing.
This looks more like Vitest, pytest, or jest than prompt grading: representative inputs, explicit runs, normal assertions, and checks over the full trajectory.

Call it what you want.
Sentry calls this harness-backed. OpenAI calls the same idea macro evals. The important part is evaluating the agent loop.

// evals/refund_agent.eval.ts

import { expect } from 'vitest'
import { describeEval, toolCalls } from 'vitest-evals'
import { refundAgentHarness } from '../harness'

describeEval('refund agent', {
  harness: refundAgentHarness()
}, (it) => {
  it('approves refundable invoice', async ({ run }) => {
    const result = await run('Refund invoice inv_123')

    expect(result.output.status).toBe('approved')
    expect(toolCalls(result.session).map((c) => c.name))
      .toEqual(['lookupInvoice', 'createRefund'])
  })
})

Keep it local. Store evals in markdown, ask Codex to run them, and use a quick local HTML viewer for pass/fail. Raindrop Workshop makes saved trajectories easy to turn into code-aware evals.

### To raise the floor, you need to

Find unknown issues, then track when each started and the percentage of users affected. A recent start can point to a change; user impact helps prioritize.

[13:06](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=786s) [Two panels call for discovering unknown issues and identifying when each started and what percentage of users it affects.](https://assets.ai.engineer/sites/aiecode2025/sha256/c0c7ddebf31b91ae25173b4f7094bc8574692750e4f38d1e812fecc503d62855.webp)

THE REQUIREMENT

To raise the floor, you need to:

01
Discover unknown issues

02
For each issue, you need to know...

When it started
+
% of Users

## Full transcript

[0:00](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=0s) [on-hold jingle] Uh, thank you all for coming, first of all.

[0:15](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=15s) And, um, I wanna talk today about, uh, raising the floor. So it's this kind of a term we use a lot. Um, mainly I want to talk about very, very practical, like what do I actually see?

[0:30](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=30s) What do we actually see working in the real world? Um, how do people-- how are people making their agents better? So the first thing I wanna say is like, um, I could just, I could just say a bunch of stuff.

[0:39](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=39s) Like, I think, you know, um, the title of this track is like continual learning. I think it's like notable that in the real world there's really not that much continual learning, right?

[0:49](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=49s) Uh, if you look at like the labs, if you look at like products that are in the real world, you really don't see a lot of continual learning. So, um, I think it's very easy to-- Like, I could, you know, spend twenty minutes just talking about like, "Hey, here's a bunch of frameworks, here's a bunch of like

[1:03](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=63s) really nice terms." But what I would rather do is actually turn this a little bit into a dialogue. Um, this is not just because I procrastinated making a bunch of slides and because Fable was delayed and I was counting on that to make the slides, but also because like the reality is that, um, there aren't really good

[1:18](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=78s) standards for these things, right? Like, there, there's not some one-size-fits-all, uh, solution. And so I'm gonna-- I do have slides, believe it or not, but what we're also going to do is sort of like I'd like to hear from you guys, people actually building agents.

[1:31](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=91s) Like, where, where have the, you know, Twitter, you know, eval discourse, where has that failed you, right? Where, where is it not working? What are the things you're actually hitting in real life?

[1:40](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=100s) I'd like to talk about that. So please, like right now, start thinking about your questions, start thinking about the annoying parts of your flow, um, when you're building agents, and I'd like to keep a lot of time for Q&A.

[1:52](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=112s) Um, so the reality is like a year ago, agents barely existed. Like, I remember being at like a speaker dinner here like a year ago, and we're like, "Yeah, do you think like agents will like, you know, what, you know...

[2:04](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=124s) Will they like keep getting better? Will they not?" Um, and I think the crazy thing is like, you know, we're at-- we were at this point in time a year ago where everything was like a chatbot mostly, right?

[2:14](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=134s) Um, I think people here, people in this room probably, you know, were a little further ahead. Um,

[2:20](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=140s) it was, it was a lot simpler then. Uh, if you remember evals, the eval discourse for like a year or two ago, it would be like, "Oh, what is the, you know, capital of the United States?"

[2:30](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=150s) And you're like, "Yeah, and you wanna make sure that it returns Washington, D.C." And, um, that was an easier time, right? It was like

[2:38](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=158s) chatbots were so much more limited in their sort of flexibility that it was kind of easy to like, uh, to do a bunch of like, you know, fact-checking things.

[2:48](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=168s) Like you knew the answer to most questions your users would ask almost is a, is another way of saying that. At least like a, like eighty percent of them or ninety percent of them.

[2:57](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=177s) Um, but yeah, we have agents being deployed in finance, healthcare, defense. And, um, I was actually really against the word. One of my like worst takes is like, uh, early agents.

[3:07](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=187s) I was like, "Ugh, I hate the word agent." Um, I think there were a lot of people that felt the same. It was like, "Come on, it's an LLM.

[3:12](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=192s) It's a whatever." But I, I actually think it's, it's valuable because I think we're seeing that agents are this like almost self-aware entity, right? They kind of like run around their environment.

[3:21](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=201s) They have these tools they're using. Um, when they hit, you know, roadblocks, they start getting really creative, right? And that's what makes agents really powerful. But like that's also what makes them like catastrophic.

[3:31](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=211s) It's like, "Oh, well, I'll just, you know, I'll just like, uh, uh, decompile this and I'll just, you know, like do this thing that you had no idea that, uh, you could've never imagined."

[3:40](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=220s) Um, sometimes those solutions are, are helpful, right? Sometimes they're, they're actually pretty harmful. Um, but yeah. What's very certain is we've come a very long way from like next token prediction.

[3:49](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=229s) Like if you think about chatbots, it was literally just like, oh, it was like, you know, it's gonna-- what is the next likely word? It was very easy to reason about.

[3:57](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=237s) Um, and I think the thing that, you know, on the evaluation front, I think the reality is like most of the things you'd read, uh, online about evals, um, are really still like stuck in this chatbot era.

[4:09](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=249s) It's very like, well, like come up with your like one thousand, you know, eval data set. And it's like the reality is like nobody's doing that. Um, very few people anyway.

[4:17](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=257s) Sorry if you are. Um, and uh, you know, I think what teams have seen over and over again is like, yeah, you can do that, uh, but those evals like break as soon as you have a new model, as soon as you like switch harnesses.

[4:29](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=269s) Like you have a bunch of like tools that you're like, oh yeah, I'm gonna make sure that I'm gonna write an eval where it has to like call this tool if I ask it this question.

[4:35](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=275s) And it's like, oh, then you switch to like Claude Code CLI, and now eighty percent of your evals suck. And it's like, okay, you could keep doing that, but the reality is like the one thing I could promise you is that things are gonna keep changing.

[4:45](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=285s) Like we're not done. And so I'd be very careful about, you know, uh, investing, you know, months in some sort of eval set that's gonna, you know, slow you down, right?

[4:54](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=294s) I think the whole thing here is like you want more safety, but you, you don't want theater. And I think that, again, I think the evals as has been sort of, uh, prescribed by, uh, what I call like big eval, um, I, I think that there's this reality where it's kind of like, oh, you sh- really should

[5:10](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=310s) eval, but then like do you actually delay, you know, uh, including a new, you know, upgrading to the new model in your product, in your products? Do you actually delay it two weeks to update your evals or not, right?

[5:22](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=322s) I, I think most people would say no. Um, so, uh, I'm the CTO and co-founder of this company called Raindrop. Um, very quickly is like we find critical issues in production agents.

[5:34](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=334s) We verify those fixes actually work without unexpected side effects, and we also simulate changes before they land in production, um, based on past behavior. Uh, we're used by the best AI companies in the world and Fortune one hundreds.

[5:46](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=346s) A lot of logos I'm not allowed to put on here yet. Uh, but companies like Vercel, Speak, Framer. Um, and uh, I think what it means is we get this like amazing peek into like, again, what is actually working in the real world.

[6:00](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=360s) I think one of our like tenets as a company is that things are changing constantly, and we have to change what we're doing constantly. And so I think, uh, if- We, we try to be very, very honest with our-- both ourselves and our customers, like what works and what does not work.

[6:14](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=374s) We try not to sell things that don't work. Um, just again, two, two things. We have this like, uh, open source tool that like thousands of people use, maybe you use it yourself.

[6:22](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=382s) It's called Workshop. Um, so that's made by us. It's like an open source tracing tool. It's really, really cool. If you're trying to experiment with like self-healing loops, I think it's the best way to do that because if there's anything it can't do, uh, your agent can just like add it, which is pretty cool.

[6:36](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=396s) Um, and so highly, highly recommend it. Like again, I know thousands of people use it. Like people-- anyway, I bump into people all the time that use it. Raindrop is like our kind of hosted offering that does issue detection.

[6:46](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=406s) Think of it like Sentry, but detects issues for agents. Sorry. Um, and we also make howtoeval.com. And so I think it's one of the most popular resources on how to evaluate AI agents.

[6:57](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=417s) It is-- again, the link is literally in the name, it's howtoeval.com. Um, and I-it's, it's, uh, uh, our attempt at a very, very no bullshit guide at, at what actually works, and I'll be talking a little bit about it today.

[7:10](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=430s) Um, I think the like root question that we're trying to figure out today together is how do you make your, your agent better, right? It's not even like what issues does your agent have.

[7:21](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=441s) Uh, it is actually how to make your agent better because your agent will have issues that potentially you can't solve or not exactly worth solving, right? Like, um, I think that we saw this, uh, over and over again where it's like, um, you know, you can imagine that, um, th-there are some things that you're like better off

[7:38](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=458s) waiting for. Like, you know, we know Fable exists now, maybe Fable, you know-- Like should you train your own like Fable level model? It's like probably not, right? Um, and there'll be benefits, uh, when you can just incorporate that into your product.

[7:48](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=468s) Um, and so there's this actual balance, like how do, how do I actually make my agent better, uh, with the tools that I have? When-- The, the way that we start thinking about it with customers is something like this, which is like, are you a benchmark maxer or a floor raiser?

[8:04](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=484s) Um, I think that one of the problems when we talk about evaluating agents is that the terms are really confused. Like you hear OpenAI has a new like, you know, uh, eval benchmark and, um, you know, they have evals, they run evals, and then you hear like, oh, well, companies have evals.

[8:21](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=501s) There's like online evals and like-- It, it like-- The word eval is like, uh, more or less a meaningless word. It literally is just like you're evaluating something, right?

[8:30](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=510s) It's like a test in some cases. It's a-- So it's a little confusing. Um,

[8:34](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=514s) I think it's helpful-- Like I think what it means is that like, uh, companies start borrowing like the language that like labs are using and like even copying similar benchmarks, but they're doing completely different things, right?

[8:46](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=526s) Like they have completely different tools at their disposal. Like what companies are-- You know, like the companies that are downstream of models, uh, they just have very, very different responsibilities than labs.

[8:57](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=537s) Um, like labs are trying to make these super general purpose things. When they fail, at least on like a API level, when I, when I say if they get something wrong, it's like, it, it's just different.

[9:06](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=546s) Um, companies are trying to like imbue all this like company specific, uh, domain knowledge. Like, oh, here's the shape of the data, and here's what all this data means, and here's like how to access it.

[9:18](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=558s) Um, and so it's very, very different. Um,

[9:23](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=563s) we have this like funny quiz on, on the how to eval site and, um, it's interesting, right? Like it's kind of like one, one of the questions we, we would think about is like, oh, are your engineers like...

[9:33](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=573s) Or sorry, are your users like domain experts in the thing they're doing? Is-- A-a-again, is it almost like replacing someone or is it augmenting them? Because if you think about like Copilot, you know, autocomplete style or Cursor, you know, tab complete now, uh, if it gets something wrong, like you can just delete it, right?

[9:49](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=589s) Even like Claude Code, CLI or like Codex, if you're an engineer, it's like it does do things wrong all the time. Um, but then when you think about products like Devin, it's actually gets more interesting, right?

[9:59](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=599s) Like De-- I-if, if something messes up on, on the Claude Code side, it could be that like you don't have something installed correctly on your computer. There's a lot more like user error.

[10:08](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=608s) There's a lot-- You leave a lot more up to the, the users to get correctly. Um, and I think when you start thinking about things like AI doctors, for example, it's like, uh, it's a very, very different shape of responsibility as far as like how much responsibility the user has in actually getting things correctly.

[10:23](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=623s) Um, so anyways, it's kind of funny. Um,

[10:26](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=626s) just kind of breaking this down. So we think about the ceiling as like, what is the best thing, like craziest capability, emerging capability that your, your product or agent is capable of?

[10:35](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=635s) Like things that people would just not expect that it could do. And then the floor is like, what is the worst thing your agent can do? Like recommend a competitor or like delete a bunch of data or like accidentally send a, you know, AI slop email to a customer because it like technically had access to like your

[10:49](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=649s) email or something. Um, and again, I think that like the floor is very interesting because I think that that is the thing that like breaks user trust. This is the thing that like the reason why people, uh, like if you think about the worst things that could start happening in society, whether that's-- A-and, and things we've already

[11:10](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=670s) seen, um, whether that's the, uh, you know, like 4o kind of sycophancy, um, or, uh, things in that vein. A lot of it is more on like the floor side rather than the capability side.

[11:21](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=681s) Um, and anyway, so I could talk about that for a long time. Um, I'll kind of skip this.

[11:28](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=688s) So the talk obviously is gonna be about floor raising. And, um, the first thing that we're gonna talk about is like offline evals. Again, we'll keep it very simple.

[11:38](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=698s) Um, I think that like we said before, things have changed a lot sort of since this chatbot era. The sort of like, oh, you just, uh, you know, like look at, you know, string contains, you know, on the, uh, uh, on like the text output or something.

[11:52](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=712s) Or even like the style of eval tools that have like a prompt playground, like this sort of thing. Like, I actually don't know many companies that use some sort of like managed prompt like in the cloud anymore.

[12:03](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=723s) There's like one or two I can think of. Um, and the reality is just like the prompt is actually like the whole thing now. It's like all the code.

[12:11](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=731s) It's, it's your whole harness. It's like everything you're connect-- Like it's not just like some string where you tell your, the agent what to do. Um, and so what I think that means is the evals themselves actually should look a lot more like code.

[12:23](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=743s) In other words, like a lot more like tests, whether that's unit tests, whether that's end-to-end tests. Um, they should look a lot more like tests. Um-

[12:31](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=751s) Uh, Sentry has this, uh, a really cool package called, like, Vitest Evals. It's literally just, like, Vitest with, like, some syntactic sugar on top. Um, OpenAI calls this, like, macro evals and, um, again, I don't think it really matters what you call it, but, like, essentially run tests on your agent, uh, locally, uh, is, is, is the

[12:51](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=771s) advice. And keep these evals as code. Um, and again, as much as possible, like, the-- I, I don't see a lot of companies using the sort of, like, prompt playground stuff anymore because of how the shape of agents has really changed.

[13:08](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=788s) Uh, w- when we think about raising the floor, we think about really, like, three things. One is that, like, discovering all these, like, unknown issues that you have in your app, like things you're just, like, not seeing.

[13:19](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=799s) That's one. Two is that for each issue, you really need to know two things. You need to know when it actually started, and you need to know how many people it affects.

[13:27](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=807s) It sounds, like, obvious, but I promise you that, like, in the day-to-day of actually, like, having an agent, uh, you're gonna get, like, you know, you already get thousands of people like, "Oh, I saw this weird thing, I saw this weird thing."

[13:37](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=817s) So again, the first thing is, like, is this new? 'Cause if it's not new, like, I probably, like, am going to care about it less. If I-- If, if, if I tell you, like, "Hey, look, this issue started yesterday," or, "This issue started, like, three or four days ago," suddenly, like, your mind starts turning and you're like,

[13:50](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=830s) "Oh, what did I do?" Like, well, what, what, what changed, right? Did we change the model? Did we change, you know, so-something else, um, downstream? And again, the second one is, like, percent of users.

[13:59](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=839s) Like, if I'm, uh, knowing that it happened to three users versus a hundred thousand users just is, uh, critical. Because again, I think agents will have an infinite number of problems.

[14:09](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=849s) That's sort of, like, the, the great and terrible thing about them, is like, by-- they're like these little stochastic, you know, crazy things exploring everywhere. And so you just, uh, in order to even start making things better, you, you really need to, need to know these two things, when it started and percent of users.

[14:24](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=864s) Um, and I think also, and ag- another question we get a lot is, like, around, like, oh, like, I, you know, what should I be doing? And, like, the first question I always ask people is, like, how many users do you have?

[14:37](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=877s) Like, we have customers with millions of users, and we have customers with, like, five. And the real-- and, like, to be clear, like,

[14:45](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=885s) uh, th- customers with five users, like, e- especially for-- let's say it's, like, an internal, um, app in an enterprise context where it's like, you know, uh, giving, like, very critical information, like, it could be very, very important to get well, uh, or sorry, to get correctly.

[14:57](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=897s) But, um, it does mean you just, like, should be taking a radically different, uh, approach. Like, so for example, on the, like, you know, uh, let's say, like, ten, twenty, a hundred million, you know, messages a day side of things, like, experiments become extremely valuable.

[15:12](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=912s) Uh, if you have a free tier, you can, like, uh, uh, run experiments on a very small sample of your free tier, um, and that can just be extremely, extremely useful.

[15:20](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=920s) Um, obviously, if you have five or ten users, like, uh, I would not recommend, you know, experiments or A/B tests, et cetera. Um, so, uh, this is one of those things that, like, uh, uh, really, really depends on the person.

[15:31](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=931s) Um, what I wanna talk about now before we get into Q&A are, like, three very, very, very, very tactical lessons on the sort of, like, issue, uh, discovery and analysis side.

[15:44](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=944s) These are, like, three things that I, I've never heard anyone talk about, like, three things that we've sort of just discovered from first principles as we do stuff at Raindrop.

[15:54](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=954s) So any competitors in the audience, please pay attention. This is very important. Um, the first one is that clusters are not issues.

[16:04](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=964s) So, um, the sort of, like, naive approach that we've seen either customers or also sometimes competitors, uh, take is like, well, like, you just take all the traces and you just cluster it, right?

[16:15](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=975s) And you get these, like, clusters. It could be, like, useful from, like, an analysis, uh, you know, like, Hamel calls this, like, error analysis. Like, the, the, the, this, you know, finding these clusters of things.

[16:25](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=985s) It could be useful to see, like, whoa, what's going on in your data, right? Going from, like, a bunch of logs to, like, something. Um,

[16:33](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=993s) the problem is that, like, and a- and again, I, I have here, it's useful for one-off analysis, but it just doesn't really scale well. Um, and there's, like, a very good reason why we also, you know, if you think about, like, normal telemetry, we, we, we try to think a lot about, like, normal telemetry.

[16:46](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1006s) What are the analogies? There's a reason why you don't sort of, like, take all of your, you know, normal logs and just, like, start clustering it, right? Because when you're building software, you, you need to know, like, when something started, um, you need to know how much it's grown.

[17:01](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1021s) Um, those things really matter. So again, with clusters, it's very, very hard to reliably, uh, track over time. Um, like, if, uh, like, again, this is called, like, temporal clustering, and there's, like, research in this, but, like, it's pretty hard, um, to do reliably.

[17:17](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1037s) You also just, like, don't have control of boundaries, um, and this also changes a lot depending on your product. Like, um, what you consider to be, like, you know, the same issue or not, um, is actually very, very unique to every company.

[17:34](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1054s) And so you sort of will get these, like, kind of weird clusters. Like, you know, you can imagine each of these as, like, oh, wrong price quoted and wrong, wrong refund calculated are, like, actually, like, you'll get a cluster like, you know, uh, uh, you know, price issues or something.

[17:51](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1071s) And it's like, yeah, sort of, but, like, actually these could have, like, extremely different root causes, right? So, so price issues is, or, like, you know, uh, issues calculating is, like, you know, as a cluster, it's not really that useful.

[18:04](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1084s) Um, and it also, again, doesn't, doesn't really tell you the things that, you know, we talked about needing. Um,

[18:11](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1091s) so, uh, yes, uh, last one here is gonna be, uh, code mode actually really scales. Like, you've heard about code mode in the context of MCPs. Um,

[18:22](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1102s) I highly recommend just trying to apply this to traces. Like, you can just write, uh, these classifiers, and you can write them, and you can run them in a sandbox, and you can run them at production scale.

[18:31](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1111s) Um, we have a, you know, feature that makes this easier, but, like, you can do this, so I highly recommend it.

[18:38](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1118s) The last lesson here is that agents are very, very bad at anomaly detection, so don't ask your agent to find anomalies. Uh, ask it to investigate anomalies you've al-already found.

[18:47](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1127s) So, uh, what I mean is, like, pull out as many de-deterministic things as you can, like keyword frequency, right? So if you see a spike in, like, a keyword,

[18:56](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1136s) it doesn't necessarily mean that there's an issue, but it does mean that you can, uh, it's, like, something more tangible, tractable that you can have an agent actually investigate.

[19:06](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1146s) Um, and I'm gonna skip through the rest because we're tight on time, and I lied to you, uh, which is that we're not gonna have enough time for Q&A because I only have a minute left.

[19:16](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1156s) But what I'd love if you could do is, uh, find me after. Um, I'll be around for the next hour, and, uh, let's just talk. It's probably a better format than standing up here, and it'll be hard to hear your questions anyway.

[19:26](https://www.youtube.com/watch?v=jHMiYtjoJfA&t=1166s) So, uh, yeah. Thank you guys so much. [outro music]
