# How agent o11y differs from traditional o11y

Phil Hetzel

AI Engineer Europe 2026

[Talk page](https://ai.engineer/talks/XBaznoTRDFI-agent-o11y-differs-from-traditional-o11y) · [Video](https://www.youtube.com/watch?v=XBaznoTRDFI)

## How agent o11y differs from traditional o11y

Agents need more than latency and error monitoring: their variable behavior, text-heavy traces, and domain-specific failures change both the storage system and the improvement workflow.

Familiarity with HTTP requests, LLM prompts, and tool calls will help; traces and spans are defined in the article.

### Why does a working prototype stall before production?

[0:52](https://www.youtube.com/watch?v=XBaznoTRDFI&t=52s)

A generative AI prototype works well enough to demonstrate. What would make you confident enough to put it into production? Phil Hetzel encountered that gap repeatedly during twelve years in consulting and systems implementation, including leading Slalom’s Global Databricks practice: customers were prolific at producing proofs of concept, but much less successful at deploying them. He began using Braintrust to address that problem before joining the company, where he leads solutions engineering.

The confidence problem runs in two directions. Once an agent is live, you need to know whether it performs as well as expected. When you change it, you need evidence that the next version deserves your confidence. Braintrust’s framing of agent quality connects these two activities: monitoring production behavior and evaluating experimental changes.

![Slide titled “What is Braintrust?” with three quality questions and a diagram linking AI in an app, scores, and three prompt variants.](https://ai.engineer/api/talk-essay-media/4302cc63379f3cd1b8d10af330f619e4dc5597a485f0fe45832ff1c041bae7d5)

Braintrust’s overview connects AI quality questions with evaluation, monitoring, and prompt comparison.

### Keep the operational measurements

[2:48](https://www.youtube.com/watch?v=XBaznoTRDFI&t=168s)

If your team already uses Grafana or Datadog, why introduce another observability system? Those tools already answer essential operational questions: is the application available, how long do interactions take, and are users encountering HTTP 400- or 500-level errors? Hetzel notes that Braintrust itself uses Datadog to monitor website errors. Agent observability does not eliminate that need.

![Slide listing uptime, technical performance, bugs, and computational cost under the scope of traditional observability.](https://ai.engineer/api/talk-essay-media/b52193503230d5211c899a950bcdf8e52ba7047ae78957b2820b74f9da684a1a)

Traditional observability asks whether the system is operational.

The underlying vocabulary carries over, too:

| Primitive | What it represents |
| --- | --- |
| Metric | A measurement, such as latency or error count, aggregated over time |
| Trace | One complete workflow interaction |
| Span | One step within that interaction |

An agent still performs a workflow made of steps. The difference is what you must capture and evaluate inside those steps.

### A fast response can still be wrong

[4:56](https://www.youtube.com/watch?v=XBaznoTRDFI&t=296s)

Conventional application code deliberately constrains control flow. An agent can choose different paths while attempting the same task—the flexibility that makes LLMs useful also makes their behavior harder to characterize. Observability must therefore help explain why the agent selected one path rather than another, not merely how long the selected path took.

Time to first token, total tokens, duration, and latency remain useful. But a response can arrive promptly and still fail the task. Quality measurements ask additional questions:

- **Grounding:** Does the answer follow from the context the application gathered?
- **Tool selection:** Did the agent use the tools expected for this request?
- **Brand alignment:** Does the response follow the standards in the system prompt?

**Technical success does not establish behavioral quality.** To evaluate these questions, the trace must retain the relevant context, tool interactions, instructions, and response content. That is much richer evidence than a duration or status code.

### Large traces, immediate inspection

[7:22](https://www.youtube.com/watch?v=XBaznoTRDFI&t=442s)

Agent traces combine a semi-structured hierarchy with substantial unstructured text. Hetzel reports seeing customer agent traces larger than one gigabyte and individual spans as large as twenty megabytes. These are reported examples, not typical trace sizes; he does not specify their composition or measurement method. The systems challenge includes ingesting and processing those payloads, but also making them usable.

As usage grows, AI engineers and product managers still expect to inspect interactions in real time. In the trace example Hetzel shows, the model calls and tool calls form a collection of spans, with large amounts of text inside those spans. A trace viewer has to expose both the execution structure and the content needed to understand the result.

The same data must support two read patterns:

- **Interactive inspection:** A user interacts with the agent, and an engineer wants to see that interaction almost immediately.
- **Programmatic improvement:** A developer issues SQL queries through a CLI and uses observability or evaluation traces to improve the application automatically.

Serving a recent interaction quickly and querying a large collection of traces are different access requirements. Both have to work against the same bulky records.

### Visibility, indexing, and text search

[9:53](https://www.youtube.com/watch?v=XBaznoTRDFI&t=593s)

Braintrust built a database specifically for this workload. The architecture separates immediate visibility from indexed query performance: incoming data enters a **write-ahead log**, while indexing supports filtering and analytical queries. The related [Brainstore architecture](https://www.braintrust.dev/blog/brainstore) explains the visibility mechanism: queries can read committed log entries alongside indexed data before background indexing finishes. Immediate visibility here means avoiding a wait for that indexing step, not zero ingestion latency.

Text search is a central requirement. Hetzel says Braintrust forked [Tantivy](https://github.com/quickwit-oss/tantivy) to provide text indexing. His example is straightforward: find every trace containing the word `Amazon`. Answering that request efficiently across large trace bodies requires an index over their text, rather than relying only on structured fields such as duration or error status.

For a simple relational representation with one text row per span, the search intent can be expressed in SQL:

```sql
SELECT DISTINCT trace_id
FROM span_text
WHERE text_content LIKE '%Amazon%';
```

Here, `span_text` names an illustrative table, and `DISTINCT` returns each matching trace once even if several spans match. The query expresses a substring search; a full-text index supplies its own token-matching semantics and avoids treating every search as a scan through every text body. The important requirement is retrieval by content across the trace.

When an audience member asks whether Tantivy is like OpenSearch, Hetzel compares it more directly to Apache Lucene, but written in Rust. The log, indexes, and text-search capabilities then need a unified SQL or SQL-like query interface. That is the purpose of the architecture diagram: incoming traces become visible, acquire indexes, and remain accessible through a common query path with results streamed to the UI.

![Diagram showing incoming traces, a write-ahead log, background indexing, Tantivy indexing, unified queries, and streamed results to the UI.](https://ai.engineer/api/talk-essay-media/67a718ab733e2c2e59e5d2e33a79f6e93f3c0c7a66d341081862e252aaea4141)

Brainstore’s architecture links incoming traces to indexing, queries, and streamed results.

### The people who can recognize a failure

[11:45](https://www.youtube.com/watch?v=XBaznoTRDFI&t=705s)

Operational observability usually serves systems engineers and product engineers. Agent quality brings in another group: people close to the users or the problem domain. Natural-language prompts give those contributors a direct way to influence behavior. Hetzel describes clinicians, registered nurses, wealth advisors, and lawyers inspecting traces to improve agents, and attributes stronger agent teams to collaboration between technical and nontechnical contributors. The person best equipped to recognize a bad answer may not be the person who operates the infrastructure.

This also connects observability to evaluations. In Braintrust’s framing, evaluations use the same basic machinery but run in batches with inputs known ahead of time. Trace storage, quality assessment, and human review are useful in both settings. The depth of the platform follows from supporting both complex data and the different people who need to work with it.

### What are people doing with the agent?

[13:50](https://www.youtube.com/watch?v=XBaznoTRDFI&t=830s)

Once traces contain the substance of user interactions, another question becomes possible: can the system explain how people are using the agent? Hetzel describes a capability rolled out in Braintrust’s SaaS offering roughly a month before the talk. At the level described in the recording, a lightweight LLM, embeddings, and clustering support topic modeling over incoming traces. The resulting analysis surfaces user intent, sentiment, and possible issues.

The purpose is to shorten the distance between a problem observed in production and a fix tested through experimentation. Instead of requiring someone to discover every recurring issue by reading individual traces, grouping interactions can help reveal where to investigate next.

### From expert judgment to automated scores

[15:24](https://www.youtube.com/watch?v=XBaznoTRDFI&t=924s)

The audience’s first distinction is between functional quality and non-functional performance. Hetzel agrees that traditional observability can measure technical performance; agent tracing also captures prompts, duration, time to first token, and cache hits alongside quality measures. These measurements are complementary: one set describes the execution, while the other assesses whether the agent did a good job.

A question about human annotation on the iceberg slide makes the quality workflow concrete. A product manager or domain expert reviews an incoming trace, but the useful artifact is more than a grade:

1. **Inspect the interaction.** Determine whether the agent performed well or poorly.
2. **Explain the judgment.** Record why the behavior deserves that grade.
3. **Identify the failure mode.** Use those explanations, potentially with an LLM, to help construct scalable scoring functions.
4. **Automate the recurring check.** Turn the recognized failure mode into something future traces can be scored against.

Human annotation helps discover what the automated evaluator should look for. The justifications connect domain expertise to repeatable quality checks.

### Bring production failures into experiments

[17:12](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1032s)

The next question joins two concerns: integrating with agent frameworks for offline optimization, and choosing a custom database instead of an OLAP system such as ClickHouse. Hetzel says Braintrust previously used ClickHouse and moved away from it. He describes the founder as an early SingleStore employee, with the background to undertake database development, but gives a specific workload reason for the decision: the text indexes they needed were not adequately available in ClickHouse **at that time**. This is a historical explanation, not a measured efficiency comparison or a claim about present-day ClickHouse capabilities.

For the optimization question, the distinction is when the inputs become available:

| Workload | Inputs | Execution |
| --- | --- | --- |
| Evaluations | Known in advance | Batch |
| Production observability | Arrive from users | Real time |

Braintrust uses the same system for both. That shared infrastructure makes a production interaction usable as evidence in an offline experiment.

When the questioner presses for the practical integration step, Hetzel gives a direct workflow: capture the trace, add it to an offline dataset, and experiment on it. The handoff is the recorded interaction becoming a test case. His answer specifies that dataset workflow rather than a framework-specific integration procedure.

### Leave room for failures you have not defined

[19:27](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1167s)

The final question asks whether observability results must be quantitative. User satisfaction might be a number, but could a result instead be prose or a flag? Hetzel separates two kinds of analysis. Online scoring addresses known unknowns: you know the dimension you want to measure, even though you do not yet know how this interaction performed. Open-ended analysis addresses unknown unknowns: it derives insights without requiring every issue to have a predefined score.

**Scores measure the failures you have learned to recognize; open-ended inspection helps you discover the next ones.** A quality workflow needs space for both, so its understanding of agent behavior can grow beyond the checks already written.

## Resources

- [Brainstore architecture](https://www.braintrust.dev/blog/brainstore) — Explains Brainstore’s object storage, write-ahead log, Tantivy indexing, and specialized trace queries.
- [Tantivy search library](https://github.com/quickwit-oss/tantivy) — Rust full-text search library inspired by Apache Lucene, with examples and development documentation.
- [Introducing Braintrust Topics](https://www.braintrust.dev/blog/topics) — Introduces automated grouping of production traces by user tasks, sentiment, and agent issues.
- [Inside the Topics pipeline](https://www.braintrust.dev/blog/topics-architecture) — Details how trace summaries become embeddings, clusters, named topics, and classifications.
- [Build evaluation datasets from production traces](https://www.braintrust.dev/docs/annotate/datasets) — Instructions for assembling versioned test cases and adding production traces to evaluation datasets.
- [Filter and query Braintrust logs](https://www.braintrust.dev/docs/observe/filter) — Search production logs through filters, SQL, and the CLI for debugging and dataset creation.
- [ClickHouse full-text search availability](https://clickhouse.com/blog/full-text-search-ga-release) — Documents production availability of ClickHouse’s native inverted text indexes in March 2026.

## Slides

### AI Engineer Europe

The recording opens with the AI Engineer Europe logo.

[0:00](https://www.youtube.com/watch?v=XBaznoTRDFI&t=0s) [AI Engineer Europe logo in white on black.](https://assets.ai.engineer/sites/aiecode2025/sha256/24e440bf5dfd4ecccd4fffdc9ef9243be07fff10b16e87c6b80fbc594828e3ea.webp)

AI Engineer
EUROPE

### Presenting Sponsor

[0:05](https://www.youtube.com/watch?v=XBaznoTRDFI&t=5s) [Google DeepMind presenting sponsor card.](https://assets.ai.engineer/sites/aiecode2025/sha256/e0a2733394b9e76d5cbcc292edcb1e7d0d4305725368334cfa74aa4dc6f38f30.webp)

PRESENTING SPONSOR

Google DeepMind

### Platinum Sponsors

Braintrust, WorkOS and OpenAI are listed as platinum sponsors.

[0:10](https://www.youtube.com/watch?v=XBaznoTRDFI&t=10s) [Platinum sponsors Braintrust, WorkOS and OpenAI.](https://assets.ai.engineer/sites/aiecode2025/sha256/0c718094ceabaf063b52992651f1bd938114cabd140b3cfee0a6769ea91a2d3e.webp)

PLATINUM SPONSORS

Braintrust
WorkOS
OpenAI

### How Traditional Observability Differs from Agent Observability

The talk compares traditional and agent observability, with a theoretical focus.

[0:25](https://www.youtube.com/watch?v=XBaznoTRDFI&t=25s) [Blue Braintrust opening slide for a talk on traditional and agent observability.](https://assets.ai.engineer/sites/aiecode2025/sha256/b28062df364b920ef08d4f10bf1f474ed9332934aff232bbc5b0cae9a1e1fbe2.webp)

Braintrust

How traditional o11y differs
from agent o11y

Ship quality AI
braintrust.dev

AIE
Google DeepMind
AI Engineer
EUROPE

### Agenda

After a brief introduction, the talk takes a theoretical look at how traditional and agent observability differ and what comes next.

[0:30](https://www.youtube.com/watch?v=XBaznoTRDFI&t=30s) [Agenda: introduction, overview, traditional versus agent observability, and what’s next.](https://assets.ai.engineer/sites/aiecode2025/sha256/f72d1592ce579b3c43feeef65129e2db14b4f8c6fd828cb80f3d1c64ddc3384e.webp)

Agenda
Intro
Overview
Traditional vs. agent o11y
What’s next

Confidential

AIE
AI Engineer EUROPE
Google DeepMind

### Phil Hetzel

Phil Hetzel leads solutions engineering at Braintrust. After 12 years in consulting and implementation, he saw customers build generative AI proofs of concept more readily than bring them to production.

[0:59](https://www.youtube.com/watch?v=XBaznoTRDFI&t=59s) [Phil Hetzel introduces his professional background and interests beside a photo of himself with his dachshund.](https://assets.ai.engineer/sites/aiecode2025/sha256/c7ae93b543a011d9899838d4e7e3985b160f4e70bddaad52957384150c0d8b5c.webp)

Braintrust

Phil Hetzel
Head of Solution Engineering, Braintrust

- Twelve years in consulting/implementation
- Former leader of Slalom’s global Databricks business unit
- Likes to play chess (poorly) and spend time with wife and dachshund (Pistol Pete, pictured)

Why building eval platforms is hard
Ship quality AI
braintrust.dev

Broadcast text:
AIE
AI Engineer EUROPE
Braintrust
WorkOS
OpenAI

### What is Braintrust?

Braintrust evaluates agent changes and monitors production responses. Its diagram connects an AI app to scores and prompt comparisons in a cycle for improving quality.

[1:50](https://www.youtube.com/watch?v=XBaznoTRDFI&t=110s) [Overview of evaluations, production monitoring and scorer refinement, with a diagram connecting an AI app, scores and prompts.](https://assets.ai.engineer/sites/aiecode2025/sha256/3c7eef2942bf0f5b877b8b2a4d846913be13a30ad41ed69bed240f8ef2ec7fa7.webp)

What is Braintrust?

Agents fail in unpredictable ways. Evals and observability catch issues before users do.

How do you know your AI feature works?
Eval test your AI with real data and score the results. You can determine whether the results improve or hurt performance.

Are bad responses reaching users?
Production monitoring tracks live model responses and alerts you when quality drops or incorrect outputs increase.

Can your team improve quality without guesswork?
Loop builds and refines scorers to measure the specific quality metrics that matter for your application.

AI IN YOUR APP

SCORES
✓ 98% Toxicity
✓ 83% Accuracy
× 74% Hallucination

✓ PROMPT A
× PROMPT B
× PROMPT C

Confidential

Interface text:
AIE
AI Engineer EUROPE

### Traditional observability is established

Existing observability tools help applications run at scale. The speaker asks whether they solve the same problem for agents, and argues that agent observability has a different scope.

[2:41](https://www.youtube.com/watch?v=XBaznoTRDFI&t=161s) [Traditional observability is necessary at scale, established tools already exist, and the slide asks whether agent observability is the same problem.](https://assets.ai.engineer/sites/aiecode2025/sha256/075671f6c33b7943282b37ec384b736740c3b19ccd56fe1616843a6b73f81890.webp)

Traditional observability is established

• Necessary to operate applications at scale
• Everyone has already established o11y tools
• Is agent o11y just the same problem?

Confidential

Other visible text:
AIE
AI Engineer EUROPE
Engineering the future of AI

### Agent applications are non-deterministic

The speaker contrasts predictable application paths with agents that may take different routes, arguing that agent observability must measure more than constrained, known metrics.

[5:22](https://www.youtube.com/watch?v=XBaznoTRDFI&t=322s) [Bullets describe deterministic application paths and testing; a callout says traditional observability measures deterministic metrics and code paths.](https://assets.ai.engineer/sites/aiecode2025/sha256/65fdc649a8f6e71e5a27c6660350dfabd55418eb000d6f440346eb18e07d8003.webp)

Agent applications are non-deterministic

• Typical applications have deterministic code paths.
• There could be control flow but given some input, you will always traverse the same path.
• Related - unit testing these applications is straightforward and also deterministic

Traditional observability is meant to measure deterministic metrics and codepaths

Confidential

AIE
AI Engineer EUROPE

### AI observability measures functional quality

A complex agent trace feeds both technical metrics and quality scores. The latter assess whether the agent used tools appropriately and grounded its answers in context.

[5:52](https://www.youtube.com/watch?v=XBaznoTRDFI&t=352s) [An agent trace branches to quality scores including factuality and tool use, and technical metrics including token counts and duration.](https://assets.ai.engineer/sites/aiecode2025/sha256/b636a341a219b44060e571272e2247cc37cd268110b3d8033de0f3da623dd020.webp)

Agent applications are non-deterministic

AI Observability is not just testing the technical metrics, also testing the functional quality

Complex agent trace

Online scoring functions
Factuality
Tool Use
Faithfulness
Brand Alignment
Context Relevance

Technical metrics
Time to first token
Total tokens
Duration

Confidential

Interface text:
AIE
AI Engineer EUROPE

### Problem 2: Agent traces are nasty

Agent traces combine semi-structured data with substantial unstructured text. The speaker has seen traces exceed a gigabyte; individual spans can be 20 MB.

[7:22](https://www.youtube.com/watch?v=XBaznoTRDFI&t=442s) [A dark section slide introduces the problem of difficult agent traces.](https://assets.ai.engineer/sites/aiecode2025/sha256/6a644587714a64230a99d02a4bd76b9c1222159411054283d8bd442f10975454.webp)

Problem 2: Agent traces are nasty

Confidential

AIE
AI Engineer
EUROPE
Google DeepMind

### The more the merrier

Agent observability brings technical and non-technical people together: engineers track performance and bugs, while people closer to users or the problem domain can judge traces and identify failure modes.

[12:13](https://www.youtube.com/watch?v=XBaznoTRDFI&t=733s) [Two columns describe technical and non-technical roles in agent observability.](https://assets.ai.engineer/sites/aiecode2025/sha256/c4615ce91b2b7d1b8b50cc2bea8202f51fe1c525bc83e5e2572539a8333a0b6d.webp)

The more the merrier

In-scope for agent o11y

Technical team members
• Gathers performance and uptime metrics
• Uses these to optimize technical performance of application
• Uncovers bugs
• Ultimately to avoid 400/500 errors and poor customer experiences

Non-technical team members
• Uncovering failure modes of an agent
• Annotating and judging traces
• Implementing agent quality checks
• “Rerunning” production inputs in offline evaluations

Confidential

Interface text:
AIE
AI Engineer EUROPE
Engineering the future of AI

### It takes a lot to complete the flywheel

An iceberg places execution, input examples and an output UI above water, with tracing, scoring, governance and other components below. The speaker treats evals and observability as closely related problems.

[13:06](https://www.youtube.com/watch?v=XBaznoTRDFI&t=786s) [An iceberg diagram lists components needed for an agent evaluation and observability platform.](https://assets.ai.engineer/sites/aiecode2025/sha256/2bfeeea741b8485172a746e2d9de57e5b59e58baee8964edd1805029c4eea968.webp)

It takes a lot to complete the flywheel

Agent/prompt/model executor
Input examples
UI to see outputs
Human annotation
Tracing
Prompt store
Observability
Prompt playgrounds
Online scoring
Custom trace views
Alerting
RBAC/Governance
Specialized database
Agent-forward workflows
Agent playgrounds
Model gateway
Topic modeling

Confidential

Interface text:
AIE
AI Engineer
EUROPE

### Looking forward

Braintrust groups agent traces into topics to reveal how people use an agent. The slide also calls for evaluation platforms that uncover unknown issues and are easy for humans and agents to onboard onto.

[13:47](https://www.youtube.com/watch?v=XBaznoTRDFI&t=827s) [The slide calls for insights into unknown agent issues and easy onboarding, alongside a topic overview.](https://assets.ai.engineer/sites/aiecode2025/sha256/8db897b39939c4887f1d3786501cf412e6b8c6a9ec94b1966c43b8b2b0c0cdb3.webp)

Looking forward

Once achieving the ability to operate at scale, your evals platform must:
• Deliver insights about the unknown-unknowns of your agents
• Be easy to onboard onto: for humans and for agents

Task v2
17.9%  35 items
Credit balance and utilization
credit  balance  current  check  utilization  account  usage  details
limit  understand

Samples
User wants to check their current credit balance and utilization.
User wants to check their current credit balance and utilization.
User wants to check their current credit balance and utilization.
The user wants to understand their current credit usage and available balance.
User wants to check their current credit balance and utilization details.
+30 more samples

13.3%  26 items
Failed payment verification

13.3%  26 items
Refund requests

10.2%  20 items
Installment plan adjustments

Confidential

Interface text:
Braintrust Demos  [illegible]-customer-service  Topics
Topic map
Sentiment
Issues
Task
Task v2
Issues v2
Topics  Edit
Generate
View 10 topics as list
Trace  Timeline  Thread  Views
Find
Go to logs topic view
Configuration
6 automated facets or [illegible]
QUERIES  87
View3

AIE
AI Engineer EUROPE
Engineering the future of AI

### Looking forward

Braintrust groups agent traces into topics to reveal how people use an agent. The demo connects a credit-balance topic to an example conversation and automated scores; the slide also calls for easy onboarding.

[13:56](https://www.youtube.com/watch?v=XBaznoTRDFI&t=836s) [A topic overview displays an opened example conversation with automated scores and a response.](https://assets.ai.engineer/sites/aiecode2025/sha256/30cfdd58df9dea2edd6a0007aea1ff100d00c774152b127a4e254f39264c8d69.webp)

Looking forward

Once achieving the ability to operate at scale, your evals platform must:
• Deliver insights about the unknown-unknowns of your agents
• Be easy to onboard onto: for humans and for agents

Interface text:
Braintrust Demos / l8r-customer-service / Topics
Topic map
Sentiment
Issues
Task
Task v2
Issues v2
Task v2
Topics  Edit
Generate  View 10 topics as list
17.9%  35 items
Credit balance and utilization
credit  balance  current  check  utilization  account  usage  details  limit  understand
Samples
User wants to check their current credit balance and utilization.
User wants to check their current credit balance and utilization.
User wants to check their current credit balance and utilization.
The user wants to understand their current credit usage and available balance.
User wants to check their current credit balance and utilization details.
+30 more samples
13.3%  26 items
Failed payment verification
13.3%  26 items
Refund requests
10.2%  20 items
Installment plan adjustments
Trace  Timeline  Thread  Views
CONVERSATION
2 messages  2 tool calls
Add Justification
AUTOMATED SCORES
[illegible] Topic Adherence  100%
Politeness & Tool Accuracy  100%
Response Conciseness  100%
What's my current balance?
get_account_balance  Calling
get_account_balance  Done  Show result
FINAL RESPONSE
Your current balance details are as follows:
Credit Limit: $5,000.00
Used Credit: $2,152.50
Available Credit: $2,847.50
Credit Utilization: 43%
Confidential
AIE
AI Engineer EUROPE
Engineering the future of AI

[14:20](https://www.youtube.com/watch?v=XBaznoTRDFI&t=860s) [Slide describing evaluation platforms that reveal unknown agent issues and support onboarding, alongside a topic list.](https://assets.ai.engineer/sites/aiecode2025/sha256/108d96cd5f6e786298c8cdaadc2d9dcc9327458b70e801022bf66f8d3e67c524.webp)

Looking forward

Once achieving the ability to operate at scale, your evals platform must:
• Deliver insights about the unknown-unknowns of your agents
• Be easy to onboard onto: for humans and for agents

Interface text:
Braintrust Demos / l8r-customer-service / Topics
Topic map
Sentiment
Issues
Task
Task v2
Issues v2
Task v2
Topics  Edit
Generate  View 10 topics as list
17.9%  35 items
Credit balance and utilization
credit  balance  current  check  utilization  account  usage  details  limit  understand
Samples
User wants to check their current credit balance and utilization.
User wants to check their current credit balance and utilization.
User wants to check their current credit balance and utilization.
The user wants to understand their current credit usage and available balance.
User wants to check their current credit balance and utilization details.
+30 more samples
13.3%  26 items
Failed payment verification
13.3%  26 items
Refund requests
10.2%  20 items
Installment plan adjustments
Trace  Timeline  Thread  Views
Confidential
AIE
AI Engineer EUROPE
Google DeepMind

### Looking forward: conversation view

Braintrust clusters agent traces to surface usage topics. The demo pairs a credit-balance topic with an example conversation and automated scores.

[14:21](https://www.youtube.com/watch?v=XBaznoTRDFI&t=861s) [The same evaluation-platform slide with a conversation and automated scores visible in the embedded demo.](https://assets.ai.engineer/sites/aiecode2025/sha256/0a1d3e2f04faa96ffe9cb953bc770013a5595c58dc5f44cb510499d62656c685.webp)

Looking forward

Once achieving the ability to operate at scale, your evals platform must:
• Deliver insights about the unknown-unknowns of your agents
• Be easy to onboard onto: for humans and for agents

Task v2
17.9%  35 items
Credit balance and utilization
credit  balance  current  check  utilization  account  usage  details  limit  understand

Samples
User wants to check their current credit balance and utilization.
User wants to check their current credit balance and utilization.
User wants to check their current credit balance and utilization.
The user wants to understand their current credit usage and available balance.
User wants to check their current credit balance and utilization details.
+30 more samples

13.3%  26 items
Failed payment verification
13.3%  26 items
Refund requests
10.2%  20 items
Installment plan adjustments

CONVERSATION
2 messages · 2 tool calls
AUTOMATED SCORES
[illegible] Topic Adherence  100%
Politeness & Tool Accuracy  100%
Response Conciseness  100%

What’s my current balance?
get_account_balance  Calling
get_account_balance  Done  Show result

FINAL RESPONSE
Your current balance details are as follows:
Credit Limit: $5,000.00
Used Credit: $2,152.50
Available Credit: $2,847.50
Credit Utilization: 43%

Confidential

Interface text:
Braintrust Demos  L8r-customer-service  Topics
Topic map  Sentiment  Issues  Task  Task v2  Issues v2
Topics  Edit  Generate  View 10 topics as list
Trace  Timeline  Thread  Views
Add Justification
Go to logs topic view
Configuration
View3
AIE
AI Engineer EUROPE
Google DeepMind

### Looking forward: topic list

The topic list groups agent traces by user intent, including credit balance and failed payment verification. The goal is to shorten the loop from production problems to fixes tested in experiments.

[14:51](https://www.youtube.com/watch?v=XBaznoTRDFI&t=891s) [Evaluation-platform slide with a topic list and the conversation panel closed.](https://assets.ai.engineer/sites/aiecode2025/sha256/e966f5a7dfb168fe34070cdf947e0a0b46b36a5e11fd7bf894a0ffc3a48bc786.webp)

Looking forward

Once achieving the ability to operate at scale, your evals platform must:
• Deliver insights about the unknown-unknowns of your agents
• Be easy to onboard onto: for humans and for agents

Topic list:
17.9% — 35 items
Credit balance and utilization
credit  balance  current  check  utilization  account  usage  details  limit  understand
Samples:
User wants to check their current credit balance and utilization.
User wants to check their current credit balance and utilization.
User wants to check their current credit balance and utilization.
The user wants to understand their current credit usage and available balance.
User wants to check their current credit balance and utilization details.
+30 more samples

13.3% — 26 items
Failed payment verification

13.3% — 26 items
Refund requests

10.2% — 20 items
Installment plan adjustments

Confidential

Interface text:
Braintrust Demos  [illegible]-customer-service  Topics
Topic map
Sentiment
Issues
Task
Task v2
Issues v2
Topics  Edit
Generate
View 10 topics as list
Trace  Timeline  Thread  Views
Find
Go to logs topic view
Configuration
6 automated facets or [illegible]
QUERIES 87
View3
AIE
AI Engineer Europe
Braintrust  WorkOS  OpenAI

### Looking forward: conversation view

A topic list sits beside a sample conversation and automated scores. Hetzel aims to shorten the loop between spotting production problems and testing fixes.

[14:58](https://www.youtube.com/watch?v=XBaznoTRDFI&t=898s) [Evaluation-platform slide with the embedded conversation panel open again.](https://assets.ai.engineer/sites/aiecode2025/sha256/29bfcd8f72c84b1e036789ab5f28c83538b377676180121fecc8c27697e697df.webp)

Looking forward

Once achieving the ability to operate at scale, your evals platform must:
• Deliver insights about the unknown-unknowns of your agents
• Be easy to onboard onto: for humans and for agents

Confidential

Interface text:
Braintrust Demos   l8r-customer-service   Topics
Topic map
Sentiment
Issues
Task
Task v2
Issues v2

Task v2
Topics   Edit
Generate   View 10 topics as list

17.9%   35 items
Credit balance and utilization
credit   balance   current   check   utilization   account   usage   details   limit   understand
Samples
User wants to check their current credit balance and utilization.
User wants to check their current credit balance and utilization.
User wants to check their current credit balance and utilization.
The user wants to understand their current credit usage and available balance.
User wants to check their current credit balance and utilization details.
+30 more samples

13.3%   26 items
Failed payment verification
13.3%   26 items
Refund requests
10.2%   20 items
Installment plan adjustments

Trace   Timeline   Thread   Views
CONVERSATION
2 messages · 2 tool calls
Add Justification
AUTOMATED SCORES
[illegible] Topic Adherence   100%
Politeness & Tool Accuracy   100%
Response Conciseness   100%
What's my current balance?
get_account_balance   Calling
get_account_balance   Done   Show result
FINAL RESPONSE
Your current balance details are as follows:
Credit Limit: $5,000.00
Used Credit: $2,152.50
Available Credit: $2,847.50
Credit Utilization: 43%

Braintrust   WorkOS   OpenAI
AI Engineer EUROPE

### Thank you

Phil Hetzel closes with a QR code inviting viewers to start an agent quality journey.

[15:06](https://www.youtube.com/watch?v=XBaznoTRDFI&t=906s) [Braintrust closing slide with speaker contact information and a QR code.](https://assets.ai.engineer/sites/aiecode2025/sha256/cb02b8ef868c00cd915ee9bae950436e4fb78c1fb4b51f272f968524e54cda19.webp)

Braintrust

Thank you.

Phil Hetzel
Head of Solutions Engineering
[redacted]

Start your agent quality journey for free today

Why Building Eval Platforms Is Hard
Ship quality AI
braintrust.dev

Interface text:
AIE
AI Engineer EUROPE

### Reviewing a conversation trace

Hetzel opens a customer-service conversation beside automated scores and rating controls. He suggests using expert justifications to develop more scalable scoring functions.

[16:24](https://www.youtube.com/watch?v=XBaznoTRDFI&t=984s) [Braintrust logs page showing a list of conversation traces.](https://assets.ai.engineer/sites/aiecode2025/sha256/0a096b9d8b2340c18c4963ba881ff069c5abbbc3efb869cecf58b1d9c48add48.webp)

Logs
All logs view
Filter
Display
Search traces
Past 3 days
Timeline: Tue 07, 12 PM, Wed 08, 12 PM, Thu 09, 12 PM

5 traces
Created | Name | Input | Output | Expected | Tags
17h ago | conversat... | [illegible] | I understand t... | – | –
17h ago | conversat... | [illegible] | Since you hav... | – | –
18h ago | conversat... | [illegible] | Once your ref... | – | –
19h ago | conversat... | [illegible] | It looks like yo... | – | –
2d ago | conversat... | [illegible] | Your current b... | – | –

Trace | Timeline | Thread | Signals | Views

Interface text:
Braintrust Demos
l8r-customer-service
Overview | Logs | Topics | Monitor | Review | Playgrounds | Experiments | Datasets | Prompts | Scorers | Parameters | Tools | SQL sandbox | Settings
Recent projects: helpr-workshop_phil, bt-livekit-demo, LangGraphScratch
Review | Automations | Related | Tag | Find | Share
Braintrust
Home | Docs | Customers | Blog | Pricing | Contact sales | Discord | Newsletter | Changelog | Status | Trust center | Legal
braintrust.dev
Knowledge & Dev | Product and Support | Notion | Linear | Issues – Pylon | Policy Documents | Braintrust Docs | Repository Manage... | ChatGPT Model | Pull Requests | APM Home | Datadog | Competitor Diff | Excalidraw Whitebo... | Child’s Play, by Sam... | github.com | notion.so/braintrust...
New Arc Version Available
Engineering the future of AI
AI Engineer EUROPE

[16:27](https://www.youtube.com/watch?v=XBaznoTRDFI&t=987s) [A customer service conversation appears beside automated scores and rating controls.](https://assets.ai.engineer/sites/aiecode2025/sha256/10bf15a4485a9fba7073fee78e223406bf74b21f6fe2dc2c09f828058d569d6e.webp)

Logs
All logs view
Filter
Display
Search traces
Past 3 days
Timeline: Tue 07, 12 PM, Wed 08, 12 PM, Thu 09, 12 PM

5 traces
Created | Name | Input | Output | Expected | Tags
17h ago | conversat... | [illegible] | I understand t... | – | –
17h ago | conversat... | [illegible] | Since you hav... | – | –
18h ago | conversat... | [illegible] | Once your ref... | – | –
19h ago | conversat... | [illegible] | It looks like yo... | – | –
2d ago | conversat... | [illegible] | Your current b... | – | –

Trace | Timeline | Thread | Signals | Views
CONVERSATION
8 messages · 4 tool calls
Add Justification

AUTOMATED SCORES
L8r Topic Adherence: 100%
Politeness & Tool Accuracy: 100%
Response Conciseness: 100%

I want to request a refund

I can assist you with that. Could you please provide me with the order ID for which you'd like to request a refund, and let me know the reason for the refund request? If you don't know the order ID, I can help you find your recent orders.

What was the refund for?

get_orders | Calling

To assist you with finding information on any potential refund, I need to look at your recent orders. Let me do that for you.It appears that there are currently no refunded orders in your account. If you're looking to refund a particular order, please provide the order ID and the reason for the refund request, or let me know if you need help finding your recent orders.

get_orders | Done | Show result

I think I bought something from Nike?

get_orders | Calling

Interface text:
Braintrust Demos
l8r-customer-service
Overview | Logs | Topics | Monitor | Review | Playgrounds | Experiments | Datasets | Prompts | Scorers | Parameters | Tools | SQL sandbox | Settings
Recent projects: helpr-workshop_phil, bt-livekit-demo, LangGraphScratch
Review | Automations | Related | Tag | Find | Share
Braintrust
Home | Docs | Customers | Blog | Pricing | Contact sales | Discord | Newsletter | Changelog | Status | Trust center | Legal
braintrust.dev
Knowledge & Dev | Product and Support | Notion | Linear | Issues – Pylon | Policy Documents | Braintrust Docs | Repository Manage... | ChatGPT Model | Pull Requests | APM Home | Datadog | Competitor Diff | Excalidraw Whitebo... | Child’s Play, by Sam... | github.com | notion.so/braintrust...
New Arc Version Available
Engineering the future of AI
AI Engineer EUROPE

### Adding a review justification

A reviewer can rate a customer-service trace and add a justification. The speaker suggests using those explanations to identify failure modes and build more scalable automated scores.

[16:45](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1005s) [A conversation trace shows automated scores, rating controls, and an empty justification field.](https://assets.ai.engineer/sites/aiecode2025/sha256/9e25b4cc3460798c799fd5f581b90879fd5da627c0a09418000f06cb7682fcee.webp)

CONVERSATION
8 messages · 4 tool calls

Add a short justification for your rating...
Save
Cancel

AUTOMATED SCORES
L8r Topic Adherence: 100%
Politeness & Tool Accuracy: 100%
Response Conciseness: 100%

I want to request a refund

I can assist you with that. Could you please provide me with the order ID for which you'd like to request a refund, and let me know the reason for the refund request? If you don't know the order ID, I can help you find your recent orders.

What was the refund for?

get_orders: Calling

To assist you with finding information on any potential refund, I need to look at your recent orders. Let me do that for you. It appears that there are currently no refunded orders in your account. If you're looking to refund a particular order, please provide the order ID and the reason for the refund request, or let me know if you need help finding your recent orders.

get_orders: Done · Show result

I think I bought something from Nike?

AIE
AI Engineer EUROPE
Engineering the future of AI

Interface text:
Braintrust Demos
l8r-customer-service
Logs
All logs view
Past 3 days
Search traces
Trace · Timeline · Thread · Signals · Views
Created · Name · Input · Output
5 traces
17h ago · 17h ago · 18h ago · 19h ago · 2d ago
Overview · Logs · Topics · Monitor · Review · Playgrounds · Experiments · Datasets · Prompts · Scorers · Parameters · Tools · SQL sandbox · Settings
Recent projects
helpr-workshop_phil · bt-livekit-demo · LangGraphScratch
braintrust.dev
Knowledge & Dev · Product and Support · Notion · Linear · Issues – Pylon · Policy Documents · Braintrust Docs · Repository Manage... · ChatGPT Model · Pull Requests · APM Home | Dat... · Competitor Diff · Excalidraw Whitebo... · Child's Play, by Sam... · github.com · notion.so/braintrust...
New Arc Version Available
Braintrust
Home · Docs · Customers · Blog · Pricing · Contact sales · Discord · Newsletter · Changelog · Status · Trust center · Legal

### Conversation topics

Braintrust groups customer-service conversations by topic: credit balance and utilization accounts for 17.9% (35 items). The speaker describes this as an open-ended way to find unknown unknowns beyond scores.

[20:12](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1212s) [A topic list groups customer-service conversations by issue and displays proportions.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c774870b87b48296a7906d87bba3172e76eab39d4477a42968bde3d14580c77.webp)

Task v2
Topics  Edit
Generate  View 10 topics as list

17.9%  35 items
Credit balance and utilization

13.3%  26 items
Failed payment verification

13.3%  26 items
Refund requests

10.2%  20 items
Installment plan adjustments

10.2%  20 items
Order history and status

9.7%  19 items
Upcoming payment obligations

7.7%  15 items
Payment forgiveness requests

7.1%  14 items
General account balance checks

5.6%  11 items
Account balance inquiries

Interface text:
Braintrust Demos
Project
l8r-customer-service
Overview
Logs
Topics
Monitor
Review
Playgrounds
Experiments
Datasets
Prompts
Scorers
Parameters
Tools
SQL sandbox
Settings
Recent projects
helpr-workshop_phil
bt-livekit-demo
LangGraphScratch
Topic map
Sentiment
Issues
Task
Task v2
Issues v2
VERSION [illegible] (latest)
Go to logs topic view
Configuration
6 automated facets or topic maps
braintrust.dev
Knowledge & Dev
Pull Requests
APM Home | Datadog
Competitor Diff
Excalidraw Whitebo...
Child's Play, by Sam...
github.com
notion.so/braintrust...
AI Engineering Europe
Agents in Data Scie...
New Tab
Schedule | AI Engine...
AIEngEurope_WhyE...
AIEngEurope_EvalM...
New Arc Version Available
AI Engineer EUROPE

## Full transcript

[0:00](https://www.youtube.com/watch?v=XBaznoTRDFI&t=0s) [upbeat music] Uh, thanks for joining me today, uh, towards the end of the day here.

[0:17](https://www.youtube.com/watch?v=XBaznoTRDFI&t=17s) So I hope, um, everyone has enough energy left for, um, maybe what is your most exciting topic of, of the day. [laughs] Remains to be seen, uh, how traditional observability differs from agent observability.

[0:30](https://www.youtube.com/watch?v=XBaznoTRDFI&t=30s) Um, quick agenda. Uh, do a quick intro about myself and the company that I work for. Um, this is not gonna be a very product-forward talk. It's gonna be more theoretical.

[0:39](https://www.youtube.com/watch?v=XBaznoTRDFI&t=39s) So I won't drown, drown you in sales slides, I promise. Um, and then we'll get into, uh, how these two ideas differ, and also talk about, um, um, like, what's next in the, um, in the space.

[0:52](https://www.youtube.com/watch?v=XBaznoTRDFI&t=52s) Uh, my name is Phil Hetzel. I lead solutions engineering for Braintrust. What that means effectively is that me and my team, we're the folks that, uh, are charged with making sure that, um, our customers are getting the most value out of the platform as quickly as possible.

[1:08](https://www.youtube.com/watch?v=XBaznoTRDFI&t=68s) Prior to Braintrust, I spent 12 years in consulting and systems implementation. Uh, I led the Global Databricks practice for Slalom Consulting before I came here, and I noticed that a lot of my customers were prolific at creating generative AI proofs of concepts, but not nearly as good at bringing those proofs of concepts to production.

[1:26](https://www.youtube.com/watch?v=XBaznoTRDFI&t=86s) So I started using Braintrust as a user first, and I really liked it, and I applied for a job, and I've been here for about a year. Uh, I like to play chess out- outside of work.

[1:35](https://www.youtube.com/watch?v=XBaznoTRDFI&t=95s) I like to spend time with my wife and dachshund. Um, that's Pistol Pete right there. That's, that's, that's my dog. Um, he's the person in brown and not black.

[1:45](https://www.youtube.com/watch?v=XBaznoTRDFI&t=105s) Um, those of you who have been to my sessions before didn't laugh at that joke 'cause you've heard it, uh, at least, at least once already.

[1:52](https://www.youtube.com/watch?v=XBaznoTRDFI&t=112s) Uh, what is Braintrust? Braintrust is a, um, agent quality platform. We mainly look at agent quality in two different ways. Is your agent performing as well as you thought it would when it's in production?

[2:05](https://www.youtube.com/watch?v=XBaznoTRDFI&t=125s) Um, i.e., uh, can you remain confident in your agent? And then on the other side of that is, as you're experimenting with new versions of your agent, um, do you feel like you can become confident as you, as you tweak it and change it over time?

[2:21](https://www.youtube.com/watch?v=XBaznoTRDFI&t=141s) Those are a couple of things that, that Braintrust does, obviously relevant to today's discussion because, uh, uh, agent observability is a massive part about what we do. Uh, anyone has, has heard of Braintrust?

[2:33](https://www.youtube.com/watch?v=XBaznoTRDFI&t=153s) Show of hands. Before this week, did you hear of Braintrust? Yeah. Okay, a couple folks. Well, welcome back for the folks that have heard of us [laughs] before. Um, I'm gonna go, like, pretty quickly through the slides.

[2:43](https://www.youtube.com/watch?v=XBaznoTRDFI&t=163s) Hopefully, we have enough time for questions a- as well. I don't have, like, a ton of content.

[2:48](https://www.youtube.com/watch?v=XBaznoTRDFI&t=168s) Uh, traditional observability is established. So e- even when, uh, folks come to us, they'll say, "Well, we already have, uh, open source tools like Grafana," or, or, uh, a- as an example.

[2:58](https://www.youtube.com/watch?v=XBaznoTRDFI&t=178s) Why wouldn't this be the same problem that we're solving with, uh, perhaps either an implementation or a contract that we already have? Uh, it's very established and, um, we know that these applications can operate at scale.

[3:12](https://www.youtube.com/watch?v=XBaznoTRDFI&t=192s) So, um, the case that I'll be making is that, uh, the scope of traditional observability is actually quite different from the scope of agent observability, and, and I'll explain why.

[3:23](https://www.youtube.com/watch?v=XBaznoTRDFI&t=203s) Scope of traditional observability, uh, it's all about uptime and technical performance. Um, is the application up, and is the application giving a user experience from a technical lens that we would expect?

[3:36](https://www.youtube.com/watch?v=XBaznoTRDFI&t=216s) So, um, latency, uh, duration of, of interactions, four hundred and, and five hundred level errors, these are all things that we're measuring with, uh, with very established tools like a Grafana, like, uh, like a Datadog.

[3:50](https://www.youtube.com/watch?v=XBaznoTRDFI&t=230s) Um, I, I will even say that, uh, at Braintrust, though we are an agent observability platform, we're happy users of Datadog. Like, it's, it's great for this specific type of use case for us to understand if people are running into five hundred or four hundred level errors on, on our website a- as an example.

[4:07](https://www.youtube.com/watch?v=XBaznoTRDFI&t=247s) Is the system operational? Are we up or are we down? That's what traditional observability is. The building blocks of this are, uh, a couple different things. Um, metrics, these are the things that you're, you're measuring.

[4:22](https://www.youtube.com/watch?v=XBaznoTRDFI&t=262s) I gave a couple examples before, but latency is, uh, is, is the most obvious one. Error count is another. The things that you can aggregate and, and, and measure over time.

[4:32](https://www.youtube.com/watch?v=XBaznoTRDFI&t=272s) And then traces and spans, um... Uh, uh, is everyone here familiar with o- observability? Does everyone know what a trace is? Okay. I don't wanna take that for granted.

[4:40](https://www.youtube.com/watch?v=XBaznoTRDFI&t=280s) A trace is just, like, a full interaction of, of some workflow. Um, and a span is just one step within that interaction. Uh, all of these things would apply to agent observability as well.

[4:52](https://www.youtube.com/watch?v=XBaznoTRDFI&t=292s) So we have, we have the same, uh, building blocks.

[4:56](https://www.youtube.com/watch?v=XBaznoTRDFI&t=296s) Um, problem one for why agent observability is different, agents are non-deterministic, whereas applications are deterministic. The reason why we love LLM so much is because they have high variety.

[5:08](https://www.youtube.com/watch?v=XBaznoTRDFI&t=308s) They can do a lot of different things. They are abstracted. So because of that, while typical applications have very deterministic code paths by, you know, uh, um... And, and it, it's, it's on purpose that, that they do that, where they're performing some type of known control flow.

[5:27](https://www.youtube.com/watch?v=XBaznoTRDFI&t=327s) Agent applications are very much non-deterministic. We're curious about why an agent might take one path versus the other. Um, this also means that traditional observability, um, are-- is going to really have to focus on very constrained and known metrics, whereas agent observability, um, needs to be a little bit broader in terms of the things that, um, that

[5:52](https://www.youtube.com/watch?v=XBaznoTRDFI&t=352s) it needs to measure. This is, this is just an example of that. Um, so at the bottom, let's start there. Agent observability can- Uh, can measure some of these more traditional metrics.

[6:05](https://www.youtube.com/watch?v=XBaznoTRDFI&t=365s) I, uh, I'll, albeit with, with more of a, an, an AI flair, time to first token, total tokens, duration, uh, latency. These are all things that you would think would be very traditional observability level metrics.

[6:21](https://www.youtube.com/watch?v=XBaznoTRDFI&t=381s) But also you might want to understand more qualitative things about your application. So it's not just how, how long did I take to start responding to my user, which is more traditional observability.

[6:34](https://www.youtube.com/watch?v=XBaznoTRDFI&t=394s) I wanna know was the information that I gave grounded in the context that I gathered with my application? Did I use the tools that I would have expected, um, in this, um, uh, in, in...

[6:46](https://www.youtube.com/watch?v=XBaznoTRDFI&t=406s) as I was reasoning towards my response? Is, is a response aligned to the brand standard that I set for this agent in, in the system prompt? These are all things that are not really able to be tested by traditional observability tools because if you think about it, like, the, the trace necessary, the information and the trace that's

[7:08](https://www.youtube.com/watch?v=XBaznoTRDFI&t=428s) necessary for us to compute these things up at the top is far larger than the, than the volume that a traditional observability trace would handle. Um, that kind of goes to, to the next point here.

[7:22](https://www.youtube.com/watch?v=XBaznoTRDFI&t=442s) Agent traces are really nasty. They're, um, in, in a variety of different ways. Uh, they're nasty because they're highly semi-structured. Even within those semi-structured, there's a ton of unstructured text data that we need to chew through.

[7:35](https://www.youtube.com/watch?v=XBaznoTRDFI&t=455s) They're voluminous, so they can be... an agent trace could be over a gigabyte in size. We've seen that, uh, uh, even with our own customers. An individual span can be twenty megabytes in size.

[7:47](https://www.youtube.com/watch?v=XBaznoTRDFI&t=467s) So it, it's just a far different systems problem that you have to solve in order to ingest, process, and most importantly, use that, that type of data. And also it's, it's just as fast as a traditional observability data.

[8:00](https://www.youtube.com/watch?v=XBaznoTRDFI&t=480s) So hopefully your, your agent that you're putting in production gets product market fit, and you have a ton of users and, and, and usage associated with it. You as the AI engineer or as the product manager for that agent, you're going to want to see that observability in real time, in true real time.

[8:19](https://www.youtube.com/watch?v=XBaznoTRDFI&t=499s) Uh, trust me, we know that's the case because we always get the feedback. Yeah. Can you, can you just make it faster? We're always trying to make it faster.

[8:27](https://www.youtube.com/watch?v=XBaznoTRDFI&t=507s) People always want it, want it to be faster. Tough to do when the agent traces look like this. Basically, this is just like an ex- example of an agent trace in Braintrust, where not only does it have a bunch of spans here enco- encompassing the model calls and, and, and, and tool calls, but even within those spans,

[8:47](https://www.youtube.com/watch?v=XBaznoTRDFI&t=527s) you saw the amount of unstructured text that's in there as well. Very different problem to solve.

[8:54](https://www.youtube.com/watch?v=XBaznoTRDFI&t=534s) Um, a little bit, a little bit more here. Um, like very... like may- maybe I'll just dive in, into the read pattern, uh, piece specifically. We need to do, uh, two things simultaneously.

[9:08](https://www.youtube.com/watch?v=XBaznoTRDFI&t=548s) We need to be able to perform like the very fast read, uh, in- ingest and read style workflows that are common with observability, i.e., if someone does an action with my agent, um, uh, I need to be able to see that interaction basically inst- instantaneously.

[9:28](https://www.youtube.com/watch?v=XBaznoTRDFI&t=568s) We also have to commit to read patterns where someone wants to use our CLI and fire off SQL commands to us so that they can incorporate either observability or eval traces to impro- to improve their application automatically.

[9:45](https://www.youtube.com/watch?v=XBaznoTRDFI&t=585s) There are just a lot of different mediums that people use now in order to query these very large, uh, uh, trace shapes.

[9:53](https://www.youtube.com/watch?v=XBaznoTRDFI&t=593s) Uh, this is a new, it's a completely new systems problem. Um, at least at, at Braintrust, we designed a, a database from the ground up specifically for agent traces.

[10:03](https://www.youtube.com/watch?v=XBaznoTRDFI&t=603s) I'm not gonna really go into depth about this. We have a, we have a blog on our website. I think it was the last blog that we published, if, if you're really interested in diving deep.

[10:11](https://www.youtube.com/watch?v=XBaznoTRDFI&t=611s) But just very quickly, there are a lot of different components that we have to build into this database in order to make it work. For example, we need to, um, immediately get data into a write-ahead log so that people can instantly see these traces as soon as they expect.

[10:30](https://www.youtube.com/watch?v=XBaznoTRDFI&t=630s) We need to be able to perform indexing on these data so that whenever someone is performing a, a filtering or analytical query, that it's fast. And, um, we have this thing called a, a Tantivy index.

[10:42](https://www.youtube.com/watch?v=XBaznoTRDFI&t=642s) Tantivy is a, um, is, is an open source framework that we forked. Anyone know what Tantivy does? Any guesses? Tantivy is how we perform, uh, like text style in- indexing.

[10:56](https://www.youtube.com/watch?v=XBaznoTRDFI&t=656s) So if you remember when I was showing this trace, it makes so much sense for someone to want to perform the workflow of, okay, I would-- I just wanna know every trace that had the word Amazon into it.

[11:09](https://www.youtube.com/watch?v=XBaznoTRDFI&t=669s) Well, it turns out it's really hard to do that unless you perform a, a full text-based index across your traces. That's another reason why, um, a- agent observability is far different than traditional observability.

[11:22](https://www.youtube.com/watch?v=XBaznoTRDFI&t=682s) You really don't have to think about the text problems in traditional observability.

[11:27](https://www.youtube.com/watch?v=XBaznoTRDFI&t=687s) Is that the same as OpenSearch?

[11:29](https://www.youtube.com/watch?v=XBaznoTRDFI&t=689s) Sorry?

[11:29](https://www.youtube.com/watch?v=XBaznoTRDFI&t=689s) It's kind of like an OpenSearch

[11:31](https://www.youtube.com/watch?v=XBaznoTRDFI&t=691s) Tantivy is most similar to like an Apache Lucene ex- except it's, uh, written in Rust. Yeah. Um, and then all of these things, uh, come together and have to be unified through a, a SQL or SQL similar language.

[11:45](https://www.youtube.com/watch?v=XBaznoTRDFI&t=705s) That's what we've... that's the route that we've gone to at Braintrust. Um, problem three, uh, this is a... whereas there's a very specific type of persona for traditional observability.

[11:58](https://www.youtube.com/watch?v=XBaznoTRDFI&t=718s) It's a systems engineer. Maybe it's a product engineer. Um, it's probably not a subject matter expert or if it's a medical application, it's not a, not a clinician or, or, or a registered nurse.

[12:10](https://www.youtube.com/watch?v=XBaznoTRDFI&t=730s) It's very technical people that align with traditional observability. Uh, that could not be further from the truth for a- agent observability if you're doing it well. We noticed that the best teams that are building agent have both technical and non-technical people in the fold performing this work because it's the non-technical people that are either, A, closest to

[12:31](https://www.youtube.com/watch?v=XBaznoTRDFI&t=751s) the users, or B, have knowledge that is closest to the problem space. And what can they do now with prompts? They can write it in natural language. So they can add real value into being able to participate in, in agents.

[12:44](https://www.youtube.com/watch?v=XBaznoTRDFI&t=764s) Um, we have folks that, that are clinicians or registered nurses or wealth advisors or, or, or lawyers. We have seen them operate in our platform looking through traces and using that information to improve their agents.

[13:00](https://www.youtube.com/watch?v=XBaznoTRDFI&t=780s) That is a workflow that you j-- that you simply don't see in traditional observability where you're more worried about uptime.

[13:08](https://www.youtube.com/watch?v=XBaznoTRDFI&t=788s) Um, I think it... Like, in, in general, people don't realize that in order to perform observability and, and also evals well, we kind of think of observability and evals as the same problem.

[13:20](https://www.youtube.com/watch?v=XBaznoTRDFI&t=800s) The only difference between evals and observability is that you're running them in batch and you know the inputs ahead of time. Um, it's, it's incredible. It, it... The, the depth that you end up going into when you create a platform like this, it looks like, it looks like this, um, because of the, the reasons that I've described.

[13:38](https://www.youtube.com/watch?v=XBaznoTRDFI&t=818s) The nuances with the data, um, the amount of and, and types of people that you have to bring into the fold. Those are some of the reasons why it's so different to perform in this space.

[13:50](https://www.youtube.com/watch?v=XBaznoTRDFI&t=830s) Uh, where is this space going? Um, we- we've done a lot of work in this area. I think the, the natural question that we used to get asked was, "If you're c-- If, if you, Braintrust, are collecting all of our agent traces and all of our agent traces have all of these valuable data in them, can't you

[14:08](https://www.youtube.com/watch?v=XBaznoTRDFI&t=848s) just tell me how people are using my agent?" Um, and it's the si- it's the simple questions that usually need the, the most com- complex systems behind them. Um, this is something that we are starting to do.

[14:20](https://www.youtube.com/watch?v=XBaznoTRDFI&t=860s) Um, we, we just rolled it out, I think about a month ago in, in our software as a service offering, where we see agent observability traces come in and then we'll run like a very lightweight LLM on top of them to perform embedding and then clustering on those traces to see how we can perf- like elevate topic,

[14:41](https://www.youtube.com/watch?v=XBaznoTRDFI&t=881s) uh, e- elevate topic modeling to see, for example, how people are using, uh, your traces, their intent, how people are feeling about interacting with your agent, the sentiment, or if they're running into issues, what those in- issues potentially are.

[14:54](https://www.youtube.com/watch?v=XBaznoTRDFI&t=894s) The whole idea is there is that you can, um, make the iteration loop between a problem that you're seeing in production and the fix that you perform experimentation on.

[15:04](https://www.youtube.com/watch?v=XBaznoTRDFI&t=904s) Whole idea is to just make that faster and a little bit more direct. Um, I promised I would go through that really fast. I've got about three minutes for questions if there are any, uh, and I'd be really happy to answer them. [audience applauding]

[15:23](https://www.youtube.com/watch?v=XBaznoTRDFI&t=923s) Anyone curious about this? Yes.

[15:24](https://www.youtube.com/watch?v=XBaznoTRDFI&t=924s) So Braintrust is clearly about, uh, the, the functional observability of agents. Um, would you say it's-

[15:33](https://www.youtube.com/watch?v=XBaznoTRDFI&t=933s) I would say technical as well

[15:34](https://www.youtube.com/watch?v=XBaznoTRDFI&t=934s) ... would you say it's also good for non-functional agent performance, or is traditional observability good for that?

[15:40](https://www.youtube.com/watch?v=XBaznoTRDFI&t=940s) That's a good question, yeah. I would-- Well, I think traditional observability can do that. Um, Braintrust specifically does do-- I like the way that you put that, functional observability.

[15:50](https://www.youtube.com/watch?v=XBaznoTRDFI&t=950s) Um, what's the quality of my agent? How I've defined it? And then the technical observability just kind of comes on the house. Like, when, when, when you, when you trace the application, you automatically get prompts, duration, time to first token, et cetera, cache hits, et cetera.

[16:07](https://www.youtube.com/watch?v=XBaznoTRDFI&t=967s) Yeah. In your iceberg slide- Yeah ... the human annotation- Yeah ... at the waterline there. Yeah. Could you explain what the human annotation part is in, in this platform?

[16:18](https://www.youtube.com/watch?v=XBaznoTRDFI&t=978s) So let's think about it this way. Um, actually if, if I can, I'll go on a high wire act here and, um, and just show and, and, and not tell.

[16:29](https://www.youtube.com/watch?v=XBaznoTRDFI&t=989s) So let's say that you have a trace come in and you want your product manager to be able to opine on whether that agent did a good job or a bad job. [clears throat]

[16:39](https://www.youtube.com/watch?v=XBaznoTRDFI&t=999s) It's really valuable for you to have an expert come in, grade the agents, but then also like justify why they're grading the agents the way that they are. 'Cause eventually you're gonna take those justifications, you're gonna probably run an LLM over it, and you're going to make more, um, scalable scoring functions from those justifications.

[17:01](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1021s) You're finding the failure modes that you can then implement in automated scores through that. Yeah. Human annotation is a really key part of this process. Okay. Thanks. Uh, yes, in the second row.

[17:12](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1032s) Um, yeah, I just have a question, uh, because I've, I... You know, you just focused on observability piece today, but I'm interested actually how you also integrate with the other agentic framework to, uh, close the loop, like for the offline optimization.

[17:27](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1047s) Mm.

[17:28](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1048s) And then also, um, I guess the... Yeah, the, I think the main, uh, difference I feel that I see in this, your, uh, your database is that you're not using OLAP or ClickHouse like some of the users.

[17:40](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1060s) We used to use ClickHouse actually.

[17:42](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1062s) Oh, okay.

[17:42](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1062s) Yeah. We moved away from it.

[17:43](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1063s) Yeah. I just wanted to-- curious to, like, why you built your own? Like, what is the efficiency you found?

[17:49](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1069s) Well, the, the funny answer there is that our, um, our founder is kind of an insane person. Like, he-- like only an insane person would build their own database.

[17:59](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1079s) But he, he is, he is cut from that cloth. [chuckles] Um, he, he was one of the first employees at SingleStore, so he's kind of used to doing that. Um, but what he found was when, um, [lip smacks]

[18:11](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1091s) I think it was, let's see, this slide. He found that when, um, he was performing some of these workloads, he just needed the more of the text-based, uh, indus- indexes, which ClickHouse wasn't really able to do, at least at that time.

[18:24](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1104s) So we, we built our own. Um, and then the first part of your question, observability and evals to us, it's like we solve it with the same system. The only difference is that with evals, we know the inputs ahead of time, with obs- and we're doing it in batch.

[18:37](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1117s) With observability, we, we don't know what the inputs are ahead of time, and we're doing them in real time.

[18:41](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1121s) Oh, right. Uh, but I mean the experiment functionality. Like, um, how, how easy is it to integrate with like,

[18:50](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1130s) uh, um, Braintrust?

[18:51](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1131s) Oh, yeah. It should be pretty easy. Like, once, once you've traced ... And I, I apologize that I'm making this like about the product. Um, but when you, when you have a, a trace come in, you've traced it, and then you just like add it to an offline data set basically, so that you can experiment upon it.

[19:06](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1146s) Yeah. And do we have, do we have... I'm not sure if there's anyone, uh, after us in this room. Do we have to- Yeah. Not sure. Okay. But ca- carry on.

[19:16](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1156s) According to agenda, yes. Oh, is it? Okay. I'm, I'm happy to go on then. Yeah. I wanna- Yeah. Great.

[19:26](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1166s) Carry on until- Perfect. Yeah ... until 11:30.

[19:27](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1167s) Do you always, um, measure it quantitatively or do you also sometimes have some kind of qualitative piece of prose as the result? Like user satisfaction can be a number.

[19:38](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1178s) You can also imagine certain metrics to be just-

[19:41](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1181s) Yeah

[19:41](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1181s) ... yeah, a flag check or something.

[19:44](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1184s) Um, we can... Do, uh, do you wanna talk specifically about Braintrust, uh, like for that answer?

[19:50](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1190s) Yeah.

[19:51](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1191s) So like there is like the online scoring piece here where it's like a known unknown where you can like very much put a score behind that. Uh, but also there are, there are ways where more ...

[20:03](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1203s) Like, like this is, these are not scores. This is like the unknown unknowns piece. Got it. Yeah. Thanks.

[20:09](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1209s) Thank you.

[20:09](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1209s) Um, where we can in a, in a more like open-ended way derive insight from it. Yeah.

[20:18](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1218s) Yeah. Probably time for, for one question. If not...

[20:26](https://www.youtube.com/watch?v=XBaznoTRDFI&t=1226s) Great. I appreciate everyone's attention today. Thank you. [outro music]
