AI Engineer World's Fair 2025
Five Hard-Earned Lessons About Evals
Read the talk
Five Hard-Earned Lessons About Evals
Useful evals turn model releases and user feedback into product decisions. Building them requires engineering the data, scoring rules, tools, and optimization process together.
From a talk by Ankur Goyal
Before you start: Familiarity with LLM prompts, tool calls, and evaluation datasets will help you follow the examples.
Can your evals support the next model release?
A new model comes out. Can your team determine whether to use it and ship the change while the opportunity is still fresh? Ankur Goyal proposes a concrete test: your evals should prepare you to incorporate a new model into your product within 24 hours. He reports that Notion had incorporated new models within 24 hours across several releases. The point is operational: the evaluation process gives the team enough confidence to make a product decision.
The next test starts with a complaint. Can a user-reported failure become an eval through a clear, straightforward process? Without that path, valuable information disappears into support conversations and scattered feedback. With it, the team can preserve the failure and use it to guide subsequent improvements.
The third test looks ahead: can your evals tell you which use cases are feasible before you ship them? Regression tests help protect behavior that already works. Evals can also explore behavior that does not work yet, giving you evidence about how well a proposed product might perform before you commit to launching it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Datasets and scorers are engineering work
A synthetic dataset and an off-the-shelf LLM judge do not automatically produce a useful evaluation system. Most datasets assembled before deployment fail to represent what users will actually experience. Goyal offers competition math as a narrow counterexample: some tasks have a relatively clear relationship between benchmark problems and the intended use. Most products need an ongoing process for reconciling their dataset with reality. That process—not just the initial collection of examples—is the engineering problem.
Scoring requires the same investment. Teams often ask which built-in scorers will let them avoid thinking about scoring. Braintrust provides AutoEvals, but Goyal emphasizes that its openness and flexibility are deliberate: the advanced customers he works with write their own scoring functions and continually modify them.
Scorers are the specification for the application. They express what counts as acceptable behavior, much as a product requirements document does. This makes custom scoring a product-design responsibility, not incidental test infrastructure. A generic scorer may be a useful starting point, but its definition of success can amount to the specification for somebody else’s project.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tools construct the context the model sees
In an agent, the system prompt is only the beginning of the model’s context. The application repeatedly:
- Calls the model with the current context.
- Executes the tools the model requests.
- Incorporates the tool results into the next prompt.
- Repeats the process.
Each iteration adds material that the original system prompt did not contain. In the agent trajectories Goyal sampled, most prompt tokens came from outside the system prompt. Improving that prompt still matters, but it leaves much of the model’s input untouched.
Tool definitions and outputs therefore deserve the same care as instructions written directly to the model. An existing API reflects the way a product was built; it does not necessarily expose the operations or information an LLM needs. Good tool design starts with what the model should see and do. That can require substantial changes to the product, rather than simply placing an agent-facing wrapper over its existing APIs.
Output format is part of that design. In one internal analysis project, Goyal reports a significant improvement after changing a tool’s output from JSON to YAML. He attributes the difference to more efficient token use and easier reading by the model. This is a task-specific observation, not a general ranking of serialization formats.
The distinction is between text the model interprets and structured data that code consumes. Once equivalent JSON or YAML has been parsed into the data a JavaScript charting library expects, the original representation need not matter to the chart. Before parsing, the model encounters different textual structures. The tool’s interface and its returned text are both opportunities to make the next model call more effective.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep an eval for the feature that does not work yet
A new model can change which product architectures are worth building. Goyal credits people at Replit with helping pioneer the practice of preparing for that possibility: the product, team, and development habits should let a capability improvement become something users can actually use.
Braintrust had maintained an eval for a prospective feature and rerun it every few months. GPT-4o had once been the strongest available option; later runs improved with GPT-4.1, Claude 3.7 Sonnet, and then Claude Sonnet 4, called Claude 4 Sonnet in the talk. A result around 10% was not viable for this feature, Goyal says. The later improvement crossed the team’s threshold for shipping. These are results from Braintrust’s internal feature benchmark, not a general model ranking; the talk does not define its scoring rule, dataset size, or run settings.
Goyal reports shipping the feature’s first version two weeks after the Claude Sonnet 4 release. The standing eval made that response possible: the team already had a way to recognize that a previously impractical capability had become usable.
This is a reason to maintain ambitious evals even when current models perform poorly. Make the model replaceable in the evaluation setup, so trying a release is a small operation. Goyal describes Braintrust Proxy and similar tools as a way to switch providers without rewriting integration code.
A new release is an opportunity to measure, not a guarantee of progress. Goyal reports that Gemini 2.5 Pro 0520 scored 1% on this particular benchmark. He had not yet tested the newer Gemini version mentioned during the talk; the point was that the existing eval would make it easy to find out whether it helped.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the optimizer the data and scores
Optimizing an AI application means working on three connected components: the evaluation data, the task implementation—including prompts, the agent loop, and tools—and the scoring functions. A prompt can only be judged in relation to the examples it must handle and the criteria used to assess its output.
Braintrust compared two ways of asking an LLM to optimize prompts on the same benchmark. One supplied only the prompt. The other supplied the prompt, dataset, and scores, with a request to optimize the whole system. The slide reports:
| Optimizer context | Displayed benchmark result |
|---|---|
| Prompt Only | 8.9% |
| Dataset+Prompt+Scorers | 39.14% |
The difference is the evidence available to the optimizer: it can inspect both the task and how performance is being assessed, rather than editing instructions in isolation.
Goyal describes the improvement as enough to move the feature from unviable to viable. The practical lesson is to inspect the entire evaluation system when improving an application. Better wording alone cannot resolve every mismatch among the task, the examples, and the definition of success.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Loop turns evaluation artifacts into working context
The feature behind these experiments is Loop, which Goyal introduces as launching during the talk. The launch-era activation path was to open Braintrust’s feature flag section and enable Loop. In the playground, users could provide a prompt, dataset, and scores and work with Loop to improve them. Loop could also create prompts, datasets, and scoring functions.
The requests extend beyond rewriting a prompt:
- Prompt improvement: optimize the current instructions.
- Dataset coverage: identify missing cases worth testing for the use case.
- Score diagnosis: explain why results are unexpectedly low or high.
- Stricter evaluation: help write a harsher scorer when the current one is too forgiving.
These requests make the evaluation artifacts themselves part of the model’s working context.
Goyal reports strong results with Claude Sonnet 4 and says Claude Opus 4 performed a couple of percentage points better in this evaluation context. He also encourages experimentation with o3, o4-mini, Gemini, and custom or fine-tuned models.
The proposed workflow change is that a model can inspect prompts and data and make constructive improvements automatically. Goyal expects this to reduce the manual labor of iterating through evals; the launch is an opportunity to test that workflow with users and learn from their feedback.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the organization to act on the evidence
An organization may not yet have evals that support these decisions. Goyal’s closing emphasis is to assess that honestly and invest in the missing engineering: supplement synthetic or downloaded datasets with application evidence, write scorers that encode the intended behavior, and treat tool interfaces and outputs as deliberate communication with the model.
Readiness also has an architectural cost. Recognizing a meaningful improvement on release day is useful only if the team can act on it—even when that means replacing a substantial part of the system with an architecture that takes advantage of the new model. Whole-system optimization includes how the team obtains its data, not just the dataset currently on disk, alongside the task and scoring functions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Does adding user feedback cause overfitting?
The first audience question challenges the feedback loop: if complaints become evals, could the dataset overfit to those complaints? Goyal is more concerned about overfitting to a static dataset that excludes user feedback. What matters is not the dataset’s state at one moment, but the team’s ability to keep reconciling it with the reality the product should serve.
That does not mean automatically accepting every complaint as a test case. At the time of the talk, Braintrust deliberately did not add user feedback to datasets automatically. A person with domain intuition selected interesting examples and applied judgment: is this something the product should clearly support, and is its failure worth preserving in the evaluation set? Human curation connects feedback to intended behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the token percentages measure
Another question returns to the earlier context chart. Its percentages describe the relative number of tokens by message type in traces from a few agents:
| Category | What it contains |
|---|---|
| System prompt | System instructions |
| Tool definitions | Specifications of callable tools |
| User and assistant | Text exchanged in the conversation |
| Tool responses | Content generated by tools |
These are shares of token composition, not accuracy figures or the frequency of tool calls.
In Goyal’s account, tool-related content dominates the token budget of these agentic systems. The sample supports paying close attention to tool definitions and returned content; it is not a universal estimate for all agents. Simply turning an existing GraphQL API into a collection of tool calls misses the opportunity to design the information the model will actually consume.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An accurate answer can still receive a thumbs down
An audience member working with government services gives a concrete reason to curate feedback. Users sometimes give an accurate answer about taxes a thumbs down because they dislike what the answer means for them. The team added an explicit feedback option: “The answer is right, but I just don't like it.” Correctness and satisfaction are different signals; a negative reaction alone does not establish that the model made an error.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Model breakthroughs depend on the task
The same audience member questions how often a new model really changes everything. Their team had seen limited differences across Claude and OpenAI upgrades, apart from an anecdotal instruction-following failure with GPT-4.1 mini. What kinds of tasks actually experience a large change?
Goyal returns to Loop’s demanding job: inspect prompts, datasets, and scores, then automatically optimize prompts using that evidence. Braintrust had run its standing benchmark across successive model launches, with results staying near the earlier GPT-4o level for a long time.
The chart revisited during the answer shows the following results for the internal Loop benchmark:
| Model, as labeled on the slide | Displayed result |
|---|---|
| GPT 4o | 10% |
| GPT 4.1 | 46% |
| Claude 3.7 Sonnet | 51% |
| Claude 4 Sonnet | 58% |
But Loop’s evaluation set consists of evals it attempts to optimize, and not every constituent task needs a breakthrough. Identifying the movie that a quote comes from, for example, had already worked well since GPT-3.5, according to Goyal.
For mature tasks, a model upgrade may make little practical difference. For ambitious tasks, it can determine whether the product works at all. Keep those currently unsolved use cases expressed as runnable evals, so that when a new model arrives, discovering whether it changes the answer is a routine test rather than a new research project.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Python and TypeScript evaluation library with built-in metrics, customizable model judges, and examples of custom scoring functions.
Further reading
Goyal's written companion covering operational evals, custom scoring, tool context, and readiness for new models.
Sarah Sachs describes Notion's regression and frontier evaluations, rapid model deployment, and use of production feedback.
Updates since the talk
A later guide to using Loop to analyze traces, generate datasets and scorers, and improve prompts throughout Braintrust.
Read the complete timestamped transcript
- 0:00
[on-hold music] Uh, let's talk about some of the interesting things we've learned, uh, over time.
- 0:19
Um, so the first thing is, I think it's super important for you to, uh, understand and define, um, whether evals are actually providing value, uh, for your organization or not.
- 0:30
Um, and I tried to come up with three signs that you should look for, um, that, that are good. Uh, so the first is, um, if a new model comes out, uh, you should be prepared, um, uh, via your evals to be able to launch an update to your product within 24 hours that incorporates the new model.
- 0:49
Uh, Sarah from Notion, um, she talked yesterday, she talked about this, um, specifically, but, um, for the past several model releases, every time something comes out, Notion's able to incorporate, um, the new model within 24 hours, and I think that's a really good sign of success.
- 1:04
If you can't do that, um, then it means that, uh, you have some work to do on your evals.
- 1:10
Um, another sign of success is if a user complains about something, do you have a very clear and straightforward path to take their complaint and add it into your evals?
- 1:21
Um, if you do, then you have a shot at actually, um, incorporating user feedback, pulling it into your evals, and ultimately doing it better. If you don't, then you're gonna lose a lot of valuable information into the ether.
- 1:32
Uh, so again, I think this is a really important kind of threshold or milestone to hit.
- 1:38
Um, and the last one, which I'm actually gonna talk about a little bit more throughout the presentation, is, um, you should really start using evals to play offense and understand which use cases you can solve, um, and how well you can solve them before you actually ship things.
- 1:53
Not like unit tests, which allow you to just test for regressions. Um, and so if you, if you really adopt evals, then I think, uh, before you launch a new product, you have a really good idea of how well the product might work, uh, given what your evals say.
- 2:10
Um, the second lesson is that great evals, uh, they have to be engineered. You... They don't just come for free with, uh, synthetic datasets and random LLM-as-a-judge scorers that you read about online.
- 2:24
Um, and I think there's maybe two ways of thinking about this. Um, there's no dataset that is perfectly aligned with reality. Uh, I think in the cases that there are, there's like basically nothing to do, and the use cases already work, which there are a few that, that are kind of like that, like solving competition math problems,
- 2:41
for example. But for most real-world use cases, any dataset that you can come up with ahead of time is not going to represent what users are actually experiencing, and I think, um, the best datasets are those that you can continuously reconcile, um, as you actually experience what happens in reality, and doing that well requires quite a bit
- 3:00
of engineering. Um, of course, Braintrust can help you with that, but I think the, the point is, you have to think about, uh, a dataset as an engineering problem, not just something that's given to you.
- 3:11
And the same is true with scorers. I think, um, a lot of people we talk to ask, "Hey, what scorers does Braintrust come with, and, and how can we use those, uh, so that we don't need to think about scoring?"
- 3:22
And we actually have a really, uh, powerful, um, open source library called AutoEvals, but it's very open source and, uh, uh, flexible for a reason, which is that, um, every company that we work with that's sufficiently advanced is writing their own scoring functions, um, and modifying them, uh, constantly.
- 3:38
And I think, uh, one way to think about scorers is they're like a spec or like a PRD for your AI application. And if you think about them that way, um, one, it, it actually justifies making an investment in scoring beyond just using something off the shelf, and two, hopefully it's fairly obvious that if you just use,
- 3:56
you know, an open source or generic scorer, that's a spec for someone else's project, not yours.
- 4:05
Um, there's been a, a real shift towards context in prompts that's not just the system prompt that you write, and I actually think that, um, just traditional prompt engineering, pe-people say this in different ways, but I think traditional prompt engineering is evolving quite a bit, and it's very important to think about context, not just a prompt.
- 4:23
Um, so th-this, um, is an example of what kind of a modern prompt looks like for an agent. Usually, you have a system prompt and then a for loop which, you know, uh, runs LLM calls, uh, issues tool calls, incorporates the tool calls into the prompt, and then iterates and iterates.
- 4:41
Um, and I, I actually took a few, uh, um, uh, uh, trajectories from agents that, that we see in the wild and summarized these numbers. And as you can see, a vast majority of the tokens in the average prompt, um, are not from the system prompt.
- 4:56
And so yes, it's very important to write a good system prompt and continue to improve it, but if you're not very precise about, uh, how you define tools and how you define their outputs, uh, then you're leaving a lot on the table.
- 5:08
And I think one of the most important things we've learned, uh, together with some customers is that, um,
- 5:15
uh, you can't just take tools as a reflection of your APIs or your product as it exists today. You have to think about tools in terms of what the LLM wants to see, um, and how you can use, you know, exactly what you, uh, present to the LLM to make it work really well.
- 5:32
And I, I think that in most projects, um, it's actually very disruptive when you write good tools. Um, it's not something that's just like an API layer on top of the stuff that you already have.
- 5:43
And the same is true with their outputs. Um, there's one example that we, uh, worked on recently for an internal project where, um, shifting the output of a tool from JSON to YAML actually made a significant difference.
- 5:57
And I know that's a little bit of a, a meme in the AI universe, but it's just so much more token efficient and easy for an LLM to look at, um, a, uh, YAML-shaped data while doing analysis than extremely verbose JSON.
- 6:11
Um, now if you're writing code and you're plugging something into, uh, you know, a charting library, it makes no difference because to JavaScript, YAML and JSON are both structured data.
- 6:22
Um, but to an LLM, they're very different. And so I think you have to be very, very thoughtful about, um, you know, how you actually construct the definition of a tool and how you construct its output for the LLM to maximally benefit from it.
- 6:38
So I think one of the most important things we've learned, um, uh, and actually I th- I would credit some of the folks at Replit, uh, for really, uh, pioneering this pattern.
- 6:47
Um, but, you know, every time a new model comes out, uh, everything might change. Um, and I think you need to engineer your product, engineer your team, um, engineer your, you know, mindset so that when a new model comes out, if it changes everything for you, you can jump on that opportunity and, and ship something that maybe
- 7:06
wasn't possible before. Um, and I'm gonna show you some numbers, uh, for a product, uh, feature that we're actually launching, and I'm gonna show you a little bit of it today.
- 7:16
Um, but, uh, we, we've had an eval for a while that tells us how well this feature might work, and we run it every few months. And you can see, you know, it wasn't, uh, that long ago that GPT-4o was the best model out there.
- 7:30
Um, but, but things have changed. Uh, and, uh, you know, progressively, uh, GPT-4 1 did a little bit better, uh, 3.7 Sonnet is much better, and, and 4 Sonnet is actually even more remarkably better.
- 7:43
Um, and, uh, what, what that's meant for us is that this feature that, um, you know, at 10% would, would really not be viable for our users to use suddenly becomes viable.
- 7:54
Um, and so, you know, Claude 4 Sonnet actually came out two weeks ago, um, and we're shipping the first version of this feature today, which is just two weeks later.
- 8:03
But we were able to jump on that opportunity because we ran this eval, um, we were ready to do it, and we, we saw that, okay, great, we've actually finally crossed, uh, this threshold.
- 8:13
Um, so everyone that I personally work with or talk to I encourage to create evals that are very, very ambitious and, um, likely not, uh, uh, viable with today's models.
- 8:25
And construct them in a way that when a new model comes out, you can just plug the new model in and try it. Um, in Braintrust we have this tool called the Braintrust Proxy.
- 8:35
Um, there's a lot of, of similar tools. You could use ours or you could use something else. Uh, but really the point is that you don't need to change any code to work across model providers.
- 8:45
And so, um, you know, Google just launched the newest version of, of, uh, Gemini. Um, a- actually Gemini 2.5 Pro 0520 scores 1% on this benchmark, uh, so we didn't even put it on here.
- 8:58
Um, but maybe the thing they launched today actually s- uh, does a lot better. We can find out, you know, with, with just a few keystrokes maybe right after this talk.
- 9:09
Um, and the last thing is it's super important if you, uh, think about, um, optimizing your prompts to optimize the entire system. Um, so that means, uh, thinking holistically about your, um, uh, AI system as the data that you use for your evals, the task which is, you know, the prompt, the agentic system tools, et cetera, and
- 9:32
the scoring functions. And, and every time you think about making, um, you know, your, your app better, you need to think about improving this overall system. Um, we actually ran a benchmark, uh, which is, uh, the same benchmark that I w- I showed previously.
- 9:48
Um, it auto-optimizes prompts, uh, using, um, uh, an LLM. And, uh, we ran it, uh, once by just giving it the prompt and saying like, "Hey, please optimize the prompt."
- 10:00
And a second time giving it the prompt, the dataset, and the scores and said, "Please optimize this whole system." Um, and you can see there's a very dramatic difference.
- 10:07
So again, um, it-- something goes from unviable to viable, um, but it's just super important to optimize the entire system, not, not just the prompt.
- 10:21
And actually, uh, this is, uh, a new product feature that we are starting to launch today. Um, if you're a Braintrust user, uh, you can go to the feature flag section of Braintrust and turn on a new feature flag called Loop.
- 10:34
Um, and, uh, th- Loop is this amazing, cool new feature that actually auto-optimizes, uh, your evals, um, directly within Braintrust. Uh, so, uh, you can work in our playground and, um, give it a, a, you know, a prompt, uh, a dataset, um, and some scores, and it can actually create prompts, datasets, and scores too.
- 10:56
Um, and just, you know, work with it. Uh, the kinds of things that we've seen work really well are, "Optimize this prompt," or, uh, "What am I missing from this dataset that would be really good to test for this use case?"
- 11:09
Um, "Why is my score so low?" Um, or, "Why is my score so high? Can you please help me write a score that is, uh, you know, harsher than the one that I have right now?"
- 11:19
Um, you can also try it out with different models. So, uh, as you can see from this, uh, we've definitely seen the best performance with Claude 4 Sonnet, and Claude 4 Opus performs a couple of percentage points better.
- 11:32
Um, but we encourage you to try it out with different models. You can use o3, you can use o4-mini, you can use Gemini. Maybe you're building your own, uh, LLM or fine-tuned model, you can try that as well.
- 11:44
Um, and yeah, we're very excited, uh, for this. I think, uh, I'm gonna talk about this a little bit later, um, and I'm happy to do it with some Q&A as well.
- 11:52
But, um, I actually, I really think that the workflow around evals is going to dramatically change now that LLMs are capable of looking at prompts and looking at data and actually making, um, you know, constructive improvements automatically.
- 12:08
A lot of the manual labor that went into iterating with evals, um, doesn't need to be there anymore, so it's, it's really exciting. Uh, we're excited, uh, to ship this and, and to start to get some feedback.
- 12:21
Uh, so just to recap, um, five lessons that I think are really important. Um, effective evals speak for themselves. It's, it's important to understand whether you've kind of reached a point of eval competence in your organization or not.
- 12:34
It's okay if you haven't. Um, it's not easy, but it's important to be honest about that and work towards it. Um, when you're working on evals, it's very important to engineer the entire system.
- 12:45
So don't just think about the prompt. Don't just think about improving the prompt. Please don't just use synthetic data or Hugging Face datasets. I know they're awesome, but please use more than just that.
- 12:56
Please don't use off-the-shelf scores only. Write your own. Think very deliberately about, um, how you can craft the spec of what you're working on into your scoring functions. Um, think very carefully about context, and I think in particular, um, what helps me personally is to think about writing tools, uh, like I would think about writing a prompt.
- 13:18
It's my opportunity to communicate with an LLM and set it up for success, and how I define the API interface of the tool and I define its output has a very dramatic impact on that.
- 13:31
Make sure that you're ready for new models to come out and to just change everything. Um, so if an, if a new model comes out, you wanna be prepared to know that immediately, ideally the day that it comes out, um, and also be prepared to, like, rip out everything and replace it with a fundamentally new architecture that
- 13:49
takes advantage of that new model. And I think part of that is obviously having the right evals. Part of it is engineering your product in a way that actually allows you to do that.
- 14:00
And then finally, when you think about optimizing or improving, uh, your eval performance, um, you have to think about optimizing the whole system, the data and how you get that data, the task itself, um, which, you know, the prompt tools, et cetera, and the scoring functions.
- 14:19
Um, and with that, uh, we have some time for Q&A.
- 14:23
Yeah, there's, uh, two microphones up here, one on the left side, one on the right side. Uh, feel free to stand up and ask your questions.
- 14:36
Hi, this is Jyoti. Um, one of your slides said take feedback and turn it into an eval.
- 14:44
Are you concerned about overfitting evals at that point where every feedback then turns into an eval?
- 14:50
Oh, that's a great question. Um, also nice to see you. Um, so, uh, the question was, um, one of the slides was about taking feedback, uh, from, you know, real data and adding it to a dataset and incorporating it in an eval.
- 15:04
Are you worried about overfitting? Um, and I think the answer is, I'm actually way more worried about overfitting to the dataset without the user's feedback than I am to, um, adjusting the fit to incorporate the user's feedback.
- 15:17
Like, the most important thing about a dataset is not the state of the dataset at any point in time. It is how well you are equipped to reconcile the dataset with the reality that you want.
- 15:30
Um, and I actually think one of the things that we discourage, uh, in the product, and some people complain to us about this, I get it, uh, if you're one of those people, um, but we don't automatically take user feedback and add it to datasets right now.
- 15:42
We actually want a human who has some taste and maybe, uh, can build some intuition about the problem to find the, uh, data points from users that are interesting and add them to the dataset.
- 15:54
And I think that is your opportunity as a user to apply some judgment about like, oh, okay, this user is trying to do something that should obviously work. It's really sad that it doesn't work in my product.
- 16:05
Let me add it to the dataset so I can make sure it does. Excuse me.
- 16:10
Yes. You had a slide, I think, in the tool descriptions about like with, with some percentages on it. Yeah. Yeah, this one. What, what is that? Yeah, so, um, we took a few agents, um, like we, you know, have a lot of traces, uh, and we analyzed the relative, um, number of tokens for different message types.
- 16:30
So the system prompt is one message type. Tool definitions, um, are, you know, the spec of what, uh, tools the model can call. User and assistant, um, uh, are, um, tokens from user and assistant, just text interactions.
- 16:46
And then tool responses are, um, tokens from the, the, you know, the, the, the tool generates itself. Oh, and this is the percentage of tokens? Correct, and this is the relative percentage of those tokens, yeah.
- 16:57
Yeah. Yeah, so the, the, the, the point that we're trying to make here is that, um, I, I think in modern agentic systems, uh, tools actually like very, very significantly dominate the, uh, token budget of the LLM, and I think that it's very important to, um, think about how you define the definition of tools and how you
- 17:19
define their outputs so that you, um, you know, engineer the LLM for success, uh, not just sort of take, you know, your GraphQL API and give it as a bunch of, uh, you know, uh, tool calls to, to the LLM.
- 17:33
Um, first off, that point about the thumbs down is such a good point. I'm working with the government, and people don't like the answer they got, for example, about taxes, and they give it a thumbs down.
- 17:46
Yeah.
- 17:46
Right? So like adding that human aspect is a really good idea. We actually even added a little thing that said, "The answer is right, but I just don't like it." [laughs]
- 17:56
That's awesome.
- 17:57
Um, but my question is about your point that the new model changes everything. We've updated our models several times and, and used Claude and OpenAI, and we haven't found huge differences other than recently someone really cheap wanted to use 4.1 Mini and like it seemed to ignore every in...
- 18:20
It-- I swear it ignored the system prompt completely.
- 18:23
Yeah.
- 18:23
But what kind of things, when you say it changes everything, can you tell me a little more about what kind of changes you're seeing?
- 18:29
For sure. I think, um, the use case that we just shipped with Loop is a really good example of that. So this is a very ambitious, uh, agent. It's looking at prompts and, uh, datasets and scores and automatically optimizing the prompts based on the datasets and scores.
- 18:46
And this is something that, um, you know, we wrote a benchmark for a while ago and we ran with every consecutive model launch and the numbers looked more like what you see for GPT-4.0 for a very long time.
- 18:59
This isn't true for every benchmark. So, um, as part of this exercise, we actually have a bunch of, uh, evals that Loop optimizes. That's our eval set and there are some evals like, uh, classifying g- taking movie quotes and figuring out what movie they're coming from that have worked really well since GPT-3.5.
- 19:19
Um, and so there are certain use cases where it just doesn't matter. There are other use cases where, um, they're so ambitious that they just don't work today, and I think you wanna create evals, uh, so that if there's something ambitious that you wanna do in the future, you are very well prepared when a new model comes
- 19:35
out to just push a button and find that out.
- 19:38
Okay. Thank you. [upbeat music]