AI Engineer Summit 2025
Anthropic for VPs of AI
Read the talk
Building AI Products Around Outcomes and Evaluations
Alexander Bricken and Joe Bayley connect model capabilities to product decisions, using adaptive learning and Intercom’s Fin to explain evaluation, latency and optimization.
From a talk by Alexander Bricken and Joe Bayley
Before you start: Familiarity with LLM prompts, APIs and basic evaluation concepts will help; no model-training background is required.
What can your product solve now?
Which customer problems can AI solve today that seemed out of reach a year ago? That is the opening business question behind Alexander Bricken and Joe Bayley’s implementation advice. Bricken works on Anthropic’s Applied AI team, helping customers implement models and carrying findings back into product and model research. Bayley works in go-to-market and describes watching capabilities evolve from Claude 2.1 to systems supporting previously impractical applications.
Their guidance draws on colleagues’ experience and hundreds of customer interactions. It sits within Anthropic’s stated mission as an AI safety and research company: developing frontier models alongside safety techniques, research and policy. The practical challenge is to turn those capabilities into a product that reliably solves its customers’ problems.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From model capability to interpretable features
The model introduction centers on the October 2024 upgrade of Claude 3.5 Sonnet, called Sonnet 3.5 New in the presentation. Bricken highlights its coding capabilities and describes it as leading SWE-bench at the time. Treat that as a historical claim: the release announcement documents strong SWE-bench Verified results, but does not establish the leaderboard position on the day of this talk.
Anthropic’s capability, product and safety research overlap. Interpretability adds a different kind of understanding: reverse engineering what happens inside a model, identifying why particular behavior occurs, and potentially steering it toward a use case. Bricken describes the work as early, placing it informally somewhere in the first quarter to first half of the roadmap.
The proposed progression builds one capability on another:
- Understanding: investigate how the model makes decisions.
- Detection: recognize particular behaviors and assign meaningful labels.
- Steering: influence the model’s output by intervening in its internal activity.
- Explainability: turn that understanding into useful explanations and business value.
Bricken points to Towards Monosemanticity and Scaling Monosemanticity as research into model-level feature activations. Better detection could eventually help uncover concealed behaviors, including sleeper-agent behavior; this is a research direction, not a deployed detection guarantee.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recognizing basketball, steering toward a bridge
Suppose a model answers a question about today’s NBA games by saying Steph Curry scored 30 points. In Bricken’s hypothetical example, that answer activates feature 304, associated with famous NBA players. The feature represents a recognizable pattern across a group of neurons, rather than a single neuron reserved for Curry. Its meaning comes from the pattern recurring across mentions of different famous basketball players. The accompanying diagram contrasts such patterns for offers of assistance, famous NBA players, famous NFL players and sarcastic praise.
Golden Gate Claude illustrates the next step: intervention rather than observation. Amplifying a Golden Gate Bridge-related activation direction made the model bring the bridge into unrelated answers. Asked what color to paint a bedroom, it would suggest bridge-like red paint and even pillars. The example makes steering tangible: changing an internal activation can change the themes that dominate an answer, even when the user did not ask for them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the core product better
Choosing an AI feature starts with the problem the product already exists to solve. Bayley encourages teams to look beyond chatbots and summarization when deciding where to place larger bets. Consider an onboarding and upskilling platform: its purpose is to help employees ramp up quickly and acquire skills for their next career stage, such as public speaking or management. Summarizing a course is useful, but it only addresses part of that purpose.
A course Q&A assistant is another reasonable addition. The larger opportunity is to change the learning experience itself:
- Personalize the content: use each employee’s context to make lessons more relevant.
- Adapt the difficulty: increase the challenge when someone moves through the material too easily.
- Update the format: as the platform learns about the learner, generate different materials, including visual content where appropriate.
These are proposed product capabilities, not demonstrated learning outcomes. The decision is whether they advance employee development more directly than summaries or question answering. Customer domain expertise supplies the context that makes the model useful.
The same reasoning applies to tax, legal and project-management products. The goal is an experience that is easier to use and more trustworthy, with output quality appropriate to the consequences. A hallucinated tax answer is not merely awkward wording; it can cause a consequential error. Bayley describes customers using AI in these business-critical workflows, although this part of the talk supplies no customer-specific measurements.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From model access to an evaluated deployment
The presentation distinguishes two product routes and two cloud access options:
| Route | Purpose |
|---|---|
| Anthropic API | Embed AI in products and services |
| Claude for Work | Support employees’ everyday work |
| Amazon Bedrock | Access models within an AWS environment |
| Google Cloud Vertex AI | Access models within a Google Cloud environment |
Bayley presents the cloud routes as a way to work within existing infrastructure rather than manage an additional environment. He also says Anthropic’s support approach includes customers accessing models directly or through a third party.
Applied AI support covers architecture design, evaluations and prompt optimization, with customer findings feeding back into Anthropic’s products and research. Bricken names Building effective agents, associated with his colleague Barry, and the Model Context Protocol, an open protocol connecting language models with data sources. At the event, Barry’s talk was scheduled for the following day; Bricken also pointed to a forthcoming MCP workshop led by Mahesh.
Deeper collaboration begins when customers using Claude heavily encounter difficult domain-specific problems. A sprint can address LLMOps, architecture or evaluation design, but it needs metrics that the customer considers meaningful. The intended progression is to define those metrics, iterate on the system, put the resulting candidate into an A/B test, and then decide whether to move it into production.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fin: optimize for resolution, not deflection
Intercom’s customer-service agent, Fin, had been in the market for roughly a year when the collaboration began. Intercom shared where it wanted to take the product; Anthropic saw an opportunity to improve its target metrics. An Applied AI lead and Intercom’s data scientists started with a two-week sprint, comparing Fin’s hardest prompt against a version optimized for Claude. Encouraging initial results led to approximately two months of broader prompt optimization. The work described here concerns prompts, not established model-weight training.
Bayley reports that Claude then outperformed the incumbent model across Intercom’s benchmarks. The commercial model matters to interpreting the goal: Intercom charges based on resolution, creating an incentive to solve the customer’s problem rather than merely deflect the conversation. After the optimization period, Intercom chose Anthropic and launched Fin 2.
The Intercom case study reports resolution of up to 86% of customer-support volume, with a 51% average resolution rate out of the box. The maximum and the average describe different conditions; the latter is before customization or tuning. These vendor-reported figures do not specify the measurement window, cohort or operational definition of resolution.
Anthropic’s own support team also adopted Fin and reportedly saw similar resolution rates. Beyond resolution, the presentation highlights control over tone and answer length, plus policy awareness—for example, following refund policies. Bricken closes the customer discussion with an anecdote about people treating Claude as a therapist on Twitter, illustrating its perceived conversational character rather than a clinical capability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the evaluation before the architecture hardens
A recurring mistake is to build a substantial workflow and only then ask how to evaluate it. Evaluations should direct development, beginning at the outset or shortly afterward. Otherwise, the architecture accumulates assumptions before the team has a way to tell whether those assumptions improve the outcome.
Poor data quality can block that work; Bricken suggests using Claude for cleanup and reconciliation. Another failure is accepting a few convincing answers as sufficient evidence. Tests need representative inputs and enough samples to support the conclusion being drawn. His example of unexpected outliers appearing in the first hundred production queries is a warning about narrow testing, not a prescription for a hundred-case evaluation set.
Bricken describes optimization as moving through a latent space between attractor states. Prompt engineering and prompt caching are among the interventions he names. The useful point of the metaphor is uncertainty: editing an instruction does not tell you, from inspection of transformer attention, how the resulting application will behave. You have to measure the outcome empirically.
Evaluations are intellectual property because they help a team find useful configurations faster than competitors. Telemetry supports that advantage by making real interactions available for backtesting. A customer-support evaluation should include plausible off-topic inputs, such as a child asking how to kill a zombie in Minecraft. The expected behavior is an appropriate response or rerouting, rather than assuming every user will ask a relevant product question.
A small evaluation record can make that expectation explicit:
json
{
"id": "off-topic-minecraft",
"input": "How do I kill a zombie in Minecraft?",
"expected_behavior": {
"recognize_out_of_scope": true,
"redirect_to_product_support": true
},
"grading_questions": [
"Does the response recognize that the question is unrelated to the product?",
"Does it offer an appropriate route back to product support?"
]
}
This turns the example into a reviewable test case. A grader still needs to inspect the actual response against the criteria; simply storing the expected behavior does not establish that the agent follows it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let the decision set the latency budget
Application design involves a triangle of intelligence, cost and latency. Bricken says teams can often optimize one or two dimensions, while improving all three is difficult. The balance should be chosen in advance for the use case, rather than discovered accidentally after deployment.
| Use case | Illustrative time budget | Reason |
|---|---|---|
| Customer support | A response within 10 seconds | The customer is waiting in an active interaction |
| Financial research | Potentially 10 minutes | The answer may inform capital allocation |
Bricken uses these as contrasting design examples. His suggestion that support users abandon a page after ten seconds invokes unspecified research, so it should not be treated as a measured universal threshold. The underlying principle is that the stakes and time sensitivity of the decision determine the acceptable wait. More instructions may be worthwhile if they improve performance, even when they increase latency.
The interface can also affect how waiting feels. For a support response taking ten to fifteen seconds, Bricken suggests an animated thinking indicator or another page the customer can read while waiting. Those choices may make the delay more tolerable, but they do not remove the need to define the important latency metric and optimize the system against it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make fine-tuning earn its cost
Fine-tuning is not a default starting point. Bricken compares it to brain surgery: changing the model for one domain can impair reasoning elsewhere. That possibility adds to the direct costs of data preparation, specialist effort and experimentation. Before tuning, a team needs an evaluation set and explicit success criteria.
The decision sequence is straightforward:
- Define the performance required in the target domain.
- Try other approaches and measure the remaining gap.
- Pursue fine-tuning only when the potential improvement justifies its cost and variable outcomes.
- Keep developing the application; substitute a fine-tuned model later if the evidence supports it.
Making fine-tuning a prerequisite can delay a useful product before the team has established whether tuning is necessary.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the capabilities already available
Prompt engineering remains important, but it is only one intervention. Features and architecture choices can change the economics and quality of an application without changing model weights.
Prompt caching is an alternative to deleting useful instructions merely to make requests cheaper or faster. Bricken cites a 90% cost reduction and a 50% increase in speed for prompt caching, without specifying workload or cache-hit conditions. The historical caching explanation gives the narrower interpretation: repeated context can be reused, cached input reads cost one tenth of the base input price, and cache writes cost extra. That input discount is not a 90% reduction in the entire request bill. Its latency examples measure time to first token, so they do not establish a universal improvement in completion speed.
Contextual Retrieval addresses the information supplied to the model. Improving retrieval can make relevant evidence easier to find and deliver. Bricken also suggests a reduced processing burden, but the supporting research establishes retrieval-quality improvements rather than a general guarantee of faster generation.
Citations and agentic architectures round out the alternatives: built-in capabilities can support source attribution, while architecture determines how the application organizes model work. The closing feature table places these alongside tool use, batch processing, PDF handling, token counting, prompt tools, computer use and MCP. The practical next experiment should follow the measured bottleneck—repeated context, missing evidence or workflow design—before the team sacrifices useful instructions or commits to fine-tuning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Historical release announcement with coding benchmark results and deployment availability.
Anthropic's research introduction to decomposing model activations into interpretable features.
Anthropic's introduction to an open protocol connecting AI applications with data sources and tools.
The customer-support case study reporting up to 86% resolution and a 51% average out-of-the-box rate.
Explains repeated-context caching, historical pricing and workload-specific latency examples.
Explains contextual embeddings, lexical retrieval and reranking, with implementation guidance.
Further reading
An accessible introduction to interpretable features and steering model behavior.
- Building effective agentsArticle
Practical guidance on choosing simple workflows and agent architectures.
Read the complete timestamped transcript
- 0:00
[on-hold music] I'm Alexander Bricken.
- 0:18
I'm on the Applied AI team at Anthropic, so I work very closely with customers to do technical implementation work, and I also bring that advice back to product research and model research.
- 0:29
Um, I'm gonna pass it over to Joe.
- 0:31
Hey, everyone. It's great to be here. My name is Joe Bayley. I work on the go-to-market team at Anthropic. I joined Anthropic, uh, over a year ago now, so I've seen our models evolve from, uh, two point one to today's capabilities.
- 0:42
And I think day to day what's really exciting is we're working with AI leaders who are solving real business problems, um, that just seemed impossible a year ago. So really excited about how quickly everything is moving.
- 0:55
Okay. For today, we will do, um, a quick overview, uh, you know, who we are, o-our mission, and then we'll focus a lot on implementing AI and best practices and common mistakes.
- 1:08
Uh, Alex and I actually didn't just take this from our own experience, but we, um, talked to a number of our colleagues, so this is all based on hundreds and hundreds of customer interactions.
- 1:18
Um, so we hope there's some actionable insights to take out of this.
- 1:23
Awesome. So what is Anthropic? So we are an AI safety and research company building the world's best and safest, uh, large language models. We were founded a few years ago by some of the leading experts in AI.
- 1:37
And since our inception, we've not only released, uh, multiple iterations of our frontier models, we've done so while being at the bleeding edge of safety techniques, of research and policy.
- 1:48
I'm gonna pass it over to Alex to talk a little about our marquee model.
- 1:52
Awesome. And so some of you are probably familiar, but the most recent model we launched was Sonnet 3.5 New, uh, in late October of last year. Um, you might be familiar with it because if you're a developer, uh, Sonnet is actually one of the leading models in the code space.
- 2:08
So if you're familiar with evaluations like SWE-bench, which is an, an agentic coding eval, uh, Sonnet is still at the top of the leaderboard for that. Um, I won't go too much into the details on the eval side, so let's keep moving.
- 2:21
Um, so yeah, in addition to what Joe mentioned, we have a lot of different research directions that we're focused on. Um, and these are really distributed but have overlap between, you know, model capabilities, product research, and AI safety.
- 2:35
The one that differentiates us, I would say, is the interpretability, and this realistically is reverse engineering the models and trying to figure out actually how they're thinking and maybe why they're thinking, and then a-an additional capability in terms of steering them in the right direction, depending on a use case.
- 2:50
So let's dive into that a little bit more. We're still very early in interpretability research, it's worth mentioning. As you can see, there's kind of like a longer timeline, and we're really only at the, the first half of that, maybe even the first twenty-five percent.
- 3:02
Um, but we're, we're really approaching it in these stages that build upon each other. So these in-include things like understanding, so grasping AI decision-making. Detection, so actually being able to understand specific behaviors and put labels on those.
- 3:16
Steering, so influencing the AI input in some, some way, shape, or form, and I'll get to an example of that in a second. And then finally, explainability, and that's really where you unlock business value associated with interpretability methods.
- 3:28
And so while we see interpretability in the long term providing, you know, a lot of significant improvements in AI safety, reliability, and usability, specifically, our interpretability team uses methods to understand feature activations at the model level, and then has published research on these, uh, in-- towards model semanticity and scaling model semanticity, which are two papers I highly
- 3:50
recommend. Um, and then as the technology improves into kinda detection landscapes, for example, you can imagine having a much better grasp at, uh, at the actual thinking and behavior of the model or even discovering sleeper agents for safety reasons that might be buried deep within, uh, model capabilities.
- 4:08
So a good example of that is imagining you ask the model, "What were the scores of the NBA matches today?" Right? And let's say it knows the answer, and it says how Steph Curry, you know, scored thirty points.
- 4:20
This would lead to a feature activation of, for example, feature number [REDACTED:generic_id], famous NBA players. Realistically, that's a group of neurons activating in a recognizable pattern that we've identified across all mentions of famous basketball players when a model's answering a question, not just Steph Curry.
- 4:40
Um, and you also might have heard of Golden Gate Claude. That was an example of us steering the model, uh, basically amping up the activation, uh, in the Golden Gate direction.
- 4:49
And thus, whenever you'd ask a question like, "What should I paint my bedroom?" Claude would respond, "Oh, you should paint it red like the Golden Gate Bridge, and maybe it should have some like, you know, pillars in it or something."
- 5:00
I'm gonna pass it over to Joe to talk a little bit about some of the customers we work with.
- 5:03
Yeah. So I'm gonna frame this in two ways. One is, uh, sort of early on discussions, and the other would be just examples of customers that are doing really cool things.
- 5:12
So in conversations, there's obviously a lot of noise and buzz and everything, and that's fantastic, but we often encourage our customers, uh, to sort of get back to the basics and how can they use AI to solve, uh, the core problem that your product is trying to solve.
- 5:28
We also get to work with a ton of, uh, sort of AI native, like, AI startups, and this is how they're thinking about their product. And I think you wanna move beyond, uh, chatbots and summarization.
- 5:39
These can be great options, but I'd be thinking more like, where do you wanna place bigger bets? And to give an example, um, if you just click one more time.
- 5:47
Fancy slide. Uh, imagine you're an onboarding and upskilling platform. The problem that you solve for customers is you help them get-- you ramp really quickly, and then you help them get to the next phase o-of their career by equipping them with skills.
- 6:01
So for instance, it might be public speaking you wanna get good at, or you might wanna become a manager. And so it would be easy to say, "Okay, let's summarize course content."
- 6:09
Or let's, um, [lip smack] uh, let's, uh, have a Q&A chatbot that answers questions along the way, and they could be helpful. But I'd actually think about it differently. So what about if you could hyper-personalize, uh, course content based on each indiv- individual employee's context?
- 6:24
Or if someone is, like, breezing through all the course content, could you adapt it dynamically to make it more challenging, uh, so they're actually getting more value out of it?
- 6:32
And then the last one that I particularly like would be, what if you could, uh, uh, dynamically update, uh, course material based on people learning about the customer? So if someone was a visual learner, great, let's make visual content for them and do-- having the AI, having the ML, uh, sorry, the, the large language model just do
- 6:50
that automatically. And you have to think, does that solve the problem more than summarization or a, or a Q&A chatbot? Um, so really good food for thought. And to sort of talk about some of the customers, uh, that we see achieving really industry-leading results, uh, by combining, uh, their own domain expertise and our, um, our model.
- 7:11
So I won't read off each, but just a couple of call-outs. One is, uh, AI impacting different industries. We have, uh, taxes, we have legal, we have, uh, project management.
- 7:21
They're using AI to, uh, drastically enhance their customer experience. They make it more, uh, like, easier to use. They make it more trustworthy. Um, and so it's really improving the experience versus just being like a nice to have.
- 7:34
And then they're achieving a real h-- a real high quality, um, of output, right? You can't be giving-- you can't be hallucinating when you're doing your taxes. Uh, it could just, you know, that could lead to all sorts of things.
- 7:44
So we're thrilled that they're seeing these, these sort of like business-critical workflows powered by AI, driving really positive outcomes for them and also their customers.
- 7:55
Awesome.
- 7:56
I can do this one. Yeah. So, [chuckles] um, getting started. I just quickly, uh, there's two, two key points here. So if you go on the next slide, um, what are our products?
- 8:06
We have our API, we have Claude for Work. Our API, uh, is for businesses that wanna embed AI in their product and services, and then Claude for Work empowers your entire organization to take advantage of AI in their day-to-day work.
- 8:19
We also have, uh, next one. We also have a partnership with AWS and GCP, and you can kinda get the best of both worlds here. You can access our frontier models on Bedrock or on Vertex.
- 8:32
You can deploy these applications in your existing environment, um, and you, but you-- and so you don't have to manage any new infrastructure. So it really sort of breaks down like any barriers to entry.
- 8:43
So you're getting the best of both worlds here. We talk a little bit about support throughout this talk. It-- to us, it doesn't matter if you're accessing us through a third party or a first party, so I just wanted to call that out.
- 8:56
Awesome. So now that we've talked a little bit about some of the customers, how do we actually set customers up for success when working with them at Anthropic? So [clears throat] just to preface on kind of what my team does, as I mentioned, it's at this intersection of product research, uh, customer-facing interaction, and then also just actual research within
- 9:13
the org. Um, and we support, support the technical aspects of the use cases, so helping to design architectures, evals, tweak Claude prompts to get the best out of our models, et cetera.
- 9:23
And then we also bring whatever we see back into Anthropic, and we try to build some of the best products we can for our customers. So some examples of projects we've worked on or things that we've published include the Building effective, uh, research, uh, paper that, uh, my colleague Barry published.
- 9:38
He's gonna be speaking tomorrow. And then as well as that, we've launched Model Context Protocol, which is a open source protocol for language models to interact with data sources.
- 9:47
And, um, Mahesh is gonna be leading a workshop on that, uh, on Saturday, I believe.
- 9:54
So Anthropic as a whole, we try to effectively support our customers, but where we really start to embed, at least my team in particular, is, um, we work closely with customers that are using Claude a lot, and they're facing really niche challenges in specific use case domains, and they need support from our team to try to apply
- 10:10
some of the newest, kind of latest and greatest research or get the most out of the models from a prompting standpoint, et cetera. And so this approach is pretty additive.
- 10:17
We often kick off a pr-- a sprint once the customer is facing those tricky challenges. Uh, and that could be LL- LLMOps, architectures, or evals. We help to define certain metrics that they, they deem to be important when they're evaluating the model against, uh, the use case.
- 10:32
And then finally, um, we help them deploy that kind of iterative loop, the result of that into an A- AB test environment, um, and then hopefully into production. And so a part of that is the importance of evals, and I'll get onto that in a second.
- 10:45
Um, but I'm gonna pass it over first to Joe to talk about some stuff that we did for Intercom.
- 10:52
Yeah. So sort of the-- I think this is a good segue on what Alex was describing. So for those of you don't-- who don't know, Intercom is an AI customer, uh, service platform.
- 11:01
They have an AI agent called Fin. By many measures, it's the best in the market, and it's a pretty competitive market. So they had their product, uh, out for, I think, about a year or so.
- 11:12
And when we spoke to them, they shared wh- where they wanted to go, where they saw the future as-- of like customer support and agents. And based on some of the capabilities of our model, we felt that we could have a pretty good impact on these metrics.
- 11:25
And so what we started with was the Applied AI, uh, lead met with their data science team, and we ran a quick two-week sprint. We took their hardest prompt for Fin, and we compared it, uh, against a prompt that we helped them sort of figure out with, uh, with Claude.
- 11:40
And they saw really good results after the first two weeks. So much so we went on this sort of sprint of about two months where we were basically, um, fine-tuning and optimi- optimizing all of their, uh, prompts to get the best performance out of Claude.
- 11:55
At the end of this, they're able to look at all their benchmarks and see that Anthropic was outperforming the current LLM. It's also worth noting that they do a resolution-based pricing model, so there's an incentive for everyone for the model to be really helpful and help customers solve problems and not be like a deflection machine where it's
- 12:11
like, you know, we've probably all experienced them before. And so, uh, at the end of this two months, they decided to move forward with Anthropic. They launched it. You can read about it.
- 12:19
It's called Fin 2. And I think just some of the metrics are really, like, mind-blowing. Like, can solve up to eighty-six percent of customer support volume, fifty-one percent out of the box.
- 12:28
Our support team
- 12:29
Thought about lots of different options, and they actually adopted, uh, Fin as well, and they saw very similar resolution rates. But also making it more human, so they-- I think with our model, we can-- there's a much more of a human element to it, so they could do, like, uh, adjustment of the tone, uh, answer length, and
- 12:45
then it was also really good at doing policy awareness, so like refund policy, for instance. So unlocking some new capabilities. And we're thrilled to be partnering with them as they sort of, I think, march forward as a leader in, in this space.
- 12:57
Yeah. On, on a kinda separate note, one of the things I've seen recently is, uh, Claude on Twitter acting as some sort of therapist for a lot of people, and I always find that an entertaining example- [laughs] ...of, like, its character being expressed.
- 13:08
Yeah.
- 13:08
Um, cool. So let's get onto some best practices and mistakes that we see in the field, uh, on the go-to-market team. So firstly, testing and evaluation. I'm sure this-- those two words have been mentioned a lot today and probably tomorrow too.
- 13:22
Um, there are some typical common mis- common mistakes that we see, uh, customers struggling with. So the first one is they build a really robust workflow. They've spent, like, a bunch of time building some architecture out, and then they're like, "Okay, now we need to evaluate it.
- 13:34
Like, let's build some evals." That's not really how it should work in practice because your evals are actually the thing that directs you towards a perfect outcome, right? You can't build a whole workflow without evals probably from the get-go or very shortly after.
- 13:48
And so, you know, sometimes customers, as a result of struggling with data problems, might not be able to design their evals. You could use Claude to clean that up, do data reconciliation.
- 13:58
Um, or they're just, you know, trusting the vibes too much. Maybe they run a couple queries, they're like, "Hey, it looks good," right? Are they really testing that on a representative sample though?
- 14:07
Like, do you have enough s-samples to say that the thing that you're looking at is statistically significant? And like, or are you gonna, you know, run a hundred things when it actually goes into broad and then there's gonna be, like, loads of outliers, uh, because you didn't actually predict correctly what, uh, the customer is gonna ask of
- 14:22
the model, for example. So I challenge you to think about your use cases as this sort of latent space, right? Let's take this kinda chart here on the left-hand side of the slide, right?
- 14:34
As you explore the latent space with different functions that you can apply to the model, let's say prompt engineering, prompt caching, stuff like that, you're, you're basically moving your kind of position in that latent space around between l- attractor states, you know.
- 14:50
And eventually you wanna find an optimized point, but you don't really know where that is, right? Like, if you're changing an instruction, you don't know how the attention mechanism of the transformer is gonna eventually result in some different outcome that might not be performant.
- 15:03
And so the only way you can truly know that is empirically, and that's through evaluations. And so I think that's why evaluations are so, so important, and a lot of people just don't understand that soon enough.
- 15:13
In many ways, I actually tell customers, "Evals are your intellectual-- intellectual property." Like, if you wanna be competitive in a space, you need to be able to outcompete people by navigating that la- latent space and finding the attractor sta-state faster than anyone else.
- 15:29
Um, and so h-- part of, you know, how you do that is, well, firstly, setting up some sort of telemetry, uh, to back test. Ideally, that architecture's set up in advance, but, you know, you should invest in it.
- 15:41
Um, designing representative test cases. So let's say you're working on that customer support agent eval. You know, you might have a kid come on your website, let's say you're building it for, and might ask some crazy question like, "How do I, you know, kill a zombie in Minecraft?"
- 15:56
Like, totally unrelated to your product. That's, you know, still probable, and so you should probably include silly examples like that in your eval set to make sure that your model's actually approaching the response in an appropriate way or rerouting the question, et cetera.
- 16:10
Cool. Moving on to the next one. Um, identifying metrics. So a lot of the time, you know, there's this intelligence, cost, latency triangle of trade-offs that people are trying to move in between.
- 16:22
And most or-organizations can optimize for one or two of those things, but it's very difficult to meet three, at least right now. But realistically, that balance should be defined in advance, and you should know that for your specific use case, you're going to make a trade-off between those things.
- 16:39
So let's say a customer support use case again. You care about your customer getting a response within ten seconds, right? If more than ten seconds, I think there's been research done on this, uh, the customer's likely gonna just log off the page, and then they won't get the response, and then they'll probably complain about your product to
- 16:53
their friends, right? Whereas if you're looking at a financial research analyst agent, you probably don't care that it works for ten minutes to come up with the actual response to your question because the decision being made after that is very important.
- 17:05
It's an allocation of capital, for example. And so the stakes and time sensity-- sensitivity of the decision should really drive your optimization choices and, you know, maybe more instruction sets lead to longer latency but higher performance, et cetera.
- 17:20
The other thing is UX could be important, right? So again, on that customer support agent, because we spoke about Intercom, you could have different ways of circumventing that ten second to fifteen seconds.
- 17:31
Specifically, you could add like a little thinking box that bounces around. You could send the customer to another webpage in the meantime, have them read something, right? Like, there's loads of ways to distract and kind of push on those boundaries, but you still need to know what that important indicator is, and you need to optimize accordingly.
- 17:49
Finally, fine-tuning. So a lot of people, you know, I go into these calls and they're like, "Oh, we wanna do fine-tuning." I'm like, "Oh, here we go again." [laughs] Um, fine-tuning's not a silver bullet.
- 18:00
So it comes at a cost, and most people aren't aware of that cost. Um, the cost is generally you're doing brain surgery on the model, and thus there can be kind of limitations to its reasoning in other fields outside of the thing you're fine-tuning towards.
- 18:14
Um, so my encouragement is try other approaches first, right? Most people, they don't even have their eval set when they're trying to do fine-tuning, right? They need to have a cri-- clear success criteria in advance.
- 18:25
And it's like, only if we can't get that in our specific intelligence domain, do we then do fine-tuning. Don't try to boil the ocean in advance. The difference in fine-tuned capabilities and the wide variance of which, you know, failure versus success looks like in fine-tuning land means that you should be able to justify the cost of fine-tuning
- 18:46
and the effort of doing it, right, getting a team, fine-tuning, working with us, for example, uh, you should be able to justify that difference. And so in, in terms of best practices, you, you know, don't want let, to let fine-tuning slow you down, right?
- 19:00
You don't wanna say, "Oh, I'm only going to convert this language model use case if we can actually finally fine-tune our model." It's like, no, no, pursue it and then realize that you need to do fine-tuning, and then you can just sub in the fine-tuned model.
- 19:12
And then explore other methods first, and there are loads of different methods that, you know, are Anthropic as well as other companies are working on these days, and I just wanted to, like, flash up a few of them as we wrap things up here.
- 19:25
And so I'm not gonna go through all of these, but alongside just base prompt engineering, which granted is very important, there are loads of different features or architectures that will change the success of your use case drastically.
- 19:38
So for example, you might not need to sacrifice on intelligence of your model if in order to speed it up by, like, removing instructions if you can just leverage prompt caching and have a 90% recut- reduction in cost and a 50% increase in speed, right?
- 19:53
Or contextual retrieval will drastically improve the performance of your retrieval, retrieval mechanisms, and thus you feed the information to the model more effectively, and thus it has less of a time processing all the instruction set that you've given it.
- 20:06
So there are quite a few things that you can apply here, and some of them are even out of the box, like citations. And then there are also architectural decisions like agentic architectures.
- 20:15
You know, Barry, my colleague who's speaking tomorrow, will have a lot to say on that. Um, but that pretty much does it. Um, thank you so much for, uh, for your time.
- 20:25
We'll be in the theater level lounge after this chat for follow-up questions. Um, anything else from you, Joe?
- 20:32
No. Thank you so much.
- 20:34
Cool. Cheers. [outro music]