# Anthropic for VPs of AI

Alexander Bricken, Anthropic · Joe Bayley, Anthropic

AI Engineer Summit 2025

[Talk page](https://ai.engineer/talks/VhPfM_aGBVc-anthropic-vps-ai) · [Video](https://www.youtube.com/watch?v=VhPfM_aGBVc)

## Anthropic for VPs of AI

Alexander Bricken and Joe Bayley connect model capabilities to product decisions, using adaptive learning and Intercom’s Fin to explain evaluation, latency and optimization.

Familiarity with LLM prompts, APIs and basic evaluation concepts will help; no model-training background is required.

### What can your product solve now?

[0:18](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=18s)

Which customer problems can AI solve today that seemed out of reach a year ago? That is the opening business question behind Alexander Bricken and Joe Bayley’s implementation advice. Bricken works on Anthropic’s Applied AI team, helping customers implement models and carrying findings back into product and model research. Bayley works in go-to-market and describes watching capabilities evolve from **Claude 2.1** to systems supporting previously impractical applications.

Their guidance draws on colleagues’ experience and hundreds of customer interactions. It sits within Anthropic’s stated mission as an AI safety and research company: developing frontier models alongside safety techniques, research and policy. The practical challenge is to turn those capabilities into a product that reliably solves its customers’ problems.

### From model capability to interpretable features

[1:52](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=112s)

The model introduction centers on the [October 2024 upgrade of **Claude 3.5 Sonnet**](https://www.anthropic.com/news/3-5-models-and-computer-use), called Sonnet 3.5 New in the presentation. Bricken highlights its coding capabilities and describes it as leading SWE-bench at the time. Treat that as a historical claim: the release announcement documents strong SWE-bench Verified results, but does not establish the leaderboard position on the day of this talk.

![Slide with a Claude 3.5 Sonnet benchmark comparison, a Replit quotation, and use cases including computer use, code generation, advanced chatbots, knowledge Q&A, visual data extraction, and RPA.](https://ai.engineer/api/talk-essay-media/88b08001785f8fab0542f7581530c3c29d07ac2048ee4ca827808db0a0a10393)

Upgraded Claude 3.5 Sonnet: benchmarks and common use cases.

Anthropic’s capability, product and safety research overlap. **Interpretability** adds a different kind of understanding: reverse engineering what happens inside a model, identifying why particular behavior occurs, and potentially steering it toward a use case. Bricken describes the work as early, placing it informally somewhere in the first quarter to first half of the roadmap.

The proposed progression builds one capability on another:

1. **Understanding:** investigate how the model makes decisions.
2. **Detection:** recognize particular behaviors and assign meaningful labels.
3. **Steering:** influence the model’s output by intervening in its internal activity.
4. **Explainability:** turn that understanding into useful explanations and business value.

Bricken points to [Towards Monosemanticity](https://www.anthropic.com/research/towards-monosemanticity-decomposing-language-models-with-dictionary-learning) and [Scaling Monosemanticity](https://www.anthropic.com/research/mapping-mind-language-model) as research into model-level feature activations. Better detection could eventually help uncover concealed behaviors, including sleeper-agent behavior; this is a research direction, not a deployed detection guarantee.

### Recognizing basketball, steering toward a bridge

[4:08](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=248s)

Suppose a model answers a question about today’s NBA games by saying Steph Curry scored 30 points. In Bricken’s hypothetical example, that answer activates feature `304`, associated with famous NBA players. The feature represents a recognizable pattern across a group of neurons, rather than a single neuron reserved for Curry. Its meaning comes from the pattern recurring across mentions of different famous basketball players. The accompanying diagram contrasts such patterns for offers of assistance, famous NBA players, famous NFL players and sarcastic praise.

![Four columns labeled offers of assistance, famous NBA players, famous NFL players, and sarcastic praise show colored neuron markers and numerical activation values.](https://ai.engineer/api/talk-essay-media/1f0eeade67074aabd0b6efc43a68c89b7ba209045d36859345bedc27c3ce1357)

Four features illustrated as distinct patterns across six neurons.

**Golden Gate Claude** illustrates the next step: intervention rather than observation. Amplifying a Golden Gate Bridge-related activation direction made the model bring the bridge into unrelated answers. Asked what color to paint a bedroom, it would suggest bridge-like red paint and even pillars. The example makes steering tangible: changing an internal activation can change the themes that dominate an answer, even when the user did not ask for them.

### Make the core product better

[5:03](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=303s)

Choosing an AI feature starts with the problem the product already exists to solve. Bayley encourages teams to look beyond chatbots and summarization when deciding where to place larger bets. Consider an onboarding and upskilling platform: its purpose is to help employees ramp up quickly and acquire skills for their next career stage, such as public speaking or management. Summarizing a course is useful, but it only addresses part of that purpose.

A course Q&A assistant is another reasonable addition. The larger opportunity is to change the learning experience itself:

- **Personalize the content:** use each employee’s context to make lessons more relevant.
- **Adapt the difficulty:** increase the challenge when someone moves through the material too easily.
- **Update the format:** as the platform learns about the learner, generate different materials, including visual content where appropriate.

These are proposed product capabilities, not demonstrated learning outcomes. The decision is whether they advance employee development more directly than summaries or question answering. Customer domain expertise supplies the context that makes the model useful.

The same reasoning applies to tax, legal and project-management products. The goal is an experience that is easier to use and more trustworthy, with output quality appropriate to the consequences. A hallucinated tax answer is not merely awkward wording; it can cause a consequential error. Bayley describes customers using AI in these business-critical workflows, although this part of the talk supplies no customer-specific measurements.

### From model access to an evaluated deployment

[7:56](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=476s)

The presentation distinguishes two product routes and two cloud access options:

| Route | Purpose |
| --- | --- |
| Anthropic API | Embed AI in products and services |
| Claude for Work | Support employees’ everyday work |
| Amazon Bedrock | Access models within an AWS environment |
| Google Cloud Vertex AI | Access models within a Google Cloud environment |

Bayley presents the cloud routes as a way to work within existing infrastructure rather than manage an additional environment. He also says Anthropic’s support approach includes customers accessing models directly or through a third party.

Applied AI support covers architecture design, evaluations and prompt optimization, with customer findings feeding back into Anthropic’s products and research. Bricken names [Building effective agents](https://www.anthropic.com/engineering/building-effective-agents), associated with his colleague Barry, and the [Model Context Protocol](https://www.anthropic.com/news/model-context-protocol), an open protocol connecting language models with data sources. At the event, Barry’s talk was scheduled for the following day; Bricken also pointed to a forthcoming MCP workshop led by Mahesh.

Deeper collaboration begins when customers using Claude heavily encounter difficult domain-specific problems. A sprint can address LLMOps, architecture or evaluation design, but it needs metrics that the customer considers meaningful. The intended progression is to define those metrics, iterate on the system, put the resulting candidate into an A/B test, and then decide whether to move it into production.

### Fin: optimize for resolution, not deflection

[10:52](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=652s)

Intercom’s customer-service agent, Fin, had been in the market for roughly a year when the collaboration began. Intercom shared where it wanted to take the product; Anthropic saw an opportunity to improve its target metrics. An Applied AI lead and Intercom’s data scientists started with a two-week sprint, comparing Fin’s hardest prompt against a version optimized for Claude. Encouraging initial results led to approximately two months of broader prompt optimization. The work described here concerns prompts, not established model-weight training.

Bayley reports that Claude then outperformed the incumbent model across Intercom’s benchmarks. The commercial model matters to interpreting the goal: Intercom charges based on resolution, creating an incentive to solve the customer’s problem rather than merely deflect the conversation. After the optimization period, Intercom chose Anthropic and launched Fin 2.

The [Intercom case study](https://claude.com/customers/intercom) reports resolution of up to 86% of customer-support volume, with a 51% average resolution rate out of the box. The maximum and the average describe different conditions; the latter is before customization or tuning. These vendor-reported figures do not specify the measurement window, cohort or operational definition of resolution.

Anthropic’s own support team also adopted Fin and reportedly saw similar resolution rates. Beyond resolution, the presentation highlights control over tone and answer length, plus policy awareness—for example, following refund policies. Bricken closes the customer discussion with an anecdote about people treating Claude as a therapist on Twitter, illustrating its perceived conversational character rather than a clinical capability.

### Build the evaluation before the architecture hardens

[13:08](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=788s)

A recurring mistake is to build a substantial workflow and only then ask how to evaluate it. **Evaluations should direct development**, beginning at the outset or shortly afterward. Otherwise, the architecture accumulates assumptions before the team has a way to tell whether those assumptions improve the outcome.

Poor data quality can block that work; Bricken suggests using Claude for cleanup and reconciliation. Another failure is accepting a few convincing answers as sufficient evidence. Tests need representative inputs and enough samples to support the conclusion being drawn. His example of unexpected outliers appearing in the first hundred production queries is a warning about narrow testing, not a prescription for a hundred-case evaluation set.

Bricken describes optimization as moving through a latent space between attractor states. Prompt engineering and prompt caching are among the interventions he names. The useful point of the metaphor is uncertainty: editing an instruction does not tell you, from inspection of transformer attention, how the resulting application will behave. You have to measure the outcome empirically.

**Evaluations are intellectual property** because they help a team find useful configurations faster than competitors. Telemetry supports that advantage by making real interactions available for backtesting. A customer-support evaluation should include plausible off-topic inputs, such as a child asking how to kill a zombie in Minecraft. The expected behavior is an appropriate response or rerouting, rather than assuming every user will ask a relevant product question.

![Slide lists delayed empirical testing, overlooked data quality, and vibes-based assessments as mistakes, alongside evaluation frameworks, telemetry, and tests aligned with real-world use cases.](https://ai.engineer/api/talk-essay-media/e5cb398fad879d24ea9a14e8ba7f80dd2ea7e1373452b3dfcb930e29af6f63a2)

Testing and evaluation: common mistakes and best practices.

A small evaluation record can make that expectation explicit:

```json
{
  "id": "off-topic-minecraft",
  "input": "How do I kill a zombie in Minecraft?",
  "expected_behavior": {
    "recognize_out_of_scope": true,
    "redirect_to_product_support": true
  },
  "grading_questions": [
    "Does the response recognize that the question is unrelated to the product?",
    "Does it offer an appropriate route back to product support?"
  ]
}
```

This turns the example into a reviewable test case. A grader still needs to inspect the actual response against the criteria; simply storing the expected behavior does not establish that the agent follows it.

### Let the decision set the latency budget

[16:10](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=970s)

Application design involves a triangle of **intelligence, cost and latency**. Bricken says teams can often optimize one or two dimensions, while improving all three is difficult. The balance should be chosen in advance for the use case, rather than discovered accidentally after deployment.

| Use case | Illustrative time budget | Reason |
| --- | --- | --- |
| Customer support | A response within 10 seconds | The customer is waiting in an active interaction |
| Financial research | Potentially 10 minutes | The answer may inform capital allocation |

Bricken uses these as contrasting design examples. His suggestion that support users abandon a page after ten seconds invokes unspecified research, so it should not be treated as a measured universal threshold. The underlying principle is that the stakes and time sensitivity of the decision determine the acceptable wait. More instructions may be worthwhile if they improve performance, even when they increase latency.

The interface can also affect how waiting feels. For a support response taking ten to fifteen seconds, Bricken suggests an animated thinking indicator or another page the customer can read while waiting. Those choices may make the delay more tolerable, but they do not remove the need to define the important latency metric and optimize the system against it.

### Make fine-tuning earn its cost

[17:49](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1069s)

Fine-tuning is not a default starting point. Bricken compares it to brain surgery: changing the model for one domain can impair reasoning elsewhere. That possibility adds to the direct costs of data preparation, specialist effort and experimentation. Before tuning, a team needs an evaluation set and explicit success criteria.

The decision sequence is straightforward:

1. Define the performance required in the target domain.
2. Try other approaches and measure the remaining gap.
3. Pursue fine-tuning only when the potential improvement justifies its cost and variable outcomes.
4. Keep developing the application; substitute a fine-tuned model later if the evidence supports it.

Making fine-tuning a prerequisite can delay a useful product before the team has established whether tuning is necessary.

### Use the capabilities already available

[19:12](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1152s)

Prompt engineering remains important, but it is only one intervention. Features and architecture choices can change the economics and quality of an application without changing model weights.

**Prompt caching** is an alternative to deleting useful instructions merely to make requests cheaper or faster. Bricken cites a 90% cost reduction and a 50% increase in speed for prompt caching, without specifying workload or cache-hit conditions. The [historical caching explanation](https://claude.com/blog/prompt-caching) gives the narrower interpretation: repeated context can be reused, cached input reads cost one tenth of the base input price, and cache writes cost extra. That input discount is not a 90% reduction in the entire request bill. Its latency examples measure time to first token, so they do not establish a universal improvement in completion speed.

**[Contextual Retrieval](https://www.anthropic.com/engineering/contextual-retrieval)** addresses the information supplied to the model. Improving retrieval can make relevant evidence easier to find and deliver. Bricken also suggests a reduced processing burden, but the supporting research establishes retrieval-quality improvements rather than a general guarantee of faster generation.

**Citations** and **agentic architectures** round out the alternatives: built-in capabilities can support source attribution, while architecture determines how the application organizes model work. The closing feature table places these alongside tool use, batch processing, PDF handling, token counting, prompt tools, computer use and MCP. The practical next experiment should follow the measured bottleneck—repeated context, missing evidence or workflow design—before the team sacrifices useful instructions or commits to fine-tuning.

![Table titled Using the model capabilities correctly lists contextual retrieval, prompt caching, Batch API, tool use, agentic architectures, PDF rasterization, token counting, prompt tools, computer use, citations, and Model Context Protocol.](https://ai.engineer/api/talk-essay-media/7bc6da388a5f064d460180dae075efd57f31b5532a89435fa49806895e439d4a)

Model features and architectures paired with their outcomes.

## Resources

- [The October 2024 Claude 3.5 Sonnet upgrade](https://www.anthropic.com/news/3-5-models-and-computer-use) — Historical release announcement with coding benchmark results and deployment availability.
- [Towards Monosemanticity](https://www.anthropic.com/research/towards-monosemanticity-decomposing-language-models-with-dictionary-learning) — Anthropic's research introduction to decomposing model activations into interpretable features.
- [Mapping the mind of a large language model](https://www.anthropic.com/research/mapping-mind-language-model) — An accessible introduction to interpretable features and steering model behavior.
- [Introducing the Model Context Protocol](https://www.anthropic.com/news/model-context-protocol) — Anthropic's introduction to an open protocol connecting AI applications with data sources and tools.
- [Intercom's Fin and Claude case study](https://claude.com/customers/intercom) — The customer-support case study reporting up to 86% resolution and a 51% average out-of-the-box rate.
- [Prompt caching with Claude](https://claude.com/blog/prompt-caching) — Explains repeated-context caching, historical pricing and workload-specific latency examples.
- [Introducing Contextual Retrieval](https://www.anthropic.com/engineering/contextual-retrieval) — Explains contextual embeddings, lexical retrieval and reranking, with implementation guidance.
- [Building effective agents](https://www.anthropic.com/engineering/building-effective-agents) — Practical guidance on choosing simple workflows and agent architectures.

## Slides

### Anthropic overview

Founded in 2021, Anthropic presents its mission as guiding the world safely through transformative AI. Its founders include former OpenAI leaders in research, safety, policy and engineering.

[1:24](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=84s) [Anthropic overview with founding year, headquarters, mission and founder biographies.](https://assets.ai.engineer/sites/aiecode2025/sha256/07a9f10934e78571228484c39741e62ab1ccdb370c29491fd138de8d0f03f8a4.webp)

ANTHROPIC

Founded
2021

Employees
1000+

HQ
San Francisco

Mission
To ensure the world safely makes the transition through transformative AI.

FOUNDED BY TOP AI EXPERTS

Dario Amodei — CEO & Co-founder / Former VP of Research at OpenAI
Daniela Amodei — President & Co-founder / Former VP of Safety at OpenAI
Jared Kaplan — CSO & Co-founder / Lead author on “Scaling Laws” paper
Sam McCandlish — CTO & Co-founder / Former AI Safety Research Lead at OpenAI
Jack Clark — Co-founder / Former Policy Director at OpenAI
Chris Olah — Co-founder / Former Interpretability Lead at OpenAI
Tom Brown — Co-founder / Former GPT-3 Engineering Lead at OpenAI
Ben Mann — Co-founder / Former GPT-3 and API Engineer at OpenAI

ANTHROPIC | Our business
CONFIDENTIAL
3

### Upgraded Claude 3.5 Sonnet

The upgraded Claude 3.5 Sonnet scores 49.0% on SWE-bench Verified, versus 33.4% for the earlier Sonnet. The slide also lists computer use, code generation, chatbots, knowledge Q&A, visual extraction, and RPA as use cases.

[1:48](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=108s) [Slide describing upgraded Claude 3.5 Sonnet, with benchmark comparisons, use cases and a Replit testimonial.](https://assets.ai.engineer/sites/aiecode2025/sha256/2f97a9e8b61a7b820f803c35d696efe7122bcd2a4df9edf7c59f001c0f5a9f21.webp)

Upgraded Claude 3.5 Sonnet

The latest version of Claude 3.5 Sonnet strikes the ideal balance between market-leading intelligence, speed, and cost.

It is the state-of-the-art model for real-world software engineering tasks, agentic capabilities, and computer use.

Anthropic’s most intelligent model yet

Columns: Claude 3.5 Sonnet (upgraded) | Claude 3.5 Sonnet | GPT-4o
Graduate level reasoning, GPQA (Diamond): 65.0% (0-shot CoT) | 59.4% (0-shot CoT) | 53.6% (0-shot CoT)
Undergraduate level knowledge, MMLU Pro: 78.0% (0-shot CoT) | 75.1% (0-shot CoT) | —
Agentic coding, SWE-bench Verified: 49.0% | 33.4% | —
Code, HumanEval: 93.7% (0-shot) | 92.0% (0-shot) | 90.2% (0-shot)
Math problem-solving, MATH: 78.3% (0-shot CoT) | 71.1% (0-shot CoT) | 76.6% (0-shot CoT)

Common use cases
• Computer use
• Code generation
• Advanced chatbots
• Knowledge Q&A
• Visual data extraction
• RPA

“The new Claude 3.5 Sonnet goes above and beyond our expectations...This is the breakthrough we’ve been waiting for.”
— Michele Catasta, President
replit

CONFIDENTIAL 4

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

[2:07](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=127s) [Slide comparing Claude 3.5 Sonnet benchmarks and listing common use cases.](https://assets.ai.engineer/sites/aiecode2025/sha256/f5bed7b8eeba68387d68bbf8a7fcd44f0cf7fa25f6ac894745b7ca530c7374ca.webp)

Upgraded Claude 3.5 Sonnet

The latest version of Claude 3.5 Sonnet strikes the ideal balance between market-leading intelligence, speed, and cost.

It is the state-of-the-art model for real-world software engineering tasks, agentic capabilities, and computer use.

Anthropic’s most intelligent model yet

Columns: Claude 3.5 Sonnet (upgraded) | Claude 3.5 Sonnet | GPT-4o
Graduate level reasoning, GPQA (Diamond): 65.0% (0-shot CoT) | 59.4% (0-shot CoT) | 53.6% (0-shot CoT)
Undergraduate level knowledge, MMLU Pro: 78.0% (0-shot CoT) | 75.1% (0-shot CoT) | —
Agentic coding, SWE-bench Verified: 49.0% | 33.4% | —
Code, HumanEval: 93.7% (0-shot) | 92.0% (0-shot) | 90.2% (0-shot)
Math problem-solving, MATH: 78.3% (0-shot CoT) | 71.1% (0-shot CoT) | 76.6% (0-shot CoT)

Common use cases
• Computer use
• Code generation
• Advanced chatbots
• Knowledge Q&A
• Visual data extraction
• RPA

“The new Claude 3.5 Sonnet goes above and beyond our expectations...This is the breakthrough we’ve been waiting for.”
— Michele Catasta, President
replit

ANTHROPIC
CONFIDENTIAL 4

### Long-term research

Anthropic’s research spans model capabilities, products, and safety. The speaker singles out interpretability: reverse engineering models to understand how they work, with steering as a potential application.

[2:18](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=138s) [Slide outlining interpretability, alignment, pretraining, retrieval, and agent orchestration research.](https://assets.ai.engineer/sites/aiecode2025/sha256/b4dd5d6ee0100bb5f49bff70107054ceea2bb263f0f20668fa2ef57789a86875.webp)

Long-term research

Interpretability
Deepening our understanding of how LLMs work.
“Reverse engineering” info by looking at digital neurons to provide insights and help you audit your completions

Alignment
Includes efforts to improve the ways we finetune models, make them more honest, allow them to use tools, and more

Pretraining
Improving the capabilities of the 'base model' including adding new abilities like multimodal reasoning

Enhancing retrieval
Developing advanced retrieval techniques, coupled with long contexts and efficient prompt caching, to improve knowledge use cases

Agents and orchestration
Research and SDKs to help Claude call Claude, and build multi-step agentic workflows including long-horizon coding tasks

ANTHROPIC
CONFIDENTIAL
5

[2:42](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=162s) [Overview of research in interpretability, alignment, pretraining, retrieval, and agent orchestration.](https://assets.ai.engineer/sites/aiecode2025/sha256/f60606af699941b910a1d63e0cffbe28b0844f65ade9ddde7526b18fe16af10b.webp)

Long-term research

Interpretability
Deepening our understanding of how LLMs work.
“Reverse engineering” info by looking at digital neurons to provide insights and help you audit your completions

Alignment
Includes efforts to improve the ways we finetune models, make them more honest, allow them to use tools, and more

Pretraining
Improving the capabilities of the 'base model' including adding new abilities like multimodal reasoning

Enhancing retrieval
Developing advanced retrieval techniques, coupled with long contexts and efficient prompt caching, to improve knowledge use cases

Agents and orchestration
Research and SDKs to help Claude call Claude, and build multi-step agentic workflows including long-horizon coding tasks

CONFIDENTIAL
5

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Phases of applied interpretability

Interpretability builds from identifying features and detecting behaviors to steering outputs and explaining decisions in a business context. The speaker estimates research may be only in the first 25% of the timeline.

[2:49](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=169s) [Timeline comparing understanding, detection, steering, and explainability by success criteria, methodology, and user benefit.](https://assets.ai.engineer/sites/aiecode2025/sha256/46610fb61b7fda0333b3e28f282d656cd42c5efc77436c879bc2515db8dec12a.webp)

Phases of applied interpretability

Timeline: Now → Near-term → Mid-term → Long-term

Understanding
Success criteria: Understand basic components of how AI models make decisions
Methodology: Identify and classify features, the building blocks of how models process information
User benefit: Demonstrate research direction, but no direct benefit at this phase

Detection
Success criteria: Detect specific behaviors or outputs from AI models
Methodology: Identify when the model might be hallucinating, showing bias, and potential jailbreaks or focus on a certain topic
User benefit: Build trust that Claude models are safe for large-scale, highly sensitive use cases

Steering
Success criteria: Influence specific behaviors or outputs from AI models
Methodology: Emphasize or suppress certain features to guide the model’s behaviors and outputs
User benefit: Offer mechanisms for customers to use prompt or steering techniques

Explainability
Success criteria: Trace, influence and rationalize the AI model’s decision-making in the broader business decision making context
Methodology: Understand feature <> behavior and causality to show how it arrived at a particular output.
User benefit: Help customers better control Claude’s outputs at a granular level across different scenarios

CONFIDENTIAL
6

AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Feature activation patterns

The famous NBA players feature corresponds to a recognizable pattern across neurons. Its strongest shown activations, neurons 4 and 6, also appear in the NFL feature.

[4:11](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=251s) [Diagram comparing neuron activation patterns for assistance, basketball players, football players, and sarcastic praise.](https://assets.ai.engineer/sites/aiecode2025/sha256/684df06a91b1b5c138039f7f354d807173f8f2cbf5897dc1962980e7ab930db2.webp)

#1207 Offers of assistance
#304 Famous NBA players
#127 Famous NFL players
#10287 Sarcastic praise

Neuron activations, in the column order above:
neuron 1: 0.0, 0.0, 0.0, 0.0
neuron 2: 0.3, 0.3, 0.2, 0.1
neuron 3: 0.8, 0.0, 0.0, 0.6
neuron 4: 0.0, 0.8, 0.7, 0.1
neuron 5: 0.0, 0.2, 0.9, 0.0
neuron 6: 0.0, 0.8, 0.8, 0.0

ANTHROPIC
7

[4:36](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=276s) [A table compares six neuron activations across four example features.](https://assets.ai.engineer/sites/aiecode2025/sha256/304038c9a0d658e4107ad99894e729b4fe07298007a6595ae71d9c5319121e42.webp)

#1207 Offers of assistance
#304 Famous NBA players
#127 Famous NFL players
#10287 Sarcastic praise

Neuron activations, in the column order above:
neuron 1: 0.0, 0.0, 0.0, 0.0
neuron 2: 0.3, 0.3, 0.2, 0.1
neuron 3: 0.8, 0.0, 0.0, 0.6
neuron 4: 0.0, 0.8, 0.7, 0.1
neuron 5: 0.0, 0.2, 0.9, 0.0
neuron 6: 0.0, 0.8, 0.8, 0.0

7

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### AI for onboarding and upskilling

An onboarding platform could help employees build skills through personalized courses, adaptive difficulty, and content updates informed by quizzes and surveys.

[5:51](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=351s) [The slide suggests personalized courses, adaptive difficulty, and content updates for employee training.](https://assets.ai.engineer/sites/aiecode2025/sha256/e7f9f0837c2fff5b0ca2ae1fec61a443ff36069817a437e0850060c938621a2d.webp)

Back to basics: Use AI to solve the core problem your product is trying to solve.

What is the core problem your product solves for users?

How is an AI-native start-up going to approach the problem differently?

Think beyond chatbots and summarization

Example:
Product: SaaS onboarding and upskilling platform.

Core problem you are solving:
Getting employees proficient in their roles as quickly as possible while ensuring they develop the right skills for their growth path.

Instead of a chatbot that answers onboarding questions, can you:
• Create hyper personalized courses based on employee individual context
• Adapt content difficulty based on progress and comprehension
• Automatically update content by identifying and learning from quizzes and surveys

AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Use AI to solve the core product problem

An onboarding platform could use AI to personalize courses, adapt difficulty to employee progress, and update material from quizzes and surveys—instead of stopping at a Q&A chatbot.

[5:53](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=353s) [Slide asks how AI-native products can solve users’ core problems, using personalized and adaptive employee onboarding as an example.](https://assets.ai.engineer/sites/aiecode2025/sha256/9de65eb7db60b36be07a78bc41c420926dc5839ea5ec6bf692412270f9756ac5.webp)

Back to basics: Use AI to solve the core problem your product is trying to solve.

What is the core problem your product solves for users?

How is an AI-native start-up going to approach the problem differently?

Think beyond chatbots and summarization

Example:
Product: SaaS onboarding and upskilling platform.

Core problem you are solving:
Getting employees proficient in their roles as quickly as possible while ensuring they develop the right skills for their growth path.

Instead of a chatbot that answers onboarding questions, can you:
• Create hyper personalized courses based on employee individual context
• Adapt content difficulty based on progress and comprehension
• Automatically update content by identifying and learning from quizzes and surveys

AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Use AI to solve the core product problem

For an onboarding platform, AI could personalize courses to each employee, adapt difficulty as they progress, and update content using quizzes and surveys—all aimed at building proficiency and skills.

[6:13](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=373s) [Slide proposes personalized, adaptive onboarding as an example of solving a product’s core problem with AI.](https://assets.ai.engineer/sites/aiecode2025/sha256/07ee940357ef950efea2908467fa25bd0feb4295e571668f9c50a9d339c7e76c.webp)

Back to basics: Use AI to solve the core problem your product is trying to solve.

What is the core problem your product solves for users?

How is an AI-native start-up going to approach the problem differently?

Think beyond chatbots and summarization

Example:
Product: SaaS onboarding and upskilling platform.

Core problem you are solving:
Getting employees proficient in their roles as quickly as possible while ensuring they develop the right skills for their growth path.

Instead of a chatbot that answers onboarding questions, can you:
• Create hyper personalized courses based on employee individual context
• Adapt content difficulty based on progress and comprehension
• Automatically update content by identifying and learning from quizzes and surveys

AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Customers achieve industry-leading results

TurboTax processes tax material, Thomson Reuters supports tax and legal analysis, and Asana applies AI to goal clarity and workflows. Multiple firms report cutting work time in half.

[7:26](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=446s) [Customer results and testimonials for TurboTax, Thomson Reuters, and Asana.](https://assets.ai.engineer/sites/aiecode2025/sha256/7966622fb7f79e52b135092dce9433dfbf828d12fb4747083ad3071c40459015.webp)

Customers achieve industry-leading results

Intuit TurboTax
Processes tax information from 100,000+ pages and 15,000+ federal/state tax forms
Serves millions of TurboTax customers
Improved filing completion rates and user confidence

“We are incredibly excited about Anthropic's current Claude models and future developments, as each advancement has the potential to unlock more capabilities”
Nhung Ho, VP of AI

THOMSON REUTERS
Powers AI features from 3,000+ subject matter experts
Multiple firms report cutting work time in half
Built comprehensive RAG architecture for tax and legal analysis

“The risk of providing wrong advice is substantial, so our quality thresholds are extraordinarily high. The partnership with Anthropic has been exceptional”
Joel Hron, CTO

asana
Serves 150,000+ global customers. AI features enhance goal clarity and reduce executive decision-making time
Multiple enterprise use cases including streamlined workflows and creative request management

“What's cool about AI now is doing things we thought would require years of engineering effort, in just a few weeks with a small team”
Eric Pelz, Head of Technology for AI

CONFIDENTIAL 10

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Getting Started

The speaker begins the getting-started section by introducing Anthropic’s products.

[7:53](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=473s) [Getting Started section title on a peach background.](https://assets.ai.engineer/sites/aiecode2025/sha256/7136014a46fda552353cf950d3174a836070e3df22c3e91134918bede83c3d25.webp)

Getting Started

CONFIDENTIAL
11

AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Claude for Work and Anthropic API

Claude for Work gives employees access to AI for day-to-day work; the Anthropic API lets businesses build AI into products and services.

[8:08](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=488s) [A comparison of use cases for Claude for Work and the Anthropic API, with product interface examples.](https://assets.ai.engineer/sites/aiecode2025/sha256/52d1c98a5ec7dcef7c5d38e7a8f63818bb321dee8f2c2c17f0ae2bbab02e636c.webp)

Claude for Work
Empower every employee with AI

Best fit use cases:
Offer employees a secure way to access AI for work initiatives
Scale organizational knowledge and expertise more effectively
Improve operational efficiency and increase collaboration around AI

Anthropic API
Build AI-powered applications

Best fit use cases:
Create revenue streams by building AI into your products and services
Reduce operational costs by streamlining workflows with AI
Improve user engagement with AI-driven experiences

ANTHROPIC | Our products
CONFIDENTIAL
12

Interface text:
Market analysis
This project contains the app repo as well as strategy and customer documents to help build features.
Sales
Chat with Code Helper...
Activity
shared by Ahsam
Project knowledge
Add
71% of project knowledge size used
Q3_2023_SalesReport.csv
Q4_2023_SalesReport.csv
Company OKRS.doc
Marketing Strategy 2024.doc
Product Strategy 2024.doc
Q3_2023_SalesReport.xlsx

ANTHROPIC
Dashboard  Workbench  Settings
Support Ticket Classification
Revisions
Revision 1
Mar 26, 2024, 1:26 PM
SYSTEM PROMPT  Set a System Prompt (Optional)
USER
You are a support representative that wants to help users connect with the right team of live agents for hands-on support.
Add Messages
Claude 3 Opus 20240229

### Partnership with two of the largest cloud providers

Anthropic’s AWS and Google Cloud partnerships let customers access frontier models through Bedrock or Vertex in their existing cloud environment, with support whether they access Anthropic directly or through a third party.

[8:21](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=501s) [A slide showing AWS and Google Cloud logos alongside unified, secure, scalable and holistic partnership benefits.](https://assets.ai.engineer/sites/aiecode2025/sha256/d871e44db41729154fe6099c328ba82afdac6893ba286bc959987a22cb3881ae.webp)

Partnership with two of the largest cloud providers

aws

Unified
Access frontier models within your cloud ecosystem to keep models and data in the same cloud environment.

Secure
Meet stringent regulatory requirements via Anthropic and your cloud provider’s security features and compliance certifications.

Scalable
Build AI solutions that adapt to your dynamic demands with flexible pricing, auto-scaling, and advanced customization options.

Holistic
Get comprehensive support and access to experts for holistic guidance throughout your AI journey.

Our products
CONFIDENTIAL
13

AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

[8:41](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=521s) [Slide listing AWS and Google Cloud and describing unified, secure, scalable and holistic benefits.](https://assets.ai.engineer/sites/aiecode2025/sha256/4914c2852a154e5f3e389d32985c13ddd6e9514dc9ca0f1ae382441a0cbc7055.webp)

Partnership with two of the largest cloud providers

aws

Unified
Access frontier models within your cloud ecosystem to keep models and data in the same cloud environment.

Secure
Meet stringent regulatory requirements via Anthropic and your cloud provider’s security features and compliance certifications.

Scalable
Build AI solutions that adapt to your dynamic demands with flexible pricing, auto-scaling, and advanced customization options.

Holistic
Get comprehensive support and access to experts for holistic guidance throughout your AI journey.

Our products
CONFIDENTIAL
13

AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Anthropic Applied AI

The Applied AI team helps customers design architectures, evaluations, and prompts, then brings what it learns back into Anthropic’s product work. Its projects include Building effective agents and Model Context Protocol.

[9:07](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=547s) [Slide describes the Applied AI team's customer and product work alongside examples of its publications.](https://assets.ai.engineer/sites/aiecode2025/sha256/9a33bc1d963990deabfd1a9fb55d24c08a76d40d9cdaee36c468e57d98c88cba.webp)

Anthropic Applied AI

• The Anthropic Applied AI team sits at the edge of product, research, and customer interactions.
• Many of our most successful customers have benefitted from a deep engagement with our Applied AI team.
• Applied AI also focuses on building and testing features we launch: working on many of the product/research releases you’ve seen from Anthropic over the past 12 months.

Building effective agents
Dec 19, 2024

Over the past year, we’ve worked with dozens of teams building large language model (LLM) agents across industries. Consistently, the most successful implementations weren’t using complex frameworks or specialized libraries. Instead, they were building with simple, composable patterns.

In this post, we share what we’ve learned from working with our customers and building agents ourselves, and give practical advice for developers on building effective agents.

Introducing the Model Context Protocol
Nov 25, 2024 · 3 min read

Introducing Contextual Retrieval
Sep 19, 2024 · 10 min read

ANTHROPIC
CONFIDENTIAL
15

### A Path to Production Readiness

Production readiness rises from task description and dataset curation through metric definition. Prompt iteration and output evaluation form a loop, followed by A/B testing and, hopefully, production release.

[9:52](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=592s) [Diagram progressing from task description and dataset curation through metrics, prompt iteration and evaluation to production release.](https://assets.ai.engineer/sites/aiecode2025/sha256/1146b21af13352b0d7666ed50732c24c271b52d196a473e6f23d1f65d0dab860.webp)

A Path to Production Readiness

Y-axis: Production Readiness
X-axis: [illegible]ime

Task Description (Prompt Creation)
Dataset Curation
Metric Definition
Prompt Iteration
Output Evaluation
A/B Evaluation
Production Release

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

[9:58](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=598s) [A rising sequence from task description and dataset curation through metric definition, prompt iteration, output evaluation, and A/B evaluation toward production release. Arrows show iterative evaluation loops.](https://assets.ai.engineer/sites/aiecode2025/sha256/7900c4141ee781c4615746cd666400cc4727ea0ca1245b2dd9de20868e0546ff.webp)

A Path to Production Readiness

Y-axis: Production Readiness
X-axis: [illegible]ime

Task Description (Prompt Creation)
Dataset Curation
Metric Definition
Prompt Iteration
Output Evaluation
A/B Evaluation
Production Release

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

[10:31](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=631s) [A staircase diagram progresses from task description and dataset curation through metrics, prompt iteration, output evaluation and A/B evaluation toward production release.](https://assets.ai.engineer/sites/aiecode2025/sha256/67f84487aeff159c2bd20f429560514e7b933627be656b352fe0e3f22284ce2e.webp)

A Path to Production Readiness

Y-axis: Production Readiness
X-axis: [illegible]ime

Task Description (Prompt Creation)
Dataset Curation
Metric Definition
Prompt Iteration
Output Evaluation
A/B Evaluation
Production Release

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Intercom Fin Case Study

After a prompt review and A/B test, Anthropic’s Applied AI team spent months optimizing Fin’s workflow. The slide reports up to 86% of support volume resolved and a 51% average resolution rate out of the box with little tuning.

[10:58](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=658s) [Intercom case study describing an optimization sprint and reported Fin resolution rates of up to 86% and 51% out of the box.](https://assets.ai.engineer/sites/aiecode2025/sha256/8489a61feccf321a015c7398f6ac787e1caa2cd30fae72cad10f370ce3e9444b.webp)

intercom

Intercom’s Fin AI agent delivers up to 86% resolution rates with Claude

“With Claude, we're not just automating customer service—we're elevating it to truly human quality. This lets support teams think more strategically about customer experience and what makes interactions genuinely valuable."

Fergal Reid,
VP of AI, Intercom

"The partnership with Anthropic was excellent because we had very good solutions engineers who shared important expertise"

Pedro Tabacof
Staff Machine Learning Scientist, Intercom

Challenge
Further increase LLM performance for Fin, their flagship AI agent that delivers human quality customer support responses at scale.

Applied AI Impact
After an initial prompt review and encouraging A/B test, the Anthropic Applied AI team engaged in a multi-month sprint to optimize every aspect of Fin’s agentic workflow.

Results
• Fin resolves up to 86% of customer support volume
• Fin delivers a 51% average resolution rate out of the box, with little tuning
• New capabilities unlocked, including custom tone of voice, answer length controls, and policy-aware responses.

ANTHROPIC
17

### Testing & Evaluation

Start evaluations early to guide workflow decisions. Test representative use cases with measurable criteria, and use telemetry to understand LLM performance.

[13:43](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=823s) [Common evaluation mistakes and best practices, including empirical testing, telemetry, and user-centric metrics.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a79f7880180558ca8423c40557b9bbe55c9bcd173a8d62b63285a3e76ae8a7e.webp)

Testing & Evaluation

Common Mistakes:
• Taking too long without empirical testing to drive decision-making
• Overlooking data quality and infrastructure needed for meaningful evaluations
• Relying on subjective "vibes-based" assessments rather than quantifiable metrics

Best Practices:
• Implement comprehensive evaluation frameworks to guide decision-making
• Invest in telemetry to track LLM calls and understand system performance
• Design tests aligned with real-world use cases and validate with user-centric metrics

CONFIDENTIAL
19

AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

## Full transcript

[0:00](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=0s) [on-hold music] I'm Alexander Bricken.

[0:18](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=18s) I'm on the Applied AI team at Anthropic, so I work very closely with customers to do technical implementation work, and I also bring that advice back to product research and model research.

[0:29](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=29s) Um, I'm gonna pass it over to Joe.

[0:31](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=31s) Hey, everyone. It's great to be here. My name is Joe Bayley. I work on the go-to-market team at Anthropic. I joined Anthropic, uh, over a year ago now, so I've seen our models evolve from, uh, two point one to today's capabilities.

[0:42](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=42s) And I think day to day what's really exciting is we're working with AI leaders who are solving real business problems, um, that just seemed impossible a year ago. So really excited about how quickly everything is moving.

[0:55](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=55s) Okay. For today, we will do, um, a quick overview, uh, you know, who we are, o-our mission, and then we'll focus a lot on implementing AI and best practices and common mistakes.

[1:08](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=68s) Uh, Alex and I actually didn't just take this from our own experience, but we, um, talked to a number of our colleagues, so this is all based on hundreds and hundreds of customer interactions.

[1:18](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=78s) Um, so we hope there's some actionable insights to take out of this.

[1:23](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=83s) Awesome. So what is Anthropic? So we are an AI safety and research company building the world's best and safest, uh, large language models. We were founded a few years ago by some of the leading experts in AI.

[1:37](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=97s) And since our inception, we've not only released, uh, multiple iterations of our frontier models, we've done so while being at the bleeding edge of safety techniques, of research and policy.

[1:48](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=108s) I'm gonna pass it over to Alex to talk a little about our marquee model.

[1:52](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=112s) Awesome. And so some of you are probably familiar, but the most recent model we launched was Sonnet 3.5 New, uh, in late October of last year. Um, you might be familiar with it because if you're a developer, uh, Sonnet is actually one of the leading models in the code space.

[2:08](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=128s) So if you're familiar with evaluations like SWE-bench, which is an, an agentic coding eval, uh, Sonnet is still at the top of the leaderboard for that. Um, I won't go too much into the details on the eval side, so let's keep moving.

[2:21](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=141s) Um, so yeah, in addition to what Joe mentioned, we have a lot of different research directions that we're focused on. Um, and these are really distributed but have overlap between, you know, model capabilities, product research, and AI safety.

[2:35](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=155s) The one that differentiates us, I would say, is the interpretability, and this realistically is reverse engineering the models and trying to figure out actually how they're thinking and maybe why they're thinking, and then a-an additional capability in terms of steering them in the right direction, depending on a use case.

[2:50](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=170s) So let's dive into that a little bit more. We're still very early in interpretability research, it's worth mentioning. As you can see, there's kind of like a longer timeline, and we're really only at the, the first half of that, maybe even the first twenty-five percent.

[3:02](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=182s) Um, but we're, we're really approaching it in these stages that build upon each other. So these in-include things like understanding, so grasping AI decision-making. Detection, so actually being able to understand specific behaviors and put labels on those.

[3:16](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=196s) Steering, so influencing the AI output in some, some way, shape, or form, and I'll get to an example of that in a second. And then finally, explainability, and that's really where you unlock business value associated with interpretability methods.

[3:28](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=208s) And so while we see interpretability in the long term providing, you know, a lot of significant improvements in AI safety, reliability, and usability, specifically, our interpretability team uses methods to understand feature activations at the model level, and then has published research on these, uh, in-- Towards Monosemanticity and Scaling Monosemanticity, which are two papers I highly

[3:50](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=230s) recommend. Um, and then as the technology improves into kinda detection landscapes, for example, you can imagine having a much better grasp at, uh, at the actual thinking and behavior of the model or even discovering sleeper agents for safety reasons that might be buried deep within, uh, model capabilities.

[4:08](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=248s) So a good example of that is imagining you ask the model, "What were the scores of the NBA matches today?" Right? And let's say it knows the answer, and it says how Steph Curry, you know, scored thirty points.

[4:20](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=260s) This would lead to a feature activation of, for example, feature number [REDACTED:generic_id], famous NBA players. Realistically, that's a group of neurons activating in a recognizable pattern that we've identified across all mentions of famous basketball players when a model's answering a question, not just Steph Curry.

[4:40](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=280s) Um, and you also might have heard of Golden Gate Claude. That was an example of us steering the model, uh, basically amping up the activation, uh, in the Golden Gate direction.

[4:49](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=289s) And thus, whenever you'd ask a question like, "What should I paint my bedroom?" Claude would respond, "Oh, you should paint it red like the Golden Gate Bridge, and maybe it should have some like, you know, pillars in it or something."

[5:00](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=300s) I'm gonna pass it over to Joe to talk a little bit about some of the customers we work with.

[5:03](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=303s) Yeah. So I'm gonna frame this in two ways. One is, uh, sort of early on discussions, and the other would be just examples of customers that are doing really cool things.

[5:12](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=312s) So in conversations, there's obviously a lot of noise and buzz and everything, and that's fantastic, but we often encourage our customers, uh, to sort of get back to the basics and how can they use AI to solve, uh, the core problem that your product is trying to solve.

[5:28](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=328s) We also get to work with a ton of, uh, sort of AI native, like, AI startups, and this is how they're thinking about their product. And I think you wanna move beyond, uh, chatbots and summarization.

[5:39](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=339s) These can be great options, but I'd be thinking more like, where do you wanna place bigger bets? And to give an example, um, if you just click one more time.

[5:47](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=347s) Fancy slide. Uh, imagine you're an onboarding and upskilling platform. The problem that you solve for customers is you help them get-- you ramp really quickly, and then you help them get to the next phase o-of their career by equipping them with skills.

[6:01](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=361s) So for instance, it might be public speaking you wanna get good at, or you might wanna become a manager. And so it would be easy to say, "Okay, let's summarize course content."

[6:09](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=369s) Or let's, um, [lip smack] uh, let's, uh, have a Q&A chatbot that answers questions along the way, and they could be helpful. But I'd actually think about it differently. So what about if you could hyper-personalize, uh, course content based on each indiv- individual employee's context?

[6:24](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=384s) Or if someone is, like, breezing through all the course content, could you adapt it dynamically to make it more challenging, uh, so they're actually getting more value out of it?

[6:32](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=392s) And then the last one that I particularly like would be, what if you could, uh, uh, dynamically update, uh, course material based on people learning about the customer? So if someone was a visual learner, great, let's make visual content for them and do-- having the AI, having the ML, uh, sorry, the, the large language model just do

[6:50](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=410s) that automatically. And you have to think, does that solve the problem more than summarization or a, or a Q&A chatbot? Um, so really good food for thought. And to sort of talk about some of the customers, uh, that we see achieving really industry-leading results, uh, by combining, uh, their own domain expertise and our, um, our model.

[7:11](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=431s) So I won't read off each, but just a couple of call-outs. One is, uh, AI impacting different industries. We have, uh, taxes, we have legal, we have, uh, project management.

[7:21](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=441s) They're using AI to, uh, drastically enhance their customer experience. They make it more, uh, like, easier to use. They make it more trustworthy. Um, and so it's really improving the experience versus just being like a nice to have.

[7:34](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=454s) And then they're achieving a real h-- a real high quality, um, of output, right? You can't be giving-- you can't be hallucinating when you're doing your taxes. Uh, it could just, you know, that could lead to all sorts of things.

[7:44](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=464s) So we're thrilled that they're seeing these, these sort of like business-critical workflows powered by AI, driving really positive outcomes for them and also their customers.

[7:55](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=475s) Awesome.

[7:56](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=476s) I can do this one. Yeah. So, [chuckles] um, getting started. I just quickly, uh, there's two, two key points here. So if you go on the next slide, um, what are our products?

[8:06](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=486s) We have our API, we have Claude for Work. Our API, uh, is for businesses that wanna embed AI in their product and services, and then Claude for Work empowers your entire organization to take advantage of AI in their day-to-day work.

[8:19](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=499s) We also have, uh, next one. We also have a partnership with AWS and GCP, and you can kinda get the best of both worlds here. You can access our frontier models on Bedrock or on Vertex.

[8:32](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=512s) You can deploy these applications in your existing environment, um, and you, but you-- and so you don't have to manage any new infrastructure. So it really sort of breaks down like any barriers to entry.

[8:43](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=523s) So you're getting the best of both worlds here. We talk a little bit about support throughout this talk. It-- to us, it doesn't matter if you're accessing us through a third party or a first party, so I just wanted to call that out.

[8:56](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=536s) Awesome. So now that we've talked a little bit about some of the customers, how do we actually set customers up for success when working with them at Anthropic? So [clears throat] just to preface on kind of what my team does, as I mentioned, it's at this intersection of product research, uh, customer-facing interaction, and then also just actual research within

[9:13](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=553s) the org. Um, and we support, support the technical aspects of the use cases, so helping to design architectures, evals, tweak Claude prompts to get the best out of our models, et cetera.

[9:23](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=563s) And then we also bring whatever we see back into Anthropic, and we try to build some of the best products we can for our customers. So some examples of projects we've worked on or things that we've published include the Building effective agents, uh, research, uh, paper that, uh, my colleague Barry published.

[9:38](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=578s) He's gonna be speaking tomorrow. And then as well as that, we've launched Model Context Protocol, which is a open source protocol for language models to interact with data sources.

[9:47](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=587s) And, um, Mahesh is gonna be leading a workshop on that, uh, on Saturday, I believe.

[9:54](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=594s) So Anthropic as a whole, we try to effectively support our customers, but where we really start to embed, at least my team in particular, is, um, we work closely with customers that are using Claude a lot, and they're facing really niche challenges in specific use case domains, and they need support from our team to try to apply

[10:10](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=610s) some of the newest, kind of latest and greatest research or get the most out of the models from a prompting standpoint, et cetera. And so this approach is pretty additive.

[10:17](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=617s) We often kick off a pr-- a sprint once the customer is facing those tricky challenges. Uh, and that could be LL- LLMOps, architectures, or evals. We help to define certain metrics that they, they deem to be important when they're evaluating the model against, uh, the use case.

[10:32](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=632s) And then finally, um, we help them deploy that kind of iterative loop, the result of that into an A- A/B test environment, um, and then hopefully into production. And so a part of that is the importance of evals, and I'll get onto that in a second.

[10:45](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=645s) Um, but I'm gonna pass it over first to Joe to talk about some stuff that we did for Intercom.

[10:52](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=652s) Yeah. So sort of the-- I think this is a good segue on what Alex was describing. So for those of you don't-- who don't know, Intercom is an AI customer, uh, service platform.

[11:01](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=661s) They have an AI agent called Fin. By many measures, it's the best in the market, and it's a pretty competitive market. So they had their product, uh, out for, I think, about a year or so.

[11:12](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=672s) And when we spoke to them, they shared wh- where they wanted to go, where they saw the future as-- of like customer support and agents. And based on some of the capabilities of our model, we felt that we could have a pretty good impact on these metrics.

[11:25](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=685s) And so what we started with was the Applied AI, uh, lead met with their data science team, and we ran a quick two-week sprint. We took their hardest prompt for Fin, and we compared it, uh, against a prompt that we helped them sort of figure out with, uh, with Claude.

[11:40](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=700s) And they saw really good results after the first two weeks. So much so we went on this sort of sprint of about two months where we were basically, um, fine-tuning and optimi- optimizing all of their, uh, prompts to get the best performance out of Claude.

[11:55](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=715s) At the end of this, they're able to look at all their benchmarks and see that Anthropic was outperforming the current LLM. It's also worth noting that they do a resolution-based pricing model, so there's an incentive for everyone for the model to be really helpful and help customers solve problems and not be like a deflection machine where it's

[12:11](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=731s) like, you know, we've probably all experienced them before. And so, uh, at the end of this two months, they decided to move forward with Anthropic. They launched it. You can read about it.

[12:19](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=739s) It's called Fin 2. And I think just some of the metrics are really, like, mind-blowing. Like, can solve up to eighty-six percent of customer support volume, fifty-one percent out of the box.

[12:28](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=748s) Our support team

[12:29](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=749s) Thought about lots of different options, and they actually adopted, uh, Fin as well, and they saw very similar resolution rates. But also making it more human, so they-- I think with our model, we can-- there's a much more of a human element to it, so they could do, like, uh, adjustment of the tone, uh, answer length, and

[12:45](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=765s) then it was also really good at doing policy awareness, so like refund policy, for instance. So unlocking some new capabilities. And we're thrilled to be partnering with them as they sort of, I think, march forward as a leader in, in this space.

[12:57](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=777s) Yeah. On, on a kinda separate note, one of the things I've seen recently is, uh, Claude on Twitter acting as some sort of therapist for a lot of people, and I always find that an entertaining example- [laughs] ...of, like, its character being expressed.

[13:08](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=788s) Yeah.

[13:08](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=788s) Um, cool. So let's get onto some best practices and mistakes that we see in the field, uh, on the go-to-market team. So firstly, testing and evaluation. I'm sure this-- those two words have been mentioned a lot today and probably tomorrow too.

[13:22](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=802s) Um, there are some typical common mis- common mistakes that we see, uh, customers struggling with. So the first one is they build a really robust workflow. They've spent, like, a bunch of time building some architecture out, and then they're like, "Okay, now we need to evaluate it.

[13:34](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=814s) Like, let's build some evals." That's not really how it should work in practice because your evals are actually the thing that directs you towards a perfect outcome, right? You can't build a whole workflow without evals probably from the get-go or very shortly after.

[13:48](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=828s) And so, you know, sometimes customers, as a result of struggling with data problems, might not be able to design their evals. You could use Claude to clean that up, do data reconciliation.

[13:58](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=838s) Um, or they're just, you know, trusting the vibes too much. Maybe they run a couple queries, they're like, "Hey, it looks good," right? Are they really testing that on a representative sample though?

[14:07](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=847s) Like, do you have enough s-samples to say that the thing that you're looking at is statistically significant? And like, or are you gonna, you know, run a hundred things when it actually goes into prod and then there's gonna be, like, loads of outliers, uh, because you didn't actually predict correctly what, uh, the customer is gonna ask of

[14:22](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=862s) the model, for example. So I challenge you to think about your use cases as this sort of latent space, right? Let's take this kinda chart here on the left-hand side of the slide, right?

[14:34](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=874s) As you explore the latent space with different functions that you can apply to the model, let's say prompt engineering, prompt caching, stuff like that, you're, you're basically moving your kind of position in that latent space around between l- attractor states, you know.

[14:50](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=890s) And eventually you wanna find an optimized point, but you don't really know where that is, right? Like, if you're changing an instruction, you don't know how the attention mechanism of the transformer is gonna eventually result in some different outcome that might not be performant.

[15:03](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=903s) And so the only way you can truly know that is empirically, and that's through evaluations. And so I think that's why evaluations are so, so important, and a lot of people just don't understand that soon enough.

[15:13](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=913s) In many ways, I actually tell customers, "Evals are your intellectual-- intellectual property." Like, if you wanna be competitive in a space, you need to be able to outcompete people by navigating that la- latent space and finding the attractor sta-state faster than anyone else.

[15:29](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=929s) Um, and so h-- part of, you know, how you do that is, well, firstly, setting up some sort of telemetry, uh, to back test. Ideally, that architecture's set up in advance, but, you know, you should invest in it.

[15:41](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=941s) Um, designing representative test cases. So let's say you're working on that customer support agent eval. You know, you might have a kid come on your website, let's say you're building it for, and might ask some crazy question like, "How do I, you know, kill a zombie in Minecraft?"

[15:56](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=956s) Like, totally unrelated to your product. That's, you know, still probable, and so you should probably include silly examples like that in your eval set to make sure that your model's actually approaching the response in an appropriate way or rerouting the question, et cetera.

[16:10](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=970s) Cool. Moving on to the next one. Um, identifying metrics. So a lot of the time, you know, there's this intelligence, cost, latency triangle of trade-offs that people are trying to move in between.

[16:22](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=982s) And most or-organizations can optimize for one or two of those things, but it's very difficult to meet three, at least right now. But realistically, that balance should be defined in advance, and you should know that for your specific use case, you're going to make a trade-off between those things.

[16:39](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=999s) So let's say a customer support use case again. You care about your customer getting a response within ten seconds, right? If more than ten seconds, I think there's been research done on this, uh, the customer's likely gonna just log off the page, and then they won't get the response, and then they'll probably complain about your product to

[16:53](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1013s) their friends, right? Whereas if you're looking at a financial research analyst agent, you probably don't care that it works for ten minutes to come up with the actual response to your question because the decision being made after that is very important.

[17:05](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1025s) It's an allocation of capital, for example. And so the stakes and time sensity-- sensitivity of the decision should really drive your optimization choices and, you know, maybe more instruction sets lead to longer latency but higher performance, et cetera.

[17:20](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1040s) The other thing is UX could be important, right? So again, on that customer support agent, because we spoke about Intercom, you could have different ways of circumventing that ten second to fifteen seconds.

[17:31](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1051s) Specifically, you could add like a little thinking box that bounces around. You could send the customer to another webpage in the meantime, have them read something, right? Like, there's loads of ways to distract and kind of push on those boundaries, but you still need to know what that important indicator is, and you need to optimize accordingly.

[17:49](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1069s) Finally, fine-tuning. So a lot of people, you know, I go into these calls and they're like, "Oh, we wanna do fine-tuning." I'm like, "Oh, here we go again." [laughs] Um, fine-tuning's not a silver bullet.

[18:00](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1080s) So it comes at a cost, and most people aren't aware of that cost. Um, the cost is generally you're doing brain surgery on the model, and thus there can be kind of limitations to its reasoning in other fields outside of the thing you're fine-tuning towards.

[18:14](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1094s) Um, so my encouragement is try other approaches first, right? Most people, they don't even have their eval set when they're trying to do fine-tuning, right? They need to have a cri-- clear success criteria in advance.

[18:25](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1105s) And it's like, only if we can't get that in our specific intelligence domain, do we then do fine-tuning. Don't try to boil the ocean in advance. The difference in fine-tuned capabilities and the wide variance of which, you know, failure versus success looks like in fine-tuning land means that you should be able to justify the cost of fine-tuning

[18:46](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1126s) and the effort of doing it, right, getting a team, fine-tuning, working with us, for example, uh, you should be able to justify that difference. And so in, in terms of best practices, you, you know, don't want let, to let fine-tuning slow you down, right?

[19:00](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1140s) You don't wanna say, "Oh, I'm only going to convert this language model use case if we can actually finally fine-tune our model." It's like, no, no, pursue it and then realize that you need to do fine-tuning, and then you can just sub in the fine-tuned model.

[19:12](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1152s) And then explore other methods first, and there are loads of different methods that, you know, are Anthropic as well as other companies are working on these days, and I just wanted to, like, flash up a few of them as we wrap things up here.

[19:25](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1165s) And so I'm not gonna go through all of these, but alongside just base prompt engineering, which granted is very important, there are loads of different features or architectures that will change the success of your use case drastically.

[19:38](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1178s) So for example, you might not need to sacrifice on intelligence of your model if in order to speed it up by, like, removing instructions if you can just leverage prompt caching and have a 90% recut- reduction in cost and a 50% increase in speed, right?

[19:53](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1193s) Or contextual retrieval will drastically improve the performance of your retrieval, retrieval mechanisms, and thus you feed the information to the model more effectively, and thus it has less of a time processing all the instruction set that you've given it.

[20:06](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1206s) So there are quite a few things that you can apply here, and some of them are even out of the box, like citations. And then there are also architectural decisions like agentic architectures.

[20:15](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1215s) You know, Barry, my colleague who's speaking tomorrow, will have a lot to say on that. Um, but that pretty much does it. Um, thank you so much for, uh, for your time.

[20:25](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1225s) We'll be in the theater level lounge after this chat for follow-up questions. Um, anything else from you, Joe?

[20:32](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1232s) No. Thank you so much.

[20:34](https://www.youtube.com/watch?v=VhPfM_aGBVc&t=1234s) Cool. Cheers. [outro music]
