AI Engineer World's Fair 2024
What's new from Anthropic and what's next
Read the talk
Beyond the AI button: building around Claude’s capabilities
From electric factories to Artifacts, Alex Albert traces how better models and new interfaces can change the work a product makes possible.
From a talk by Alex Albert
Before you start: Basic familiarity with LLM prompts, API calls, and web application development will help with the examples.
A new motor in the same factory
Picture a factory in 1882: a large coal-fired steam engine at one end, overhead transmission lines running across the building, and hundreds of machines moving to the same rhythm. Alex Albert opens with the arrival of commercial electric power and a puzzle: why didn’t a superior power source immediately produce a more productive factory? His answer lies in the machinery surrounding the engine.
If one station needed power, the entire steam engine had to run. The transmission system also dictated where machines could sit, so the layout served power distribution rather than the manufacturing process or its workers. Replacing the steam engine with an electric motor removed the nearby coal furnace and made lighting easier, but left those constraints intact.
The larger change came when factories were redesigned around electricity. Smaller, specialized tools made the arrangement more flexible; workers could bring tools to an item instead of carrying the item back to a fixed workstation. The productivity opportunity was in redesigning the process around the new capability, not merely changing its power source.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The magic-star-icon phase
An LLM added behind a star-shaped button can preserve a product’s existing assumptions just as faithfully as an electric motor preserved a steam-powered factory’s layout. Albert draws a second comparison with early mobile development: shrinking a website onto a phone preceded applications designed around an always-available camera and GPS. Snapchat and Uber illustrate the latter approach—products whose behavior depended on what the new platform could do.
There are practical reasons AI products remain in this phase. LLMs are nondeterministic, reliability is difficult, and prompts can require repeated optimization. Developers are learning to build with a component that behaves differently from conventional software. In Albert’s account, the text box is still the main interaction pattern that has stuck; finding useful alternatives requires both stronger models and experiments with the product surface itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From benchmark scores to iterative coding
The model enabling the next set of examples is Claude 3.5 Sonnet, released the Thursday before this 2024 talk. It was the first release in the Claude 3.5 family, and Albert describes this middle-sized model as outperforming the previous flagship, Claude 3 Opus. He points to MMLU, HumanEval, GPQA, and tool-use evaluations, then turns from academic tests to application behavior.
Albert highlights a 200K-token context window for retrieval-augmented generation and claims near-perfect recall across it, without specifying the recall test conditions. A large context gives an application room to supply retrieved material, but that claim alone does not establish end-to-end RAG accuracy. For coding, his observation is more behavioral: Sonnet seems to understand debugging problems better and gets caught in repeated, unproductive loops less often than earlier models.
Pull requests provide a more concrete way to test multistep work. They have a defined objective, usually require several changes, and let a model iteratively write code and test its progress. Anthropic’s internal evaluation involved fixing bugs or adding functionality in open-source codebases with coding tools.
Albert reports scores of 64% for Claude 3.5 Sonnet and 38% for Claude 3 Opus on Anthropic’s internal pull-request evaluation. These figures describe that evaluation, rather than a general success rate for arbitrary pull requests; the talk does not provide its sample size, attempt budget, or full scoring protocol.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A table image becomes editable data
Vision extends the same model beyond text and code. Albert reports improvements over Claude 3 Opus on essentially every vision benchmark Anthropic tested. His concrete example is table transcription: supply an image of a table and ask the model to reproduce it in Markdown. He says the result was almost perfect and that he checked the numbers himself.
The Workbench frame shows the uploaded table and transcription request on the left, with the resulting earnings-history table on the right. This turns visual source material into a representation that can be copied, edited, or incorporated into another document—the kind of practical use that made vision particularly compelling to Albert.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Capabilities need affordable access
The launch offered three access paths: Anthropic’s API, AWS Bedrock, and Vertex AI. That choice let developers use Claude through the provider that fit their environment. The prices and availability here describe the launch-era offering, while the later model and steering announcements describe forecasts made during the talk.
Claude 3.5 Sonnet’s launch pricing was $3 per million input tokens and $15 per million output tokens. Albert describes it as five times cheaper than Claude 3 Opus. His application-level point is that intelligence, speed, and cost must work together: a model becomes easier to embed in a product when its useful capabilities are also economical to invoke.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Artifacts give the work its own surface
Artifacts separate generated content from the conversation used to create it. An essay, SVG, or React website can occupy its own work surface while the chat remains the place to request changes. This is the first product example that directly answers the factory analogy: the interface changes to accommodate what the model produces.
The experience combines Sonnet’s coding, reasoning, and vision abilities. A screenshot or Figma diagram supplies a visual target; Claude produces code and components that a developer can use. Albert demonstrates the combination by recreating the claude.ai chat layout in React from a single screenshot. The displayed result places the recreated interface beside the conversation that produced it.
Albert describes the idea as having hidden in plain sight for more than a year and a half. A tweet shown afterward frames LLM product development as early in its S-curve. The encouraging implication is that a useful interaction pattern can still emerge from capabilities developers have already been experimenting with.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Projects make context a shared resource
Projects, released the Tuesday before the talk, address another part of the workflow: grounding outputs in knowledge a team already has. Style guides, codebases, transcripts, and past work become context for collaboration with Claude. On the Claude Team plan, teammates can share projects and chats.
Albert reports that Anthropic engineers upload repositories and documentation, and that he has begun seeing people share chats and Artifacts in place of some Google Docs or site documentation. The Project view shows a chat input alongside a populated Project knowledge sidebar. The shared object is now more than a finished document: it includes the knowledge and conversation involved in creating the work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tools connect model output to application behavior
The developer-facing counterpart is the Tool Use API, which Albert says had launched about a month earlier. It lets an application supply custom client-side functions for Claude to select. Tool use also provides a way to request consistent structured JSON, so the model’s response can fit an application’s data contract rather than arrive only as prose.
For example, a tool definition for extracting a task from text can describe the expected fields explicitly:
json
{
"name": "record_task",
"description": "Extract a task and its owner from the supplied text.",
"input_schema": {
"type": "object",
"properties": {
"task": {
"type": "string",
"description": "The work to be completed."
},
"owner": {
"type": "string",
"description": "The named owner, or an empty string if none is given."
}
},
"required": ["task", "owner"]
}
}
The schema makes the desired structure explicit; a tool request still needs application code to handle it. Albert also reports seeing developers give Sonnet hundreds of tools at once, an observation about emerging usage rather than a measured reliability result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A shorter path from task description to prompt testing
The developer console reduces the distance between describing a task and trying a prompt:
- Enter a task description into the prompt generator. Claude produces a prompt intended to suit that task.
- Open the generated prompt in the Workbench and edit it directly.
- Use variables to turn the prompt into a template, substituting inputs for cases such as RAG testing.
This gives developers a starting point and a place to iterate, rather than requiring every prompt to begin as a blank text field.
Albert then previews the console’s Evaluate feature, already visible with a beta tag. He promises further details and iteration; the demonstration at this point establishes the direction toward evaluation without walking through a complete testing or grading workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Plan around a moving capability frontier
The roadmap preview names Claude 3.5 Haiku and Claude 3.5 Opus as models expected later that year. Albert describes the goals for each generation as greater intelligence, lower latency, and lower cost. His advice to developers follows from that pace: plan ambitiously enough that a new model arriving during development can expand what the product does. He expects meaningful improvement on a timescale of months rather than years.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Controlling features as well as prompts
A separate research direction asks whether developers can influence model behavior by manipulating internal features. Albert introduces Scaling Monosemanticity, covered in Anthropic’s research account. The research concerns Claude 3 Sonnet, distinct from the newly released 3.5 model. It identifies features that activate for different topics; clamping a feature’s value higher or lower can steer the output.
Golden Gate Claude made that mechanism visible to users by amplifying the Golden Gate Bridge feature. Instead of relying only on instructions in a prompt, the demonstration altered an internal feature’s influence. Albert then describes a few beta testers experimenting with a steering API that lets them find and clamp features for specific attributes. Broader developer access is something he hopes to offer, not a general release announced here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a prototype and start the feedback loop
The immediate next step is a prototype: make something quickly enough to begin validating it and learning from feedback. To encourage that work, Albert announces a Build with Claude contest launched the previous day. Albert says the contest runs until July 10, with each of the top three projects receiving $10,000 in Anthropic API credits. He directs attendees to the on-screen destination and the top of the documentation page for the contest details.
For questions beyond the session, Albert offers to talk at the AWS booth down the hall over the next few hours. He also gives his X/Twitter handle, @alexalbert__, explicitly noting the two underscores, and says he tries to read all his direct messages. The invitation leaves the next product experiment with the developers in the room.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Anthropic's historical announcement covering Sonnet's evaluations, launch pricing, availability and Artifacts preview.
How Projects introduced reusable knowledge, custom instructions and shared team conversations.
The historical tool-use announcement explains external-tool integration and structured data extraction.
Anthropic's account of the demonstration that amplified a Golden Gate Bridge feature.
Further reading
An accessible introduction to the feature-extraction and steering research behind Scaling Monosemanticity.
Updates since the talk
The July 2024 announcement explains test suites, generated inputs, output comparisons and human grading in the Console.
Read the complete timestamped transcript
- 0:00
[upbeat music] Morning, everyone. Good morning.
- 0:17
So today, I want to start with a, uh, a little story. Um, a short history lesson, if you will. So, you know, sit back, get comfortable. Uh, I'm gonna take us back to the year 1882.
- 0:31
It's the dawn of the electrical revolution, really. The world's first commercial power plant just opened up.
- 0:38
Electricity, this amazing new force, is all the rage in the manufacturing industry.
- 0:44
People are claiming that it's gonna change everything.
- 0:47
And yet, something very interesting happened around this time,
- 0:52
or rather, it didn't happen. You see, despite electricity's obvious superiority in comparison to the traditional techniques at the time, like steam engines, it didn't immediately improve manufacturing productivity.
- 1:06
Why? Well, because factory owners were simply trying to replace their old technology with this new technology into an outdated paradigm.
- 1:17
Let's picture a typical factory at the time. So we have a huge coal-fired steam engine on one end, and we have a network of transmission lines running across the top, all driving hundreds of machines locked in the same rhythm.
- 1:29
These legacy steam-powered factories were incredibly inefficient. You know, if one station needed power, all of a sudden you had to turn on that entire steam engine and it had to power all of them.
- 1:39
Factory layouts were dictated by the limitations of the transmission lines, not by what was best for the process or for the workers.
- 1:48
When electricity arrived, many factory owners simply swapped out the steam engine for an electric one. And sure, you know, they added some lights and, you know, workers didn't have to toil next to a coal-fired furnace all day, but the fundamental limitations of the factory remained.
- 2:05
So this real electrical revolution, well, it didn't actually come until we imagined factories from the ground up with electricity at its core.
- 2:16
Factories started to become flexible and adaptable. They allowed for smaller, specialized tools. Workers could bring their tools to the items instead of having to lug the items back to their workstations.
- 2:29
The entire manufacturing process became more efficient, more humane, and more productive.
- 2:35
Now let's fast-forward 140-something years to today, and you can see that we find ourselves at a similar point in regards to AI and LLMs. Enterprises, startups, developers are all building and integrating LLMs into their products, but often they're just tacking it onto their existing product surface, adding a few star icon buttons in the top left corner and
- 2:56
calling it a day. And this is not the first time we've seen this in Silicon Valley. [clears throat] Let's think back to when mobile first emerged, right? Companies simply tried to just shrink down their website and put it on a phone.
- 3:11
It wasn't until we redesigned apps from the ground up with the unique capabilities of mobile, like always-on camera and GPS, that we actually began to see true innovation in the space and adoption.
- 3:24
This is when the Snapchats and the Ubers of the world started to emerge.
- 3:29
So just as companies-- And s- just as factories went through their, you know, replace steam engines with electric one phase, and tech companies went through their just hire a couple mobile web dev people phase, we're now in our magic star icon phase with respect to AI.
- 3:46
And, uh, yeah, it's funny, but the thing is you can't blame any of the companies or developers that are actually trying to do this right now, right? Like, all of us are trying to do this, but in many ways we're just still so early.
- 3:58
LLMs are non-deterministic. They're hard to build on. They're completely different than what most developers are used to using.
- 4:05
Reliability is still an issue. Prompts still take rounds and rounds of optimization.
- 4:12
And we've also just started to scratch the surface of potential product opportunities. So far, not much has really stuck beyond just the text box.
- 4:21
We've been missing something, something that's a little hard to put a finger on. But just last week, I think we scratched the surface of a potential new product feature that we can build.
- 4:33
As some of you may have heard, last Thursday we released our new model, Claude 3.5 Sonnet.
- 4:39
3.5 Sonnet is the first model that we released in the new Claude 3.5 family. It's only the middle model, and yet it is better than our last best model, Claude 3 Opus.
- 4:51
In my opinion, Claude 3.5 Sonnet is one of the best models in the world right now, and the benchmarks seem to back it up.
- 4:59
MMLU, HumanEval, GPQA, tool use, all the common characters here. It's top of its class in many regards in these academic lab-type environments, but what I am most excited for is how it actually does in the real world.
- 5:14
The model is particularly strong in RAG use cases, thanks to its 200K context. It also has near-perfect recall over that entire context as well. On coding tasks, 3.5 Sonnet seems to grasp debugging problems better.
- 5:28
It's not getting stuck in those same loops as much as previous models.
- 5:33
One of the best methods that we've found for actually measuring more complicated chains of reasoning is pull requests. They have a defined task, they usually take a few steps to solve, and the model's able to iteratively write and test its way to a solution.
- 5:48
In our own internal pull request evals, we're seeing that Claude 3.5 Sonnet scores a sixty-four percent. And to put that number in comparison, Claude 3 Opus only scored a thirty-eight percent.
- 6:00
3.5 Sonnet also has state-of-the-art vision abilities. It shows considerable improvement over 3 Opus in basically every benchmark that we tested it on.
- 6:10
Things like table transcriptions and OCR are a breeze now. Passed this table in a s- uh, 3.5 Sonnet and basically replicated it perfectly in markdown. Um, probably can't read all those numbers, but trust me, I, I double-checked them to make sure they're all right.
- 6:25
Uh, vision capabilities were actually what amazed me the most when I started playing around with this model. It feels like we are really on the cutting edge of unlocking so many more use cases.
- 6:36
And, you know, as you're hearing me say all this, you might be thinking, "Well, that's great, Alex, but, I mean, it doesn't mean anything if I can't actually use the model."
- 6:42
And you're right, and we heard you, and that's why 3.5 Sonnet is available on our API, AWS Bedrock, and Vertex AI.
- 6:51
We understand that developers want choice when they're building, and we want Claude to be available wherever you are.
- 6:59
In terms of pricing, 3.5 Sonnet is five times cheaper than 3 Opus. It's only $3 per million input tokens and $15 per million output tokens.
- 7:10
3.5 Sonnet's combo of speed, intelligence, and low cost makes it much more economical to use and embed in your apps than 3 Opus.
- 7:20
But 3.5 Sonnet is not all that we've released in the past week. We also released a new product feature that I think is actually more inspiring to developers in terms of thinking about and building those AI products from the ground up.
- 7:32
It's called Artifacts. Artifacts separate the content that Claude produces from the actual chat dialogue itself.
- 7:40
This allows you to work collaboratively with Claude on things from essays to SVGs to React websites.
- 7:47
Artifacts become really powerful when you combine it with 3.5 Sonnet. Those coding skills plus that reasoning ability plus that strong visual acuity enables a new product experience that's really fun to use.
- 8:00
It's also a developer's best friend in that it allows you to quickly take screenshots and Figma diagrams and quickly turn it into code and components that you can actually just go use.
- 8:09
Uh, as you can see in this, I basically cloned our entire claude.ai chat layout in React just from a single screenshot.
- 8:19
And this feature has practically been hiding in plain sight now, uh, just waiting really to be discovered for over a year and a half. Maybe this tweet is right, and we really are early on this S-curve of production- productionizing LLMs, which I think is actually pretty inspiring.
- 8:37
And Artifacts is not the only AI feature that we launched recently. On Tuesday, we released Projects.
- 8:43
Projects enables dev teams to work and collaborate much more efficiently by grounding Claude's outputs in your own knowledge, whether it's style guides or code bases or transcripts or even your past work.
- 8:56
On our Claude team plan, you can even share these projects in your chats with all your teammates.
- 9:03
At Anthropic, our engineers now upload code repos and documentation that they use, and I've started to see people actually just share the chats and the artifacts instead of Google Docs or site documentation.
- 9:15
Projects is another great example of when you think from an LLM and an AI standpoint first, you can actually start to build product experiences that complement these technologies and don't feel like a simple add-on to what you already have.
- 9:29
So now that hopefully the creative product uses are flowing in everyone's minds, I want to dive a little bit into API improvements that we've rolled out recently and things that allow you to actually build this cool stuff.
- 9:41
I also wanna give a preview of what's coming next that will enable you to build even more.
- 9:47
So a month ago, we released our new Tool Use API. Tool Use allows you to give Claude custom client-side functions that it can then intelligently leverage.
- 9:57
Tool Use also enables things like consistent structured JSON output. With 3.5 Sonnet, I've actually started to see devs give Claude hundreds of tools at a time.
- 10:08
On the developer console front, we're also continuing to iterate. We added a prompt generator that uses Claude to write prompts for you based on a task description. So you can see in this video, we put in a task description, and then out comes a optimized prompt.
- 10:24
And then once that prompt is all done, you can actually just start editing it right in the workbench itself. You can see we've also added support for variables so you can edit prompt templates as well, test things like RAG use cases.
- 10:39
And finally, we're also working on a new evaluate feature, which is currently in console right now with a beta tag, uh, and we will plan to share more on this and continue i- to iterate on it very soon.
- 10:52
So what else is next? Um, well, there's, there's two things that I can share right now. First is that you can expect more models.
- 11:00
3.5 Haiku and 3.5 Opus are coming later this year. With each model generation, we're looking to increase the intelligence, decrease the latency, and decrease the cost. The number one thing that I tell developers is to not forget to build with that in mind.
- 11:17
Models will become smarter, cheaper, and faster in orders of months, not years. When you're planning your product roadmap, be ambitious enough to build with the belief that new models may arrive during your development period.
- 11:31
We are also working on other areas of research like interpretability.
- 11:35
In one of our latest papers called Scaling Monosemanticity, we explained how we've been able to find features within models that activate for different topics. Once you identify a feature, you're able to clamp its value and turn it up or down to actually steer the model's outputs.
- 11:51
A few weeks ago, we showed claude.ai users how this worked through Golden Gate Claude, which was a version of Claude that had the Golden Gate Bridge feature turned up significantly.
- 12:03
Whoo! [laughs] Yeah, fan favorite. We currently have a few beta testers also experimenting with a steering API. Um, this allows developers to find and clamp features for sp- specific attributes and actually turn that dial up or down, which again allows you to control Claude's outputs in a, in addition to actually just prompting it.
- 12:23
We hope to be able to roll this out to more developers in the very near future as well.
- 12:28
Now, if anything in this talk has sparked any ideas, I wanna encourage you guys to just go out there and build and make quick prototypes as fast as you can to get that validation and that feedback loop started.
- 12:39
And for even more of an incentive, uh, we actually just launched another Build with Claude contest yesterday. Whoo. It runs until July 10th. The top three projects will each receive 10K in Anthropic API credits.
- 12:50
To see more details, just visit that link below. It's just at the top of our docs page as well, so you can find it there too. I'll leave that up for a second.
- 13:02
And finally, if you have any questions or you wanna hear more about just what we're thinking about, uh, I'll be at that AWS booth down the hall for the next few hours.
- 13:09
You can also find me on X/Twitter [REDACTED:username] with two underscores. Um, I do try to read all my DMs. I spend way too much time on that site. So feel free to, uh, ask questions there as well.
- 13:22
And with that, I wanna say thank you guys very much and enjoy the, uh, last day of the summit. [upbeat music]