AI Engineer World's Fair 2025
Designing AI-Intensive Applications
Read the talk
Designing AI-Intensive Applications
From conference feedback loops to AI News’s daily pipeline, reusable application patterns offer a more useful design vocabulary than arguing over what counts as an agent.
From a talk by Shawn Wang (swyx)
Before you start: Basic familiarity with language models and application workflows is helpful; the TypeScript example uses asynchronous functions and promises.
A conference that tracks its field
How do you organize a conference for a field whose interests keep changing—and whose attendees keep registering late? Opening the June 2025 AI Engineer conference, Shawn Wang reports 3,000 attendees and jokes about measuring last-minute registration through a Gini coefficient of organizer stress. The comparison with the previous year is a joke about timing, not a supplied numerical coefficient. It introduces a practical problem: responding to demand while building an event that reflects where AI engineering is going.
Wang says the conference doubled every previous year’s track. That expansion buys breadth at the cost of more simultaneous choices and attendee decision fatigue. The ambition is to cover the field, even when nobody can attend everything.
Programming also has a feedback loop. The organizers aim to respond faster than research conferences such as NeurIPS while remaining more technical than general-interest events such as TED. They survey attendees about topics—including computer-using agents and AI with crypto—and use those responses to shape the program. At the opening, the survey is still running: wider participation should produce a more representative sample, with results promised the following day and responses informing the next year’s event.
The conference itself is an application surface. Wang describes it as the first conference with an MCP interface and the first to accept an MCP talk through MCP. He credits Sam Julien from Writer for the official chatbot, Quinn and John from Daily for the official voice bot, and Elizabeth Trykin from Vapi for helping prototype the voice experience. These are attempts to improve the attendee experience by putting the field’s tools to work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A growing field does not require elaborate scaffolds
The changing conference structure follows a changing job. In 2023, Wang organized his opening talk around three types of AI engineer. In 2024, the emphasis shifted to multidisciplinary work, reflected in the World’s Fair’s multiple tracks. Early 2025 in New York brought a focus on agent engineering. By this June gathering, the once-dismissive label “GPT wrapper” sits awkwardly beside the commercial success of applications built around models. Wang’s joke that everyone is now rich captures that reversal in status, rather than establishing a financial result.
Increasing capability does not imply increasing scaffold complexity. Wang points to Erik Schluntz’s Latent Space discussion of Anthropic’s SWE-bench success with a simple scaffold. The point is a strong benchmark result from a relatively simple system, not that the benchmark was completely solved. Greg Brockman’s discussion of Deep Research and Amp Code supply related examples. The field is still young enough that useful discoveries need not depend on elaborate infrastructure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What would a standard model of AI engineering explain?
Wang compares this formative moment to the 1927 Solvay Conference, where Einstein, Marie Curie and other major figures in physics gathered. The analogy is about bringing a developing field’s practitioners together while its basic ideas are taking shape. He then points to the physics standard model: in his telling, a concentrated period of development from the 1940s through the 1970s produced a framework with decades of durability. The corresponding engineering opportunity is to identify ideas that will remain useful after today’s products and terminology change.
Software already has this kind of shared vocabulary. ETL describes moving and transforming data; MVC separates interface concerns; CRUD names common record operations; MapReduce organizes distributed computation. These patterns help engineers recognize a problem and discuss a solution without starting from scratch. AI engineering has RAG, but one pattern cannot describe the whole application. Wang treats recurring claims that long context or fine-tuning will kill RAG skeptically. The useful question is which additional models can guide design.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Architecture and lifecycle describe different concerns
Several candidates address different parts of the problem. Wang names DSPy and points attendees toward Omar’s talk, then starts with LLM OS, which he attributes to Andrej Karpathy in 2023. His 2025 update incorporates multimodality, an emerging standard tool set and MCP, which he characterizes as the default protocol for connecting to the outside world. This is an architectural lens: it places the model alongside the capabilities and connections an application needs.
The LLM software development lifecycle offers another lens. Wang mentions two versions, including one organized around the intersecting concerns of purchased tooling, and directs readers to the Latent Space blog. The consequential distinction comes from a conversation with Ankur of Braintrust: getting started is becoming increasingly commoditized, while making an application dependable for customers remains substantial engineering work.
| Lifecycle concern | Practical distinction |
|---|---|
| LLM access, monitoring and RAG | Free tiers lower the cost of getting started |
| Evaluations, security and orchestration | Hard engineering becomes central when serving customers |
“Free” here means access through free tiers, with payment beyond those limits—not zero operating cost. Wang associates the latter concerns with moving from demos into production and explains that the conference added tracks to support that transition.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Agent definitions leave some useful applications unexplained
Anthropic’s Building effective agents supplies another candidate model. Co-author Barry had presented it at the previous conference, and Wang describes it as received guidance for building agents. Its architectural distinction separates predefined workflows from processes in which the model directs execution and tool use. OpenAI uses a different definition, illustrating that the vocabulary is still evolving.
Wang also mentions a recent Agents SDK improvement from Dominik, described as building on OpenAI’s swarm concept. The linked OpenAI Agents SDK repository documents the current project; its present API should not be read as a snapshot of that June 2025 announcement. Wang’s own earlier Agent Engineering framing takes a descriptive approach, organizing the words people use around intent, control flow, memory, planning and tool use. But that vocabulary has a practical limit: he does not use all of it to build AI News.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Human input versus valuable AI output
Wang reports that AI News had more than 70,000 readers at the time of the talk. It began as a tool he wanted for himself. At the previous conference, Soumith, introduced as PyTorch’s lead, told him he enjoyed reading it but did not consider it an agent. Wang initially resisted, then accepted the distinction: it is a workflow. Its usefulness survives that change in label.
Compare the human input required with the valuable AI output produced. This shifts attention from whether an application deserves an agent label to what the user gets from using it. Copilot’s interaction is tightly coupled to typing: a debounced trigger produces an autocomplete after a few characters. ChatGPT moves to a query-and-response interaction. Reasoning models suggest more work from a single request; Wang illustrates this with a one-to-ten input/output ratio, a conceptual comparison without a defined measurement unit or benchmark.
Deep Research and NotebookLM extend the idea toward larger outputs from an initial request. Wang points to Raiza Martin’s product-management talk for the NotebookLM story. Ambient agents stretch the comparison further: can a system produce useful output without a new human request? His zero-to-one framing describes that interaction pattern, rather than the absence of setup, permissions or operating costs. The completed slide’s ratios are a mental model for these interactions, not measured productivity multipliers.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The repeated workflow behind AI News
AI News makes the alternative concrete. Wang describes it as scripts in a trench coat and recognizes that he has written essentially the same system three times: once for Discord, once for Reddit and once for Twitter. Each source repeats the same sequence:
- Scrape the source material.
- Plan the processing work.
- Recursively summarize the material.
- Format the result.
- Evaluate the output.
Recursive summarization is the crucial many-to-one operation: intermediate summaries can themselves become inputs to further summarization, bringing a large collection toward a consolidated result.
The workflow runs daily, with incremental improvements alongside Wang’s work organizing the conference. Repetition across sources suggests a reusable application pattern. An AI-intensive application can coordinate thousands of AI calls toward a single purpose; that is the scale of the generalized design he proposes, not a measured daily call count for AI News.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
SPADE: from many inputs to one delivered result
Generalizing that repeated workflow produces SPADE. Sync brings material into the system; planning organizes parallel processing; analysis reduces many intermediate results into one; delivery puts the result in the user’s hands; evaluation assesses it. The pattern separates collecting information, doing work over it and presenting the result, without requiring the model to choose the entire control flow.
A TypeScript orchestration function can express those stage boundaries directly. Here, the stage implementations are supplied by the application, while the function preserves the sequence and parallel processing:
typescript
type SpadeStages<Input, Job, PartialResult, Result, Output, Evaluation> = {
sync(): Promise<Input[]>;
plan(inputs: Input[]): Promise<Job[]>;
process(job: Job): Promise<PartialResult>;
analyze(parts: PartialResult[]): Promise<Result>;
deliver(result: Result): Promise<Output>;
evaluate(output: Output): Promise<Evaluation>;
};
async function runSpade<Input, Job, PartialResult, Result, Output, Evaluation>(
stages: SpadeStages<Input, Job, PartialResult, Result, Output, Evaluation>
): Promise<{ output: Output; evaluation: Evaluation }> {
const inputs = await stages.sync();
const jobs = await stages.plan(inputs);
const parts = await Promise.all(
jobs.map((job) => stages.process(job))
);
const result = await stages.analyze(parts);
const output = await stages.deliver(result);
const evaluation = await stages.evaluate(output);
return { output, evaluation };
}
For AI News, the inputs come from the scraped sources, the processing includes summarization, and delivery formats the consolidated result. Promise.all makes the parallel stage visible; at the thousands-of-calls scale, an implementation also needs to bound concurrency to its service limits.
The stages admit richer representations than a text-in, text-out pipeline. Material can become a knowledge graph, and model results can become structured outputs. Delivery can produce generated code rather than prose: ChatGPT Canvas and Claude Artifacts illustrate that possibility. The completed slide connects the stages to planning and read tool calls, summarization and write tool calls, code generation and generative UI, and evaluation with an LLM judge. These are possible techniques within the pattern, not requirements that every application adopt them all.
SPADE remains a developing mental model, with the detailed implementation deferred. Its value is in making a recurring shape recognizable: gather many inputs, coordinate work over them, consolidate the results and deliver something useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Adding intelligence people want to use
The test for any of these models is whether it helps an application take its next useful step. Adding intelligence should help the user without becoming annoying. LLM OS, lifecycle models, agent frameworks and SPADE are hypotheses about how to do that, not settled standards.
Wang’s closing invitation is to use the conference’s conversations to discover patterns that other engineers can reuse. A shared model earns its place when it helps people improve their applications—and, echoing Laurie’s opening point, build products people want to use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Podcast and transcript explaining Anthropic's coding-agent scaffold, tool design and SWE-bench results.
Anthropic's guide to workflows, autonomous agents and simple composable application patterns. The live page includes later tooling updates.
Browse the published output of the AI news aggregation project used as the talk's central application example.
Documentation for programming and optimizing language-model applications with DSPy.
Further reading
Technical account of a 49% SWE-bench Verified result, with the prompt, tool specifications and evaluation limitations.
- Agent EngineeringArticle
Companion to swyx's earlier keynote, organizing agent definitions around intent, control flow, tools, planning, memory and authority.
Updates since the talk
Current SDK source and examples for agents, tools, handoffs, guardrails and tracing.
Read the complete timestamped transcript
- 0:00
[on hold music] Okay.
- 0:18
Hi, everyone. Welcome to the conference. How you doing? [clapping]
- 0:22
Excellent. Usually, I s- open these conferences with a small little talk to introduce, uh, you know, what's going on, and then give a little update on where the state of AI engineering is, and how we put together the conference for you.
- 0:34
Uh, this is a c- this is one of those combined talks. I- I'm trying to answer every single question you have about the conference, about AI News, about where this is all going, and we'll just dive right in.
- 0:47
Okay, so, um, 3,000 of you, all of you registered last minute. Uh, thank you for that stress. [laughing] Um, I actually can quantify this. I call this the Gini coefficient for, uh, the AI o- uh, AIE organizer stress.
- 1:01
Um, this is compared to last year. Uh, it is... Uh, please just buy tickets earlier, like, I mean, you know you're gonna come. Just, just do it, okay? Um,
- 1:09
we also, uh, like to use this conference as a way to track the evolution of AI engineering. Uh, that's-- Those are the tracks from last year. We've just doubled every single track for you.
- 1:19
Um, so basically, it's basically, you know, like double the value for whatever you, uh, get here. And I think like, uh, you know, I think this is a m- as much concurrency as we wanna do.
- 1:29
Like I know, I, I hear that people have decision fatigue and all that, uh, totally, but also we try to cover all of AI, so deal with it.
- 1:38
Um, we also pride ourselves in doing well by being more responsive than other conferences like NeurIPS, and being more technical than other conferences, uh, like TED or whatever, what have you.
- 1:49
So we asked you what you wanted to hear about. These are the surveys. Uh, we tried all sorts of things. We tried computer-using agents, we tried AI and crypto. [chuckles]
- 1:58
It's always a fun one. And, uh, but you guys told, told us what you wanted, and we put it in there. Um, for all-- for more data, um, we would actually like you to, to finish out our survey.
- 2:07
The survey's not done. So if you wanna head to that URL, um, we will present the results in full tomorrow. We would love all of you to, to fill it out so we can get a representative sample of what you want, and, uh, that'll inform us next year.
- 2:20
Okay. Um, you know, I think the other thing about AI engineering is that we also have been innovating as engineers, right? We-- We're the first conference to have an MCP.
- 2:28
We're the first conference to have an MCP talk accepted by MCP. [chuckles]
- 2:33
We're... [clears throat] Shout out to Sam Julian from Writer for working with us on the official chatbot, and Quinn and John from Daily for working with us on the official voice bot, as well as Elizabeth Trykin from, uh, Vapi.
- 2:45
I need to give her a shout-out because she originally, uh, helped us, uh, prototype, uh, the, the, the voice bot as well. So we're trying to constantly improve the experience.
- 2:54
Uh, the other thing I think I want to emphasize as well is, like, these are the talks that I give. Like, in 2023, uh, the, the very first AIE, I talked about the, uh, the three types of AI engineer.
- 3:06
In 2024, I talked about, um, how AI engineering was becoming more multidisciplinary, and that's why we started the World's Fair with, with, uh, multiple tracks. In 2025, in, in New York, we talked about the evolution and the focus on agent engineering.
- 3:20
So where, where are we now in sort of June of 2025? Um, that's where we're gonna focus on. I think we- we've come a long way regardless. Like we, you know, we...
- 3:28
People used to make fun of AI engineering, and, and I anticipated this. We used to be low status. People just deride GPT wrappers. And look at all the GPT wrappers now.
- 3:36
All of you are rich. Um, so we're gonna hear from some of these folks, uh, in the room. Um, and, uh, thank you for sponsoring as well. [chuckles]
- 3:46
Um, but, uh, you know, I think the other thing that's also super interesting is that, like, you should, we-- The, the consistent lesson that we hear is to not overcomplicate things.
- 3:54
From Anthropic on the Latent Space Podcast, uh, we hear, we hear, we hear from, uh, Eric Schutt about how they beat, uh, SWE-bench with just a very simple scaffold.
- 4:03
Uh, same about Deep Research from Greg Brockman, who you're gonna hear later on, um, in the, uh, sort of closing keynotes, as well as Amp Code. Where's the Amp folks here?
- 4:12
Amp? Amp? Amp? Uh, I think they're probably back in the other room. [chuckles] But, um, also, you know, there's, there's a sort of emperor has no clothes. Like there's-- It's still a very early field, and I think the, um, AI engineers in the room, like, should be very encouraged by that.
- 4:23
Like there's, there's still a lot of alpha to mine.
- 4:27
Um, if you watch back all the way to the start of this conference, we actually compare this moment a lot to, uh, the time when sort of physics was in, was in full bloom, right?
- 4:35
This is the Solvay Conference in 1927 when Einstein, Marie Curie, and all the other household names in physics were all gathered together, and that's what we're trying to do for this conference.
- 4:43
We've gathered the entire, the best, um, sort of AI engineers in the, in the world, um, and, and researchers and, and, and all that, uh, to, to build and push the frontier forward.
- 4:53
Um, the thesis is that there's, this is the time, this is the right time to do it. I said that two and a half years ago. Still true, still true today.
- 5:01
But I think, like, there's a very specific time when, like, ba- basically what people did in, in that time of the formation of an industry is that they set out all the basic ideas that then lasted for the rest of that industry.
- 5:12
So this is the standard model in physics, and there was a very specific period in time from like the '40s to the '70s where they figured it all out, and the, the next 50 years, we haven't really changed the standard model.
- 5:22
So the question that I want to phrase here is, what is the standard model in AI engineering, right? We have standard models in the rest of engineering, right? Everyone knows ETL.
- 5:31
Everyone knows MVC. Everyone knows CRUD. Everyone knows MapReduce. And I've used those things in like building AI applications, and like it's pretty much like, yes, RAG is there, but I heard RAG is dead.
- 5:43
I, I don't know. You guys can tell me. Um, this day it's like long, long context filled RAG. The other day it's fine-tuning kills RAG. I don't know. But I, I don't think, I definitely don't think it's the full answer.
- 5:53
So what other standard models might emerge to help us guide our thinking? And that's really what I wanna push you guys to.
- 6:00
So, uh, there are a few candidates, standard models in AI engineering. I'll pick out a few of these. Uh, I, I don't have time to talk about all of them, but definitely listen to the DSPy talk from Omar later, uh, tomorrow.
- 6:10
Um, so we're gonna cover, uh, a, a, a few of these. So first is the LMOs. Uh, this is one of the earliest standard, standard models, um, basically, uh, from Karpathy in 2023.
- 6:20
Um, I have updated it for 2025, um, for multimodality, for the standard set of tools that have come out, um, as well as, um, uh, MCP, which, uh, is, is, has become the default protocol for, uh, connecting with the outside world.
- 6:34
Um, second one would be the LLM SDLC, software development lifecycle. Um, I have two versions of this, one with the intersecting concerns of all the tooling that you buy.
- 6:44
Uh, by the way, this is all on the Latent Space blog if you want, and I'll tweet out the slides. So, uh, you- and it's live streamed, so whatever. [laughs]
- 6:51
Um, but I think, uh, for me, the most interesting insight and the aha moment when I was talking to Ankur of Braintrust, who's gonna be keynoting tomorrow, um, is that, you know, the early parts of the SDLC is in- are increasingly commodity, right?
- 7:05
LLM's kind of free, you know, um, monitoring kind of free, and RAG kind of free. Obviously, there's- it's just free tier for all of them, and you only, you only get start paying.
- 7:14
But, like, when you start to make real money from your customers is when you start to do evals, and you start to add in security orchestration, then do real work, uh, that is real hard engineering work.
- 7:24
Um, and I think that's- those are the tracks that we've added this year, um, and I'm very proud to, you know, I guess, push AI engineering along from demos into production, which is what everyone always wants.
- 7:34
Another form of standard model is building effective agents. Uh, our last conference, we had, uh, Barry, one of the co-authors of Building Effective Agents from Anthropic, give an extremely popular talk about this.
- 7:44
Um, I think that this is now at least the, the received wisdom for how to build an agent, and I think, like, that's, like, that is one definition. OpenAI has a different definition, and I think we're, we're just sort of continuing to iterate.
- 7:56
I think Dominic yesterday, uh, released another improvement on the Agents SDK, which builds upon the swarm concept that OpenAI is pushing. Um, um, w- the way that I approach sort of the agent standard model has been very different, so you can refer to my talk from the previous conference on that.
- 8:12
Um, basically trying to do a descriptive, uh, top-down, uh, model of what people use, the words people use to describe agents, like intent, um, you know, control flow, um, uh, memory, planning, uh, and tool use.
- 8:28
So there's all these like, there's all these, like, really, really interesting things. But I think that the thing that really got me, um, is, like, I don't actually use all of that to build AI News.
- 8:36
Um, by the way, who here reads AI News? I don't know if there's, like, a... Yeah? Oh my God, like, it's half of you. [laughs] Thanks. Uh, uh, it's, it's a really good tool I built for myself and, you know, hopefully, uh, now over 70,000 people are, are reading along as well.
- 8:49
Um, and the thing that really got me was Soumith [laughs]
- 8:52
at the last conference. Uh, you know, he's the lead of PyTorch, and he says he reads AI News, he loves it, but it is not an agent. And I was like, "What do you mean it's not an agent?
- 8:59
I call it an agent, and sh- you should call it an agent." Um, but he's right. Um, it's actually, uh, uh, it's actually... I'm gonna talk a little bit about that.
- 9:07
But, like, like, why does it still deliver value even though it's, like, a workflow and, like, you know, is that still interesting to people, right? Like, why do we not brand every single track here, voice agents, uh, you know, like, uh, uh, like workflow agents, computer-use agents, like, why is every single track in this conference not an
- 9:24
agent? Well, I think basically we want to deliver value instead of argue about terminology. So the assertion that I have is that it's really about human input versus valuable, um, AI output.
- 9:37
And you can sort of make a mental model of this and track the ratio of this, and that's more interesting than arguing about definitions of workflow versus agents. So for example, in the Copilot era, you had, uh, sort of like a debounce input of, like, every few characters that you type, then maybe it would do an autocomplete.
- 9:52
Um, in ChatGPT, every few queries that you type, it would maybe output a re- responding query. Um, and it starts to get more interesting with the reasoning models, uh, with, like, a one-to-ten ratio.
- 10:01
And then obviously with, like, the new agents, now it's like more sort of Deep Research, NotebookLM. Uh, by the way, Raisa Martin also speaking on the product, uh, product management track.
- 10:10
Um, she's, she's incredible on, uh, talking about the story of NotebookLM. Um, the other really interesting angle, if you wanna take this mental model to the stretch, to stretch it, is the zero-to-one, the ambient agents with no human input.
- 10:23
What kind of interesting, uh, AI output can you get? So to me, that's, that's more a, a useful discussion about input versus output than what is a workflow, what is an agent, how agentic is your thing versus, versus not.
- 10:35
Um, talking about AI News, uh, so you know, it, it is, it is, like, a bunch of scripts in a, in a, in a trench coat. Um, and I realize I've written it three times.
- 10:43
I've written it for the Discord scrape. I've written it for the Reddit scrape. I've written it for the Twitter scrape. And basically, it's just, it's always the same process.
- 10:50
You scrape it, you plan, you recursively summarize, you format, and you evaluate. Um, and, and yeah, that's the three kids in a trench coat. [laughs] Um, and that's really how, what it is.
- 10:59
I run it every day and, like, we improve it a little bit, but then I'm also running this conference. Um, so if you generalize it, that actually starts to become an interesting model for building AI-intensive applications where you start to make thousands of AI calls to service, to serve a particular purpose.
- 11:16
Um, so you sync, you plan, and, and you sort of parallel process. You analyze and sort of reduce that down to, uh, from, from many to one, and then you, uh, deliver, uh, you deliver the contents, um, to the, to the user, and then you evaluate.
- 11:30
And to me, like, that conveniently forms an acronym, SPADE, um, which is, which is really nice. There's also sort of interesting AI engineering elements that are, that are fit in there.
- 11:40
So you can process all these into a knowledge graph. You can, um, turn these into, like, structured outputs, and you can generate code as well. So for example, um, you know, ChatGPT with Canvas or Claude with, um, Artifacts is a way of just delivering the output as a code artifact instead of just, uh, text output.
- 11:58
And I think it's, like, a really interesting way to think about this. So this is my mental model so far. Um, I, I wish I had the space to go into it, but ask me later.
- 12:05
This is what I'm developing right now. But I think what I, what I would really emphasize is, you know, I think, like, there's all sorts of interesting ways to think about what the standard model is and whether it's useful for you in, in taking your application to the next step of, like, how do I add more intelligence
- 12:20
to this in, in a way that's useful and not annoying? Uh, and for me, this is it. Okay, so I've, I've thrown a bunch of standard models in here, but that's just my current hypothesis.
- 12:30
I want you at this conference when, in all your conversations with each other and with the speakers, to think about what the new standard model for AI engineering is.
- 12:36
What can everyone use to improve their applications, and I guess ultimately build products that people want to use, which is what Laurie, uh, mentioned at the start. So, um, I'm really excited about this conference.
- 12:47
It's so, it's such, such an honor and a joy to get it together for you guys, and I hope you enjoy the rest of the conference. Thank you so much. [audience applauds] [outro music]