AI Engineer World's Fair 2025
On Engineering AI Systems that Endure The Bitter Lesson
Read the talk
Engineering AI Systems That Endure the Bitter Lesson
Models, prompts, and inference strategies keep changing. Durable AI software separates what an application must accomplish from the machinery used to accomplish it.
From a talk by Omar Khattab
Before you start: Familiarity with language-model prompts, evaluation criteria, and basic software interfaces will help; no prior DSPy experience is required.
Why AI software keeps forcing redesign
A new language model arrives, and the trade-offs behind your application change. It may offer better quality, lower cost, better performance on your particular task, or much faster generation. Unlike a hardware upgrade every few years, this can become a weekly engineering decision. Which parts of your system should change with the model, and which should survive it?
The difficulty extends beyond selecting a model. Models are trained through data, feedback, and evaluation; their behavior is not exhaustively designed in advance. A release can introduce new prompting quirks. The provider may document them, leave you to discover them, or supply guidance that does not work well for your application. Meanwhile, new learning algorithms, prompt optimization methods, inference strategies, and agent architectures create separate reasons to revisit the implementation.
Even a competent team can find itself continually scrambling. Keeping the same API model name does not necessarily preserve the underlying behavior: a provider can change the model behind that name. Omar Khattab pushes the question further. Can engineers keep adapting their systems—and will those systems retain their value as the models themselves become more capable?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The Bitter Lesson creates an engineering puzzle
Rich Sutton’s essay The Bitter Lesson supplies the apparent challenge. In Khattab’s account, AI researchers repeatedly build elaborate methods around their understanding of a domain, only to see those methods overtaken by more general approaches that benefit from greater scale. The complicated methods can encode constraints that prevent further progress. Search and learning provide more room to improve as computation grows.
Search here means exploring a space of possibilities, not retrieving documents. Khattab connects it to what LLM engineers call inference-time scaling: spending more computation while solving a problem. He offers this as his interpretation, without claiming to speak for Sutton.
That leaves an uncomfortable question. Engineering depends on understanding a domain and applying human ingenuity through repeatable principles. If incorporating domain knowledge is what eventually makes an AI approach fail, what exactly should an AI engineer build?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Engineer the objective, not intelligence itself
The resolution begins by separating two goals. Maximizing intelligence means improving the ability to figure things out in unfamiliar environments. Building software means delivering a particular behavior reliably. General intelligence already exists in people, yet people have not eliminated the need for software. We still need systems that are robust, controllable, scalable, and understandable.
Checks and balances make this distinction concrete. A reliable system selectively removes agency where discretion would undermine its requirements, while leaving intelligence available elsewhere. Engineering does not require maximizing the freedom of every component. It requires deciding where freedom helps and where the system must enforce a rule.
Define what search should seek and what learning should improve. Scalable search and learning may be the right methods for developing intelligence, but an application still needs an objective. What problem is it solving? What counts as doing that problem well? Those questions are the engineer’s responsibility; the particular search and learning mechanisms can evolve underneath them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The machine-specific square root
The warning against complicated methods has a familiar software counterpart. Khattab invokes Donald Knuth’s warning, associated with Structured Programming with go to Statements: “Premature optimization is the root of all evil.” Domain knowledge is not inherently harmful. Applying it too early, in ways that constrain a poorly understood system, is the problem.
His example is an opaque piece of code that he describes as computing a square root using a particular floating-point representation on an old machine. The implementation obscures the operation behind constants and low-level manipulation. Change the architecture or number representation, and those assumptions can become invalid. Even where it remains correct, a newer processor instruction or compiler optimization may outperform the handcrafted implementation. These are portability and performance risks, not measured results from the talk.
There might have been a good reason to write that code for its original environment. But repeatedly working at that level creates a system whose useful intent is inseparable from yesterday’s machinery. The practical question is therefore not whether optimization is bad, but how to recognize when it is premature.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the highest abstraction you can justify
No design strategy can predict the next several years of AI. Still, Khattab points to ColBERT and DSPy as projects whose core abstractions have persisted across substantial changes in the ecosystem—from the era of BERT and text-davinci-002 to o4-mini. The comparison concerns the projects’ durability across model generations; it does not mean that ColBERT successively used those generative models.
His hypothesis is that optimization is premature when you hardcode below the level of abstraction you can justify. If the requirement is a square root, express a square root rather than the bit operations that happen to suit one machine. Then ask whether even that operation is an implementation detail of something more general you could express directly. Descend to a lower level only after demonstrating that the higher-level expression is insufficient. This is a decision rule for managing change, not a promise that any abstraction will last forever.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recurring architectures should not require recurring rewrites
Unnecessary coupling is a familiar software design problem, but applied machine learning often treats it as normal practice: a new model arrives, and the system is rewritten around its particulars. Khattab recalls arguing in a May 2024 post that failures associated with the Bitter Lesson reflect a lack of good high-level ML abstractions. After paradigm shifts, useful systems again acquire modular specializations. Those recurring structures ought to be reusable.
A multilingual question-answering architecture from 2006 makes the point visually. It contains an execution manager, question analysis, retrieval strategies, and multiple corpora; the displayed diagram also connects extraction and answer generation. Its decomposition looks familiar beside modern agent frameworks. The resemblance does not make the older system a modern agent, but it does show that the need to coordinate specialized components predates today’s models.
In ordinary software, an architecture can often survive an upgrade to the machine or operating system beneath it. Khattab’s objection is that ML systems frequently cannot make an analogous transition. Their overall decomposition may remain sensible, yet their components are expressed in ways that bind them to an obsolete implementation. A reusable diagram is not enough; the interfaces and implementation must preserve the separation too.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What a monolithic prompt entangles
A prompt can be a reasonable interface for managing an agent, much as a message can direct a remote employee. Training through tensors and objectives serves another purpose: developing the model itself. Khattab’s criticism concerns prompts as a programming abstraction. A large string gives essential requirements and incidental implementation choices the same representation.
The entanglement accumulates in layers:
- Task definition: What the application fundamentally needs to accomplish sits beside wording and examples retained because one model responded well to them.
- Inference strategy: Instructions to reason or act as an agent bake the chosen solving method into the same text as the task.
- Formatting and parsing: XML or JSON instructions mix the application’s meaning with how a response must be transported and consumed.
A later reader cannot easily tell which text defines the requirement and which records an old experiment. The prompt becomes the square-root bit manipulation again, except the underlying operation is no longer clearly named.
Emphatic warnings, expert personas, and promises of tips make the problem especially visible. Such language may be an attempt to influence a particular model, but it does not cleanly express a durable application contract. If every model upgrade requires reconsidering all of it, the specification has become coupled to the mechanism used to obtain compliance.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the specification from language, evals, and code
Separation of concerns starts with accepting that a specification needs more than one form. Natural-language definitions express requirements that are difficult to state otherwise. They should localize that ambiguity rather than turn the whole program into an undifferentiated prompt. Saying what a task means is different from repeatedly adjusting wording to appease a model.
Evals capture the criteria behind those adjustments. If the engineer keeps tinkering because an output does not meet a requirement, the useful investment is to express that requirement as an evaluation. The model may change while the success criterion remains. But evals cannot replace all instructions: asking a system to infer its core behavior from examples is harder than telling it what to do. Explicit definitions and evaluation criteria are complementary.
Code supplies tools, structure, and control over information flow. Privacy boundaries should not depend solely on whether a model follows an instruction. Nor must a model rediscover the order in which functions should compose. Explicit composition fixes that structure by construction, although it does not guarantee that every function is correct.
| Specification form | Primary responsibility |
|---|---|
| Natural language | Explain task meaning and intent |
| Evals | Define what successful behavior looks like |
| Code | Enforce structure, tool access, and information flow |
The programming interface must let these forms work together without collapsing them back into a single string.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make models, search, and learning independently replaceable
A useful programming interface expresses language, evals, and code while keeping them separate from changing implementation choices. Replacing a model should not require redefining the application. Changing the inference strategy—from chain of thought to an agent, or from an agent to Monte Carlo tree search—should likewise preserve what the system is supposed to accomplish. These are desired properties of the abstraction, not a claim that every strategy is interchangeable without engineering work.
Learning also needs to operate against the application’s objective. A component can improve on a generic criterion without making the whole system better at its actual job. Application-level evals provide the target for optimizing the system as a whole, whether the available method is reinforcement learning, prompt optimization, or another approach. The durable investment is the system-specific objective and structure; the lower-level methods used to improve them should remain replaceable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
DSPy makes the task a first-class object
DSPy is Khattab’s proposed implementation of this separation. He describes three years of development around a division of responsibility: application authors write higher-level AI software, while the framework supplies evolving tools for search and learning, plus adapters for different LLMs. He positions that separation as DSPy’s distinguishing feature. Its central concept is the signature, which gives the task an explicit place in the program.
The recording skips the detailed walkthrough for time. A small Python example using the current documented signature style makes the concept concrete:
python
import dspy
class AnswerFromContext(dspy.Signature):
"""Answer the question using only the supplied context."""
context: str = dspy.InputField()
question: str = dspy.InputField()
answer: str = dspy.OutputField()
Here, context, question, and answer name the task’s interface. The natural-language definition states the intended behavior. The class does not select a model, prescribe a reasoning strategy, or mingle the task with response-format instructions. It also does not itself enforce factual grounding; that requirement still needs evaluation and an appropriate surrounding program. This illustrates the signature concept rather than reconstructing the skipped demonstration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Invest in what the model cannot know for you
None of this guarantees that tomorrow’s code will survive indefinitely. Khattab presents a baseline: avoid hand engineering below the abstraction level that today’s tools allow. A higher-level expression preserves the possibility of benefiting from future machinery without encoding every detail of the current machinery into the application.
The safer investments follow from what remains specific to your system. A model cannot read an unstated specification from your mind, and it will not automatically assemble all the structure and tools your application requires. Put that knowledge into signatures, essential control flow, and tools. Put the criteria behind repeated manual tuning into evals.
As models improve, replace them. As better modules become available, use them. As optimizers gain stronger reinforcement learning or prompt optimization methods, let them improve the application against its own criteria. The engineering work that endures is the work that makes those improvements usable without repeatedly rediscovering what the system was meant to do.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Current guides and examples for defining signatures, composing modules, and optimizing AI programs against evaluation metrics.
Code and instructions for training neural retrieval models, indexing passage collections, and searching them with ColBERT.
Further reading
The original DSPy paper explains declarative language-model modules and compilation of pipelines against a chosen metric.
Knuth's December 1974 paper examines readable, correct programs and their systematic transformation when efficiency matters.
A JAVELIN study shows how gold-standard inputs for individual modules help locate errors in multilingual question-answering pipelines.
Read the complete timestamped transcript
- 0:00
[on hold music] So thanks everyone for showing up, and, uh, thanks to the organizers for inviting me and having me here.
- 0:21
Um, I'm excited to talk to you all about, uh, engineering AI systems that endure the bitter lesson. So I'm Omar. Uh, I guess the intro has already happened, so let's not repeat that.
- 0:33
So I mean, if you're here, I think it's probably because you engineer what we might call AI software or you m- maybe manage people or work with people that do.
- 0:42
Um, it's not really a term that's, like, been used as a very special thing this way for that long, so we're all kind of trying to figure out what are the right sort of, uh, basics and fundamentals here and what are the things that are fleeting.
- 0:56
So this is what the talk will be, uh, largely about.
- 0:59
And, you know, like, uh, the name of the game is, it's kind of a meme at this point, every week there's a new large language model, maybe every week is actually too slow at this point, um, that actually changes something, um, in terms of the trade-offs you can strike.
- 1:12
It might not be the state-of-the-art in terms of the best quality necessarily, although sometimes it is. Um, but maybe it's the best performance for certain costs, or it's the best performance for certain types of applications.
- 1:23
Or maybe it's the, you know, the speed that's really incredible. We've seen, like, things like, you know, diffusion now. Um, so every, every week there's a new LLM that you kind of have to think about if you're engineering software in this space, which is really unusual.
- 1:35
Like, if you think back to normal software engineering, you change your hardware every two, three years maybe, um, if that. So this is pretty unusual. Um, the other part that's actually also a little bit weirder is, if you're lucky, the LLM provider has recognized that they're not really building these LLMs.
- 1:54
They're training them. They're emerging based on a lot of nudging and data and iterating on a lot of evals and a lot of vibes as well. Um, and they have realized, you know, if you're lucky, that there are new quirks in their latest models that weren't there before.
- 2:09
And to the surprise of many people to these days, you know, you still get longer and longer prompting guides for the latest models that are supposed to be, you know, closer and closer to AGI.
- 2:18
Um, and if you're less lucky, you have to figure that out on your own, right? Um, if you're even less lucky, the prompting guides from the provider are not even that good, so you have to actually kind of figure out what, what the right thing is.
- 2:29
And every day, uh, maybe at an even faster pace, someone is releasing an archive paper or a tweet or something that introduces a new learning algorithm, maybe some reinforcement learning, uh, bells and whistles, maybe some prompting tricks, maybe a prompt optimization te- you know, technique, something or the other that promises to make your system learn better and
- 2:48
sort of fit your goals better. Someone else is introducing some search or scaling or inference strategies or agent frameworks or agent architectures that are promising to finally unlock levels of reliability or quality better than what you had before.
- 3:02
And I think if you're actually doing a reasonable job now, most likely you're scrambling every week. That's not if you're doing a bad job. That's if you're doing a good job, right?
- 3:09
'Cause you're like, "You know, I've gotta stay on top of at least some of this stuff so that, like, I don't fall behind." Um, and in many cases, like, you know, model APIs actually change the model under the hood even though, you know, it-- you're, you're using the same name.
- 3:22
So it's actually you're forced to scramble. And actually, I would say maybe the question isn't whether you will scramble every week. Uh, maybe a, a different question is, will you even get to scramble for long if you think about the rate of progress of these LLMs?
- 3:35
Like, are they gonna eat your lunch? Right? So these are, I think, questions that are on a lot of people's minds, and this is what the talk is, is, is going to be addressing.
- 3:42
So the talk mentions The Bitter Lesson, which is-- which sounds like this, you know, really ancient, [REDACTED:age] kind of, um, AI lore, but it's just, you know, six years [REDACTED:age], uh, where the current year's, uh, Turing Award winner, uh, Rich Sutton, who's a pioneer of reinforcement learning, wrote this short essay, uh, on his website basically, that says
- 3:59
seventy years of AI has taught him and taught, you know, other people in the AI community from his perspective that when AI researchers leverage domain knowledge to solve problems like, I don't know, chess or something, we build complicated methods that essentially don't scale, and we get stuck, and we get beat by methods that leverage scale a lot
- 4:17
better. What seems to ha-- to work better, according to, uh, to Sutton, is general methods to scale, and he identifies search, which is not like retrieval, more of like, uh, you know, exploring large spaces and learning, um, so getting the system to kind of understand its environment maybe, for example, um, work best.
- 4:36
And search here is what we'd call in the LLM land maybe inference-time scaling or something. So I don't speak for Sutton, and I'm not, you know, uh, suggesting that I have the right understanding of what he's saying or that I necessarily agree or disagree, but I think this is just fundamental and important kind of, um, um, concept
- 4:50
in this space. So I think it's, it's-- it raises interesting questions for us as people who build, uh, you know, and engineer AI systems. Because if leveraging domain knowledge is bad, what exactly is AI engineering supposed to be about?
- 5:03
I mean, engineering is understanding your domain and working in it with a lot of human ingenuity in repeatable ways, let's say, or with principles. So, like, are we just doomed?
- 5:11
Like, are we just wasting our time? Why are we at an AI engineering, you know, fair? And I'll tell you, um, how to resolve this. I've not really seen a lot of people discuss that.
- 5:19
Sutton is talking about-- And a lot of people, you know, throw The Bitter Lesson around, so clearly somebody has to think about this, right? Sutton is talking about maximizing intelligence.
- 5:27
All of us probably care about that to some degree, but which is like something like the ability to figure things out in a new environment really fast, let's say.
- 5:34
Um, all of us kind of care about this to some degree. I'm also an AI researcher. Um, but when we're building AI systems, I think it's important to remember that the reason we build software is not that we lack AGI.
- 5:45
We build software, um, you know, and, and the reason for this and the way you kind of understand this is we already have
- 5:51
General intelligence everywhere. We have eight billions of them. Um, they're unreliable because that's what intelligence is. Um, and they've not solved the problems that we want to solve with software.
- 5:59
That's why we're building software. Um, so we program software not because we lack AGI, but because we want reliable, robust, controllable, uh, scalable systems. Um, and we want these things that-- to be things that we can reason about, understand at scale.
- 6:15
Um, and actually, if you think about engineering and reliable systems, if you think about checks and balances, in any case where you try to systematize stuff, it's about subtracting agency and subtracting intelligence in exactly the right places, uh, carefully, uh, and not restricting the intelligence otherwise.
- 6:29
So, this is a very different axis from the kinds of lessons that you would draw on from The Bitter Lesson. Now, that does not mean The Bitter Lesson is irrelevant.
- 6:36
Let me tell you the precise way in which it's relevant. Um, so the first takeaway here is that scaling search and learning works best for intelligence. This is the right thing to do if you're an AI researcher interested in building, you know, agents that learn really well, really fast in new environments, right?
- 6:50
Don't hard code stuff at all, um, or unless you really have to. But in building AI systems, it's helpful to think about, well, sure, search and learning, but searching for what, right?
- 7:01
Like, what is your AI system even supposed to be doing? What is the, the, the fundamental problem that you're solving? It's not intelligence. It's something else. Um, and what are you learning for, right?
- 7:10
Like, what is the system learning in order to do well? And that is what you need to be engineering, um, not the specifics of search and not the specifics of learning, as I'll talk about in the rest of this talk.
- 7:20
So he's saying, um, uh, Sutton is saying complicated methods get in the way of scaling, uh, especially if you do it early, uh, like before you know what you're doing, essentially.
- 7:30
Did we hear that before? I feel like I heard that back in the 1970s, although I wasn't around. This is, you know, the notion of structured programming, uh, with, with Knuth saying his popular s- you know, uh, uh, phrase in a paper, "Premature optimization is the root of all evil."
- 7:45
I think this is The Bitter Lesson for software and thereby also for AI software. So it's human ingenuity and human knowledge of the domain. It's not that it's harmful.
- 7:56
It's that when you do it prematurely in ways that constrain your system, in ways that reflect poor understanding, they're bad. But you can't get away in an engineering field with not engineering your system.
- 8:06
Like, you're just quitting or something, right? So here's a little piece of code. Um, if you follow me on X, on Twitter, you might recognize it, but otherwise, it-- I think it looks pretty opaque to me in, like, uh, three seconds.
- 8:18
And I can't really look at this and tell exactly what it's doing, and I als-also honestly don't really care. Um, so lo and behold, um, this is computing a square root in a certain floating point representation on an [REDACTED:age] machine.
- 8:28
And I think the thing that jumps at me immediately is this is not the most future-proof program possible. If you change the machine architecture, different floating point representations, better CPUs.
- 8:38
First of all, it would be wrong because, you know, like, it's just hard coding some values here. Um, and second of all, it'll probably be slower than a normal, you know, square root, like maybe is a single instruction or maybe the compiler has a really smart way of doing it or, you know, a lot of other things
- 8:52
that could be optimized for you, right? So someone who wrote this, maybe they had a good reason, maybe they didn't. But certainly, if you're writing this kind of thing often, you're probably messing up as an engineer.
- 9:03
So premature optimization is maybe the, um, square root of all evil or something, but what counts as, uh, premature?
- 9:12
Like, I mean, that's kind of the name of the game, right? Like, if we could just say that, but it doesn't mean anything. Um, so I don't think any strategy will work in tech.
- 9:19
Nobody can anticipate what will happen in three years, five years, ten years. But I think you still have to have a conceptual model that you're working off of. And I happen to have built two things that are, you know, on the order of several years [REDACTED:age] that have fundamentally stayed the same over the years from the days
- 9:34
of BERT, text-davinci-002 up to 4-- uh, o4-mini, and they're bigger now than they ever were. And they're sort of like these, um, f-stable, fundamental kind of, um, abstractions or AI systems around, around LLMs.
- 9:46
So what gives? What happens, um, in order to get something like ColBERT or something like DSPy in this ecosystem, um, and sort of endure a few years, which is like, you know, centuries in AI land?
- 9:57
Um, I'll try to reflect on this and, you know, again, none of this is guaranteed to be something that lasts forever. So here's my hypothesis. Um, premature optimization is what is happening if and only if you're hard coding stuff at a lower level of abstraction that you can-- than you can justify.
- 10:13
Um, if you want a square root, please just say, "Give me a square root." Don't start doing random bit shifts and bit stuff like, you know, m-bit manipulation that happens to appease your particular machine today.
- 10:24
Um, but actually take a step back. Do you even want a square root, or are you computing something even more general? And is there a way you could express that thing that is more general, right?
- 10:32
And only s- you know, uh, stoop down or go down to the level of abstraction that's lower if you've demonstrated that a higher level of abstraction is not good enough.
- 10:42
So I think the bigger picture here is applied machine learning and definitely prompt engineering has a huge issue here. Tight coupling is known-- Tighter coupling than necessary is known to be bad in software, but it's not really something we talk about when we're building machine learning systems.
- 10:55
In fact, the name of the game in machine learning is usually like, "Hey, this latest thing came out. Let's rewrite everything so that we're working around that specific thing."
- 11:03
And I tweeted about this a year ago, thirteen months ago, last May in twenty twenty-four, saying, "The Bitter Lesson is just an artifact of lacking high-level-- good high-level ML abstractions."
- 11:13
Deep learning, uh, scaling deep learning helps predictably, but after every paradigm shift, the best systems always include modular specializations because we're trying to build software. We need those. Um, and every time they basically look the same, and they should have been reusable, but they're not because we're writing code bad.
- 11:27
But we're writing bad code. So here's a n-nice example just to demonstrate this. It's not special at all. Here's a two thousand and six paper. The title could have really been a paper of now, right?
- 11:36
"A modular approach for multilingual question answering." And here's the system architecture. It looks like your favorite multi-agent framework today, right? It has an execution manager. It has some question analyzers and retrieval strategy-- strategists from a bunch of corpora.
- 11:49
And it's like a figure, you know, if you color it, you would think it's a paper maybe from last year or something. Um, now here's the problem. It's a pretty figure.
- 11:56
The system architecturally is actually not that wrong. I'm not saying it's the perfect architecture, but in a normal software environment, you could actually just upgrade the machine, right? Put it on a new hardware, put it on a new operating system, and it would just work, and it would actually work reasonably well because the architecture is not that
- 12:11
bad. But we know that that's not the case for these ML sort of architectures because they're not expressed in the right way. So I think fundamentally, I can express this most, um, passionately against prompts.
- 12:24
A prompt is a horrible abstraction for programming, and this needs to be fixed ASAP. Um, I say for programming 'cause it's actually not a horrible one for management. If you wanna manage an employee or an agent, a prompt is a reasonably kind of like it's an, a Slack channel.
- 12:37
You have a remote employee. If you want to be a pet trainer, you know, working with tensors and, you know, objectives is a great way to iterate. That's how we build the models.
- 12:45
But I want us to be able to also engineer AI systems, and I think for engineering and programming, a prompt is a horrible abstraction. Here's why. It's a stringly typed canvas, just a big blurb, no structure whatsoever, even if structure actually exists in a latent way, um, that couples and entangles the fundamental task definition you want to
- 13:04
say, which is really important stuff. This is what you're engineering with some random over- uh, -fitted, half-baked decisions about, "Hey, this LLM responded to this language, you know, when I ta-talked to it this way," or, "I put this example to demonstrate my point, um, and it kind of clicked for this model, so I'll just keep it in."
- 13:22
And there's no way to really tell the difference. What was the fundamental thing you're solving and, like, you know, what was the random, uh, trick you applied? It's like a square root thing, except you don't call it a square root, and we just have to stare at it and be like, "Wait, why are we shifting to the
- 13:34
left five, you know, by five bits or something?" Um, you're also in-using the inference time strategy, which is, like, changing every few weeks or people are proposing stuff all the time, and you're baking it, literally entangling it into your system.
- 13:46
So if it's an agent, your prompt is telling it it's an agent. Your system has no big, like, deal knowing about the fact it's an agent or a ch- reasoning system or whatever.
- 13:54
What are you actually trying to solve, right? If it's like if you're writing a square root function, and then you're like, "Hey, here is the layout of the structs in memory or something."
- 14:02
Um, you're also talking about formatting and parsing things. You know, write in XML, produce JSON, whatever. Like, again, that's really none of your business most of the time. So you wanna write a human-readable spec, but you're saying things like, "Do not ignore this.
- 14:16
Uh, generate XML. Answer in JSON. You are Professor Einstein, a wise expert in the field. Uh, I'll tip you $1,000," right? Like, that is just not engineering, guys. Um, so what should we do?
- 14:28
Trusty [REDACTED:age] separation of concerns, I think, is the answer. Your job as an engineer is to invest in your actual system design and, you know, starting with the spec.
- 14:38
The spec, unfortunately or fortunately, cannot be reduced to one thing, and this is the time I'll talk about evals. I know everyone here is about evals, so this is the one line about evals that makes this talk about evals.
- 14:48
A lot of the time, um, you want to invest in natural language descriptions because that is the power of this new framework. Natural language definitions are not prompts. They are highly localized pieces of ambiguous stuff that could not have been said in any other way, right?
- 15:03
I can't tell the system certain things except in English, so I'll say it in English. But a lot of the time, I'm actually iterating to, uh, to appease a certain model and to make it perform well relative to some criteria I have, not telling it the criteria, just tinkering with things.
- 15:19
There, evals is the way to do this because evals say, "Here is what I actually care about. Change the model. The evals are still what I care about." It's a fundamental thing.
- 15:27
Now, evals are not for everything. If you try to use evals to define the core behavior of your system, you will not learn. Induction, learning from data, is a lot harder than following instructions, right?
- 15:36
So you need to have both. Code is another thing that you need. You know, a lot of people are like, "Oh, it's just like a, you know, just, just ask it to do the thing."
- 15:43
Well, w-who's gonna define the tools? Who's gonna define the structure? How do you handle information flow like, you know, like things that are private should not flow in the wrong places, right?
- 15:51
You need to control these things. Um, how do you apply function composition? LLMs are horrible at composition because neural networks kind of essentially don't learn things that reliably. Function composition in, in software is always perfectly reliable, basically, right?
- 16:04
By construction. So a lot of the things, um, are often best delegated to code, right? But it's, it's hard, and it's really important that you can actually juggle and combine these things, and you need a canvas that can allow, that can allow you to combine these things well.
- 16:18
When you do this, a good canvas, the definition here of a good campus or a cam- uh, the, the, the criteria for a good canvas is that it should allow you to express those three in a way that's highly streamlined and in a way that is decoupled and not entangled with models that are changing.
- 16:32
I should just be able to hot-swap models. Uh, inference strategies that are changing. Hey, I wanna switch from a chain of thought to an agent. I wanna switch from an agent to a Monte Carlo tree search, whatever the latest thing that has come out is, right?
- 16:43
I should be able to just do that. Um, and new learning algorithms. This is really important. We talked about learning, but learning, uh, is, you know, always happening at the level of your entire system if you're engineering it, or at least you've gotta be thinking about it that way, where you're saying, "I want the whole thing to
- 16:57
work as a whole for my problem, not for some general default," right? So that's what the evals here are going to be doing. And you want a, a, a way of expressing this that allows you to do reinforcement learning but also allows you to do prompt optimization, but also allows you to do any of these things at
- 17:10
the level of abstraction that you're actually working with. So the second takeaway is that you should invest in defining things specific to your AI system and decouple from, uh, the lower-level swappable pieces 'cause they'll expire faster than ever.
- 17:23
Um, so I'll just conclude by telling you we've built and been building for three years this DSPy framework, which is the only framework that actually decouples, uh, your job from, which is writing the lower-level AI software from our job, which is giving you powerful evolving toolkits for learning and for search, which is scaling, um, and for swapping
- 17:43
LLMs through adapters. Um, so there's only one concept you have to learn. It is a new concept, which is, which we call signatures, a new first-class concept. If you learn it, you've learned DSPy.
- 17:54
Um, I'll have to unfortunately skip this 'cause of the, 'cause of the time for the other speakers, uh, but let me give you a summary. I can't predict the future.
- 18:01
I'm not telling you if you do this, you know, this, the code you write tomorrow will be there forever. But I'm telling you the least you can do, this is not like, uh, the, the, the kind of the top level.
- 18:09
It's just like a, the base, the baseline, I would say, is avoid hand engineering at lower levels than today allows you to do, right? That's the, that's the big lesson from The Bitter Lesson and from premature optimization being the root of all evil.
- 18:21
Um, among your safest bets, they could, uh, turn out to be wrong, I don't know, is, um, models are not anytime soon gonna read, uh, specs off of your mind.
- 18:30
I don't know if, like, we'll figure that out. Um, and they're not going to magically collect all the structure and tools specific to your application. So that's clearly stuff you should invest in, right?
- 18:39
When you're building a system. Invest in the signatures, which I, again, uh, you can learn about on the DSPy site, uh, dspy.ai. I-invest in essential control flow and tools, and invest in evals for things that you would otherwise be iterating on by hand.
- 18:53
And ride the wave of swappable models. Ride the wave of the modules we build. Um, you just swap them in and out. And ride the wave of optimizers, which can do things like reinforcement learning or prompt optimization for any application that it is that you've built.
- 19:04
Uh, all right. Thank you, everyone. [upbeat music]