On Engineering AI Systems that Endure The Bitter Lesson

A new language model arrives, and the trade-offs behind your application change. It may offer better quality, lower cost, better performance on your particular task, or much faster generation. Unlike a hardware upgrade every few years, this can become a weekly engineering decision. Which parts of your system should change with the model, and which should survive it?

The difficulty extends beyond selecting a model. Models are trained through data, feedback, and evaluation; their behavior is not exhaustively designed in advance. A release can introduce new prompting quirks. The provider may document them, leave you to discover them, or supply guidance that does not work well for your application. Meanwhile, new learning algorithms, prompt optimization methods, inference strategies, and agent architectures create separate reasons to revisit the implementation.

Even a competent team can find itself continually scrambling. Keeping the same API model name does not necessarily preserve the underlying behavior: a provider can change the model behind that name. Omar Khattab pushes the question further. Can engineers keep adapting their systems—and will those systems retain their value as the models themselves become more capable?

Rich Sutton’s essay The Bitter Lesson supplies the apparent challenge. In Khattab’s account, AI researchers repeatedly build elaborate methods around their understanding of a domain, only to see those methods overtaken by more general approaches that benefit from greater scale. The complicated methods can encode constraints that prevent further progress. Search and learning provide more room to improve as computation grows.

Search here means exploring a space of possibilities, not retrieving documents. Khattab connects it to what LLM engineers call inference-time scaling: spending more computation while solving a problem. He offers this as his interpretation, without claiming to speak for Sutton.

That leaves an uncomfortable question. Engineering depends on understanding a domain and applying human ingenuity through repeatable principles. If incorporating domain knowledge is what eventually makes an AI approach fail, what exactly should an AI engineer build?

The resolution begins by separating two goals. Maximizing intelligence means improving the ability to figure things out in unfamiliar environments. Building software means delivering a particular behavior reliably. General intelligence already exists in people, yet people have not eliminated the need for software. We still need systems that are robust, controllable, scalable, and understandable.

Checks and balances make this distinction concrete. A reliable system selectively removes agency where discretion would undermine its requirements, while leaving intelligence available elsewhere. Engineering does not require maximizing the freedom of every component. It requires deciding where freedom helps and where the system must enforce a rule.

Define what search should seek and what learning should improve. Scalable search and learning may be the right methods for developing intelligence, but an application still needs an objective. What problem is it solving? What counts as doing that problem well? Those questions are the engineer’s responsibility; the particular search and learning mechanisms can evolve underneath them.

The warning against complicated methods has a familiar software counterpart. Khattab invokes Donald Knuth’s warning, associated with Structured Programming with go to Statements: “Premature optimization is the root of all evil.” Domain knowledge is not inherently harmful. Applying it too early, in ways that constrain a poorly understood system, is the problem.

His example is an opaque piece of code that he describes as computing a square root using a particular floating-point representation on an old machine. The implementation obscures the operation behind constants and low-level manipulation. Change the architecture or number representation, and those assumptions can become invalid. Even where it remains correct, a newer processor instruction or compiler optimization may outperform the handcrafted implementation. These are portability and performance risks, not measured results from the talk.

There might have been a good reason to write that code for its original environment. But repeatedly working at that level creates a system whose useful intent is inseparable from yesterday’s machinery. The practical question is therefore not whether optimization is bad, but how to recognize when it is premature.

No design strategy can predict the next several years of AI. Still, Khattab points to ColBERT and DSPy as projects whose core abstractions have persisted across substantial changes in the ecosystem—from the era of BERT and text-davinci-002 to o4-mini. The comparison concerns the projects’ durability across model generations; it does not mean that ColBERT successively used those generative models.

His hypothesis is that optimization is premature when you hardcode below the level of abstraction you can justify. If the requirement is a square root, express a square root rather than the bit operations that happen to suit one machine. Then ask whether even that operation is an implementation detail of something more general you could express directly. Descend to a lower level only after demonstrating that the higher-level expression is insufficient. This is a decision rule for managing change, not a promise that any abstraction will last forever.

Unnecessary coupling is a familiar software design problem, but applied machine learning often treats it as normal practice: a new model arrives, and the system is rewritten around its particulars. Khattab recalls arguing in a May 2024 post that failures associated with the Bitter Lesson reflect a lack of good high-level ML abstractions. After paradigm shifts, useful systems again acquire modular specializations. Those recurring structures ought to be reusable.

A multilingual question-answering architecture from 2006 makes the point visually. It contains an execution manager, question analysis, retrieval strategies, and multiple corpora; the displayed diagram also connects extraction and answer generation. Its decomposition looks familiar beside modern agent frameworks. The resemblance does not make the older system a modern agent, but it does show that the need to coordinate specialized components predates today’s models.

In ordinary software, an architecture can often survive an upgrade to the machine or operating system beneath it. Khattab’s objection is that ML systems frequently cannot make an analogous transition. Their overall decomposition may remain sensible, yet their components are expressed in ways that bind them to an obsolete implementation. A reusable diagram is not enough; the interfaces and implementation must preserve the separation too.

A prompt can be a reasonable interface for managing an agent, much as a message can direct a remote employee. Training through tensors and objectives serves another purpose: developing the model itself. Khattab’s criticism concerns prompts as a programming abstraction. A large string gives essential requirements and incidental implementation choices the same representation.

The entanglement accumulates in layers:

• Task definition: What the application fundamentally needs to accomplish sits beside wording and examples retained because one model responded well to them.
• Inference strategy: Instructions to reason or act as an agent bake the chosen solving method into the same text as the task.
• Formatting and parsing: XML or JSON instructions mix the application’s meaning with how a response must be transported and consumed.

A later reader cannot easily tell which text defines the requirement and which records an old experiment. The prompt becomes the square-root bit manipulation again, except the underlying operation is no longer clearly named.

Emphatic warnings, expert personas, and promises of tips make the problem especially visible. Such language may be an attempt to influence a particular model, but it does not cleanly express a durable application contract. If every model upgrade requires reconsidering all of it, the specification has become coupled to the mechanism used to obtain compliance.

Separation of concerns starts with accepting that a specification needs more than one form. Natural-language definitions express requirements that are difficult to state otherwise. They should localize that ambiguity rather than turn the whole program into an undifferentiated prompt. Saying what a task means is different from repeatedly adjusting wording to appease a model.

Evals capture the criteria behind those adjustments. If the engineer keeps tinkering because an output does not meet a requirement, the useful investment is to express that requirement as an evaluation. The model may change while the success criterion remains. But evals cannot replace all instructions: asking a system to infer its core behavior from examples is harder than telling it what to do. Explicit definitions and evaluation criteria are complementary.

Code supplies tools, structure, and control over information flow. Privacy boundaries should not depend solely on whether a model follows an instruction. Nor must a model rediscover the order in which functions should compose. Explicit composition fixes that structure by construction, although it does not guarantee that every function is correct.

The programming interface must let these forms work together without collapsing them back into a single string.

A useful programming interface expresses language, evals, and code while keeping them separate from changing implementation choices. Replacing a model should not require redefining the application. Changing the inference strategy—from chain of thought to an agent, or from an agent to Monte Carlo tree search—should likewise preserve what the system is supposed to accomplish. These are desired properties of the abstraction, not a claim that every strategy is interchangeable without engineering work.

Learning also needs to operate against the application’s objective. A component can improve on a generic criterion without making the whole system better at its actual job. Application-level evals provide the target for optimizing the system as a whole, whether the available method is reinforcement learning, prompt optimization, or another approach. The durable investment is the system-specific objective and structure; the lower-level methods used to improve them should remain replaceable.

DSPy is Khattab’s proposed implementation of this separation. He describes three years of development around a division of responsibility: application authors write higher-level AI software, while the framework supplies evolving tools for search and learning, plus adapters for different LLMs. He positions that separation as DSPy’s distinguishing feature. Its central concept is the signature, which gives the task an explicit place in the program.

The recording skips the detailed walkthrough for time. A small Python example using the current documented signature style makes the concept concrete:

import dspy

class AnswerFromContext(dspy.Signature):
    """Answer the question using only the supplied context."""

    context: str = dspy.InputField()
    question: str = dspy.InputField()
    answer: str = dspy.OutputField()

Here, context, question, and answer name the task’s interface. The natural-language definition states the intended behavior. The class does not select a model, prescribe a reasoning strategy, or mingle the task with response-format instructions. It also does not itself enforce factual grounding; that requirement still needs evaluation and an appropriate surrounding program. This illustrates the signature concept rather than reconstructing the skipped demonstration.

None of this guarantees that tomorrow’s code will survive indefinitely. Khattab presents a baseline: avoid hand engineering below the abstraction level that today’s tools allow. A higher-level expression preserves the possibility of benefiting from future machinery without encoding every detail of the current machinery into the application.

The safer investments follow from what remains specific to your system. A model cannot read an unstated specification from your mind, and it will not automatically assemble all the structure and tools your application requires. Put that knowledge into signatures, essential control flow, and tools. Put the criteria behind repeated manual tuning into evals.

As models improve, replace them. As better modules become available, use them. As optimizers gain stronger reinforcement learning or prompt optimization methods, let them improve the application against its own criteria. The engineering work that endures is the work that makes those improvements usable without repeatedly rediscovering what the system was meant to do.