DSPy: The End of Prompt Engineering

How do you search ten thousand contracts for a particular clause or paragraph? In Kevin Madura’s consulting work, that problem sits alongside law-firm investigations, process improvement, and internal AI deployment. DSPy gives his team a way to express these tasks as modular Python programs whose functions can call language models. The work becomes defining the processing steps and their inputs and outputs, rather than repeatedly adjusting strings by hand.

The workshop repository collects small examples: sentiment classification, PDF analysis, multimodal questions, web research, document-boundary detection, recursive summarization, and GEPA optimization. These are demonstration utilities assembled over several days, not a finished production application. Their value is in showing how the same primitives combine across quite different tasks.

Keep the program’s intent and control flow stable while changing the model underneath. A DSPy call looks like a function with declared inputs and outputs. That removes routine work such as navigating .choices[0].message.content, parsing strings, and moving JSON between stages. Madura connects this to Omar Khattab’s systems-minded description of DSPy: the system’s design should remain useful even as model capabilities change. Generated prompts remain inspectable when the abstraction needs investigation.

Pydantic AI, LangChain, and Agno are also ways to build applications around models. Madura’s preference comes from how DSPy fits his team’s work, and he acknowledges that its declarative style initially took some effort to understand. The examples are an invitation to test that fit against a concrete workload.

DSPy separates six responsibilities that otherwise tend to accumulate inside one prompt-building function:

Modules follow a PyTorch-inspired organization. Adapters bridge the declarative interface and the strings ultimately sent to a model. Optimization is optional: the programming abstractions are useful before any search for better prompts begins.

A signature can be a short string or a class backed by Pydantic. In the class form, field names do more than label Python attributes: they become part of the instructions the model sees. A name such as sentiment communicates an intended task; a description explains its scale or constraints. The adapter carries those names, descriptions, and types into the generated message. Choose names for both the Python reader and the model.

An existing, carefully developed prompt can go in the signature’s docstring. DSPy also permits additional instructions and lower-level changes to the final message. Adopting signatures does not require throwing away useful wording; it gives that wording a place inside a composable program.

For a sentiment prototype, the shorthand declares a text input and an integer output. Instructions then establish the direction of the scale:

import dspy

sentiment_signature = dspy.Signature(
    "text: str -> sentiment: int",
    instructions=(
        "Classify the sentiment of the text. "
        "Lower numbers mean more negative sentiment; "
        "higher numbers mean more positive sentiment."
    ),
)

classify_sentiment = dspy.Predict(sentiment_signature)

This creates a predictor to use with a configured language model. The type alone cannot define the meaning of an integer, so the instructions remain part of the contract. Start with that small declaration, inspect its behavior, and add constraints or a surrounding module as the task becomes clearer.

A module packages a reusable piece of behavior. DSPy includes modules for several familiar techniques, and a custom module can make an existing approach available to the rest of a program.

Madura questions how much explicit chain-of-thought prompting adds for newer models, but uses it to illustrate how a prompting technique can live inside a replaceable module. ProgramOfThought provides a Python interpreter and can use a custom execution harness. Other modules support output comparison, refinement, and parallel processing.

The time-entry corrector shows why the surrounding Python matters. A client needed hundreds of thousands of entries to follow consistent conventions, including tense, capitalization, and periods at the end of sentences. Initialization creates a predictor for changing tense and supplies the relevant instructions. The module’s forward method then calls that predictor and applies hard-coded business rules. Linguistic judgment and deterministic formatting can occupy separate steps while remaining one callable component.

Tools are ordinary Python functions exposed through DSPy’s tool interface; LiteLLM underlies the model interface. The introductory agent supplies get_weather and search_web to ReAct, uses the signature question -> answer, and caps the loop at five rounds. Asking about the weather in Tokyo then gives the model a question, a bounded opportunity to use the tools, and an answer field to populate. The functions implement the capabilities; the agent chooses when to invoke them.

An adapter turns a signature, its inputs, and associated instructions into model messages. In the clinical-note example, a string input and a nested patient-details output become a prompt containing the fields and their schema. A default adapter handles the ordinary case; replacing it changes how the model sees the task without requiring a rewrite of the program.

Madura reports a use-case-dependent 5–10% performance improvement from JSON-versus-BAML adapter testing, without specifying the task or metric. He attributes the testing to Prashanth and suggests that the more readable format might also use fewer tokens. The historical BAMLAdapter discussion describes simplified schema notation derived from BAML, rather than the full BAML language. That workshop-era formatting example should not be read as a version-independent API contract or a general performance guarantee.

Multimodal inputs use the same signature-based approach, provided the selected model supports the modality. The Attachments library prepares files as model-ready content and integrates with DSPy. In the first document example, it fetches an NVIDIA SEC Form 4, prepares the document, and passes it with a question asking how many shares were sold. Answering requires combining two transactions; the displayed answer is 497,797 shares.

An audience question establishes the actual data path: there is no vector store, embedding step, or retrieval search in this example. The model receives the document images and extracted text directly. Madura initially speculates about OCR, but the mechanism needed here is simply document preparation followed by a model call over that content.

Once a program has measurable inputs and outputs, an optimizer can search for instructions that improve its behavior. Madura recommends trying this before committing to fine-tuning infrastructure. One motivation is model transfer: preserve the application’s structure while finding a prompt that makes a less expensive model adequate for the task.

Madura illustrates model transfer with a hypothetical million-call-per-day classifier moving from GPT-4.1 to GPT-4.1 nano, whose score rises from 70% to 87% after optimization. Those are teaching values, not results from a run. His accompanying suggestion of multiple orders of magnitude in savings is too broad for the named models’ like-for-like token rates: their launch pricing differs by a factor of twenty. Total workload savings also depend on token usage and the cost of optimization.

The optimizer works on prompts inside the composed program, measuring the resulting input-to-output behavior. That is why decomposition matters: a system with several modules exposes several places where instructions can improve. Madura returns to Khattab’s distinction here—DSPy supplies programming abstractions that happen to be optimizable. Its usefulness does not depend on running an optimizer first.

A discussion between Karpathy and Dwarkesh provides Madura with an intuition about this search. Models can discover weaknesses in an LLM judge; he wonders whether prompt optimization can exploit a related sensitivity in a useful direction, finding instructions that elicit better behavior. This is his analogy for the search process, rather than its formal definition. The practical procedure is simpler:

1. Decompose the task into modules.
2. Define metrics that express the behavior you need.
3. Optimize the program against those metrics.

Madura then cites Chris Potts’s discussion of GEPA matching or exceeding GRPO in studied settings. That comparison belongs to specific research experiments, not every model or fine-tuning workload.

The metric determines what the optimizer learns to improve. Each proposed change produces outputs that the metric scores. For time-entry correction, some requirements admit exact checks; others can use an LLM judge to assess whether the generated text meets qualitative criteria. These choices define the feedback available to the optimizer, so a vague or misaligned metric leaves the search pursuing the wrong behavior even when its score improves.

After pointing to the DSPy community and Discord, Madura opens the notebook and attempts to start Arize Phoenix for tracing the calls underneath the examples. The setup does not clearly settle into a working tracing demonstration, so he proceeds with the notebook. It loads .env credentials and creates reusable model configurations: a harder workload might use GPT-5, a lighter one GPT-4.1 nano, and another Gemini 2.5 Flash. OpenRouter supplies access to several providers for the next comparison.

Asked which AI company is best, Claude, Gemini, and GPT-4.1 mini initially hedge. Restricting the output to a Literal containing Google, Anthropic, and OpenAI removes that escape hatch; each displayed answer selects its own provider. The demonstration shows how an output type changes the space of allowed responses, rather than establishing anything about which provider is best.

The responses arrive quickly because Madura has already run those inputs and DSPy serves cached results. Adding a space to hello changes the request and triggers a live call. That distinction matters when inspecting timing or usage: a repeated notebook cell may be exercising the cache rather than the provider.

The sentiment example then returns to text: str -> sentiment: int, now with instructions specifying lower as more negative and higher as more positive. Madura changes the input from a complaint about a hotel to a statement about feeling happy; the latter returns 8. The result is accessed as a typed prediction rather than parsed from a free-form response. In the usage inspection that follows, the cached result has an empty usage object, while a changed call through Azure exposes token-usage details.

The live Form 4 example makes document preparation visible. Attachments uses pdfplumber, produces Markdown, and prepares images before the question about total shares sold is passed to the model. Madura then changes the output contract: instead of answering one question, a ChainOfThought document analyzer returns a dictionary of information it considers important. The model chooses the keys and values within that broad structure.

For a specific business workflow, the schema can instead prescribe the required fields. Madura defines Pydantic classes for filing information and transactions, then makes that schema the analyzer’s output type. The document input still contains prepared text and images; the response now contains filing date, form date, form type, transactions, and the answer in a predictable structure.

The structured result supports dot notation, so downstream logic can use the returned object directly.

inspect_history exposes the messages beneath the abstraction. For the document analyzer, Madura shows the generated system message, input and output fields, extracted document content and metadata, and the formatted model response. DSPy parses that response into the object available as response.document_schema. The short signature is therefore an interface to a longer message-building and parsing process, not the literal prompt by itself.

The clinical example makes the adapter boundary easier to see. A patient-details model contains a nested patient-address object, and a signature extracts that structure from a clinical note. Python’s with context lets a call override the globally configured language model or adapter. In the live sequence, Madura switches from his Smart LLM configuration, which he identifies tentatively as GPT-4.1, to Anthropic while also using the BAML adapter. After a connection interruption, both calls show the same extracted output.

Inspecting the last two calls reveals the representation change:

import dspy

dspy.inspect_history(n=2)

One message contains the expanded JSON schema; the other uses the more compact BAML-derived notation. Since the live comparison changes both the model and the adapter, it demonstrates inspectability and interchangeable configuration, not an isolated measurement of adapter quality.

A photograph of street signs becomes another typed task: given the image and the current time, can the driver park now, and when should they leave? The signature asks for a Boolean and a departure-time string. The displayed prediction contains can_park_now=True and a departure time of 15:30, alongside a reasoning field. These are the shown model’s interpretation of that input, not a general parking rule.

The reasoning field prompts an audience question. It appears because the call uses dspy.ChainOfThought, which adds reasoning even though the original signature does not declare it. Switching to dspy.Predict produces the requested output fields without that automatically added reasoning field.

Wrapping the predictor in an image-analyzer module adds ordinary program state. The demonstrated instance has a module identifier, stores its predictor on self, and increments a counter each time it is called. The same arrangement can surround inference with business rules or database operations. forward is where that control flow runs; the language-model call is one operation within it.

The biography agent exposes two functions, perplexity_search and get_url_content, to a ReAct answer generator. Its module selects Gemini 2.5 and forwards the requested criteria to the agent. The question is whether a person has been at their company for more than ten years, so the workflow can seek current evidence rather than rely entirely on model memory. Madura reports true for Satya and expects false for Brian.

A ReAct result includes a trajectory: the tool names, arguments, and observations from the agent’s steps. This is the tool-using counterpart to inspecting the generated response, and it provides a concrete debugging surface when a conclusion looks wrong. An asynchronous variant runs separate biography requests in parallel.

The next example composes a support analyzer from several classification modules. Messages about a broken sink or a light being out receive sentiment labels—positive, neutral, or negative—plus urgency and category. The enclosing module combines these outputs, and metrics compare them with the dataset’s labels. Urgency uses a simple exact-match check; other criteria can be more subjective.

The intended optimization sequence evaluates the baseline, proposes new prompts, and measures their behavior. Madura estimates that the example would take about twenty minutes, so he explains it rather than completing the run live. GEPA adds textual feedback to the score: a teacher model can explain why a classification was wrong and what the expected answer should have been. That teacher need not be more capable than the model being optimized. The feedback guides subsequent prompt proposals and exploration of candidates along a Pareto frontier, preserving useful alternatives rather than treating every error as an unexplained scalar penalty.

A client’s file dump may contain contracts, SEC filings, and unrelated images. Processing every file the same way wastes both task-specific knowledge and model choice. Madura’s Python program configures regular, small, and image-oriented models, prepares each file with Attachments, classifies it, and branches on the result. Images can be routed to Gemini while other work uses GPT-4.1; these are configurable workload choices, not a demonstrated ranking of the models.

The classification result drives ordinary Python control flow:

• SEC filing: Identify the subtype and apply the corresponding extraction logic.
• Contract: Generate a summary and locate document boundaries.
• City infrastructure: Interpret the image visually.

The classifier itself is a Predict call over page images. In the shown setup, the first three pages help select a document type from SEC filing, patent filing, contract, city infrastructure, or other. A small model handles Form 4 extraction, and an assertion checks that the returned result matches the expected kind of document.

The contract branch requests both recursive summaries and boundaries, then runs on a sample contract. An audience challenge about the city-infrastructure label exposes the classifier’s scope: the workshop folder contains a few sharply different examples, including the parking signs. Madura recommends stricter classification for production, potentially with multiple passes or different models. A successful decision among these few files does not establish reliability on an arbitrary client archive.

The contract summarizer uses three signatures to decompose document content and build a summary by working through chunks. This places long-document processing in the program’s control flow instead of assuming the entire document must fit into one request. The resulting summary appears alongside boundaries separating the main agreement from its schedules.

Madura checks the narrated thirteen-page example against the PDF using zero-based page indices:

He walks through the main document, the next two pages of schedule one, the single page of schedule two, and schedule three through the end. Underneath, the PDF becomes a list of page images. A classifier labels each page, and a subsequent signature converts those classifications into section names and pairs of boundary indices.

An attempted return to the city-infrastructure example is interrupted by a useful question: does the boundary stage receive the original document? Initially, it receives page/classification tuples. Madura opens the boundary detector and points to his companion article, Using DSPy to Detect Document Boundaries. Each page image is classified asynchronously, the results are gathered, and the next signature requests a dictionary mapping section names to pairs of integers.

That compact intermediate representation does not cut the agent off from the source. The boundary stage uses ReAct with a get_page_images tool. If a proposed boundary needs checking, the agent can request selected page images—Madura describes inspecting three pages—and use them to construct its final answer. The sequence is:

1. Classify page images independently.
2. Gather page indices and classifications.
3. Propose section boundaries from that compact list.
4. Retrieve relevant page images to check a proposed boundary.
5. Return the section-to-range mapping.

The tool creates a path back to the evidence without placing the full document in every stage’s initial input. Madura describes the implementation as a prototype with substantial room for improvement.

Asked whether DSPy is essentially structured-output enforcement, Madura agrees that structured outputs are part of it, but emphasizes the coordination around them: moving results into the next function, selecting another model for a call, changing the message format, and preserving control flow. Pydantic and native structured outputs remain valid tools. The demonstrated applications show how those concerns fit together; they do not establish production readiness by themselves.

A question about GEPA leads Madura to a previously run time-entry example that actually uses MIPRO, a different optimizer. He suggests starting with perhaps ten to a hundred well-structured input/output examples when the metrics accurately describe the task. Dataset quality and the definition of success matter more than merely accumulating examples.

Madura reports that the time-entry corrector’s MIPRO run improved its displayed score from 86 to 89; the score unit and evaluation split are not stated. The example is still under development. More useful than the aggregate alone is its per-metric breakdown: a weak criterion may need better data, a more precise definition, or decomposition into separate checks. A criterion that does not matter to the application may not belong in the objective at all.

The optimizer returns another callable module. It can be serialized, saved, loaded, and used wherever the original module was used. Inside it, prompt changes can express requirements the initial instructions omitted. Madura’s example is a recurring capitalization failure: the optimization process can add an explicit instruction to capitalize names correctly. He connects this to Chris Potts’s description of discovering latent requirements from data.

For GEPA, a language model proposes new prompts, the program runs with those prompts, and evaluation feeds the next iteration. The resulting instruction string is accessible separately, but the module also retains the program interface and signature state. That explains why the artifact is more useful than an isolated prompt pasted into a document: the optimized behavior remains attached to a callable component with defined inputs and outputs. The audience’s request to inspect that state leads to the final demonstration.

Madura’s experimental DSPy Hub explores a repository of pre-optimized programs, analogous to downloading a model from Hugging Face. He loads a downloaded program, calls it, and inspects its state. The object contains a Predict component, a signature he describes in terms of time and reasoning, a long optimized instruction string, and definitions of the inputs and outputs. This is the concrete handoff from an optimization run to a reusable component.

A possible use is a document classifier that finds invoice-bearing pages in a collection of a hundred thousand documents, then moves into another project or another team’s processing pipeline. Conventional machine-learning classification is also an option. Madura is unsure whether a general hub will be useful when optimized tasks are highly specific, but the ability to save and reuse the state is valuable independently of a public sharing service.

Delayed user feedback raises a separate question. A thumbs-up or thumbs-down may arrive minutes or a day after an answer. The proposed loop adds that feedback to a truth dataset and reruns optimization using the latest state. The participants explicitly recognize this as offline reoptimization. Online adaptation with delayed feedback is discussed only as something an engineer might build; no implementation is shown.

The strongest candidate remains a well-defined LLM task with known inputs and assessable outputs. Madura particularly values optimization when it can preserve useful performance on a smaller, cheaper model. That is a reason to evaluate the technique for a workload, not to replace conventional ML indiscriminately.

The last question concerns expense and large contexts in optimization datasets. DSPy makes model calls convenient, but their number still matters: Madura uses a million asynchronous calls as an example of how a program can create substantial spend. A short signature also expands into a longer prompt containing instructions and schema information, so the visible declaration is not an estimate of input tokens.

A custom adapter can compress the message representation. Program logic or modules can track context state and decide what to retain, summarize, or pass onward. Madura does not offer a concrete recipe for shrinking large-context optimization datasets; his expectation that larger windows or more abstracted context management may help is an intuition. The operational responsibility stays with the program: choose the calls, inspect the messages, measure the behavior, and manage the context that each stage actually needs.