AI Engineer World's Fair 2024
Pydantic is STILL all you need
Read the talk
Pydantic is still all you need
Typed responses turn model output into ordinary application data: objects you can validate, stream, search with, render and send to an execution engine.
From a talk by Jason Liu
Before you start: Familiarity with Python type hints, basic Pydantic models and chat-model API calls will help you follow the examples.
An API should return more than a string
Imagine hiring an intern to write an API and receiving a string that you must feed into json.loads, followed by a hopeful check that the expected fields exist. Jason Liu’s response is to replace the intern with Devin and ask it to use FastAPI and Pydantic. The joke opens a practical question: why accept this interface from a language model when it would be unacceptable from an ordinary API?
Parsing and repairing strings was familiar work with GPT-3, but application developers already have better tools for representing data. Pydantic in Python, Ecto in Elixir and Active Record offer established ways to work with structured records. Without schemas, an integration loses compatibility, composability and reliability. Liu’s earlier Pydantic talk proposed function calling and typed responses as the alternative: nested models express modular structures, while validators define additional conditions the returned data must satisfy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Instructor grows around a small interface
The installation remains pip install instructor. By this talk, Instructor had reached 1.0, launched implementations in Python, TypeScript, Ruby, Go and Elixir, and added a newly built Rust version. Liu describes its value as roughly 600 lines of code that application developers do not want to write themselves. Liu reports 40% month-over-month growth for the Python library and downloads around 2% of OpenAI’s library downloads. He does not specify the measurement window or counting methodology.
The major-version change was modest: the client setup moved from instructor.patch to instructor.from_openai. Provider coverage expanded to Ollama, LlamaCPP, Anthropic, Cohere, Gemini and Groq. The intended stability comes from keeping the application contract small: as models gain function-calling capabilities, the same typed interface can sit above them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Return an object, stream objects, or stream a partial
Start by defining a Pydantic model, wrapping the model client and passing response_model=User. This resembles FastAPI’s response-model interface: the model declaration determines what the caller receives. Improved typing also lets the IDE infer a User return value and flag incorrect field access. A Python call using that interface looks like this, with the provider’s model identifier supplied through configuration:
python
import os
import instructor
from openai import OpenAI
from pydantic import BaseModel
class User(BaseModel):
name: str
age: int
client = instructor.from_openai(OpenAI())
user = client.chat.completions.create(
model=os.environ["OPENAI_MODEL"],
response_model=User,
messages=[
{"role": "user", "content": "Jason is 30 years old."}
],
)
print(user.name, user.age)
The application works with fields on an object instead of extracting a JSON string from the provider response.
When the input contains two users, iterable streaming returns each completed User as it becomes available. With stream=True, the application can process the first object before the full extraction finishes, reducing the wait for useful output. The inspected slide uses create_iterable and shows a generator of typed users.
Partial streaming addresses a different problem: rendering one evolving object, such as the structure behind a generative UI. It avoids making the application maintain its own JSON parsing stack while tokens arrive. Liu describes this in terms of whole-object validation; current partial-response documentation instead describes incremental snapshots with optionalized fields and explicitly says validators are unsupported during streaming. An intermediate snapshot should therefore not be treated as a final object that has passed every validator.
Liu summarizes the interface as one noun—the client—and three verbs, spoken as create, create_with_iterable and create_with_partial. The naming differs from the slide’s create_iterable and today’s create_partial, but the three consumption patterns are straightforward:
| Pattern | What the caller consumes |
|---|---|
| Create | One completed model |
| Iterable | Completed models as they arrive |
| Partial | Successive snapshots of one model |
The remaining application decisions live in the response model, validators and messages array. Features that fit within that messages interface need not introduce another application abstraction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Validation errors become conditional instructions
Generation, RAG and extraction all benefit from the same shift: program with data structures and explicit rules. The principle also applies to vision and agents, although Liu warns that models differ in how well they respond to retry feedback. The programming language has not changed; the task is to bring familiar programming techniques back into the model interaction.
A field validator can define correctness without putting every rule into the initial prompt. In Liu’s example, the prompt does not request uppercase names. Python checks the returned name and raises an informative error if it is not uppercase:
python
from pydantic import BaseModel, field_validator
class User(BaseModel):
name: str
@field_validator("name")
@classmethod
def require_uppercase(cls, value: str) -> str:
if value != value.upper():
raise ValueError("Name must be uppercase.")
return value
With retrying enabled, Instructor catches the validation failure and supplies the error message to the model for another attempt. The validator rejects rather than silently uppercasing the value: its message becomes a conditionally added instruction. Liu’s displayed result is JASON; he describes it as appearing after one API call, without spelling out the underlying retry accounting.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Validate relationships across receipt fields
A receipt requires more than independently valid fields. In a Ramp-style processing flow, a vision model extracts product prices, quantities and a total cost. A model-level validator checks whether those fields agree. For a receipt model whose total is defined as the sum of its line items, the rule can be expressed directly:
python
from decimal import Decimal
from typing import Self
from pydantic import BaseModel, Field, model_validator
class Product(BaseModel):
price: Decimal = Field(ge=0)
quantity: int = Field(gt=0)
class Receipt(BaseModel):
products: list[Product]
total_cost: Decimal
@model_validator(mode="after")
def check_total(self) -> Self:
expected = sum(
(item.price * item.quantity for item in self.products),
Decimal("0"),
)
if expected != self.total_cost:
raise ValueError(
f"Line items sum to {expected}; "
f"total_cost is {self.total_cost}."
)
return self
This validates a relationship among extracted values, not just their individual types.
Liu says this receipt discrepancy does not occur in 99% of cases, without supplying an evaluation. His concern is what happens when it does: an explicit validation failure can appear as a red bar in Datadog instead of quietly becoming accepted data. Re-asking then gives the model an opportunity to correct the inconsistency.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give generated answers a useful shape
An ordinary chat response is already an object, but its useful payload is often just a content string. That leaves the application responsible for parsing anything more elaborate. A structured RAG answer can instead contain both answer content and a list of follow-up questions. Those questions can be grounded in the supplied context, exposing other things the system is equipped to answer.
Liu also describes a production validator that extracts URLs with a regular expression and checks whether they return HTTP 200. He describes the request method as POST. When a check fails, the next attempt is instructed to omit the link rather than keep trying to invent a replacement. This catches some bad links, but HTTP success means the particular request succeeded; it does not establish that the destination supports the generated claim. POST also has different semantics from retrieving a page.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn search intent into a typed request
Consider a request for the latest news from Z. Semantic similarity alone does not enforce recency. BM25 may help match the source, and different sources may require different indices, but the application still needs to represent the user’s constraints explicitly. Liu introduces a Search object that makes those decisions inspectable:
| Field | Purpose |
|---|---|
query | Search text |
start_date | Beginning of the date range |
end_date | Optional end of the range |
limit | Requested result count, such as five |
source | Backend or index to query |
The language model translates intent into these fields; the search implementation knows how to apply them.
A create_search function takes a string and returns the search object. Both the model API call and the mechanics of querying the chosen endpoint can remain behind that boundary. Structured requests also make behavior measurable: the application can validate date ranges and inspect how often the model produces zero-day, one-day or longer intervals.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compose retrieval with two models and two functions
A comparison question—what is the difference between X and Y?—can require more than one search. Iterable output lets the model produce a search request for each subject. The application can execute those requests in parallel and collect their results. Liu organizes the model-facing portion around two data models, search requests and answers, and two functions that return them.
The resulting RAG flow is ordinary orchestration:
- Pass the question to
create_searchand obtain multiple search objects. - Execute each search against its selected backend.
- Gather the retrieved context.
- Pass the question and context to
answer_question. - Render the structured answer.
An OpenAPI endpoint or React interface can consume the resulting structure without taking over the model interaction. The LLM sits behind a typed boundary. That boundary guarantees conformance to declared types and validation constraints, not the factual truth of every answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extract labels and meeting records
Classification can begin with a single constrained field: label: Literal["spam", "not spam"]. The schema restricts the accepted result to the application’s label set instead of accepting explanatory prose where a label belongs. Liu reports about a 15% classifier accuracy improvement from adding chain of thought. He supplies no dataset, model, baseline or evaluation conditions, and does not distinguish relative percent from percentage points. The concrete schema lesson is that a reasoning field can coexist with a tightly constrained classification field.
Meeting extraction extends the same approach to a richer record: meeting type, title, action items and summary. An action item’s owner may initially be a string, but a domain validator could require that owner to be one of the participants. This moves correctness beyond the shape of a record toward whether its values belong to the application’s known entities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An image table becomes a DataFrame
A Google Calendar integration could supply the participant list for that ownership check. The next extraction example pushes the type boundary further: a Table has a caption string and a field whose application-facing value is a pandas DataFrame, even though the model produces a Markdown table from an image.
The mechanism combines a native Python type with parsing, serialization and schema metadata. Annotated, standardized in PEP 593, provides the metadata mechanism; Liu discusses annotations without naming the proposal number. The Markdown DataFrame pattern gives each layer a distinct job:
- Native type:
InstanceOf[pd.DataFrame]describes the runtime object and enables DataFrame method completion. - Input conversion:
BeforeValidatorparses the model’s Markdown into a DataFrame before the instance check. - Output conversion:
PlainSerializerturns that DataFrame back into Markdown when serialized. - Model-facing schema:
WithJsonSchemadescribes the string representation the model should generate.
The model and the application can therefore use different representations of the same table.
For simple pipe-delimited tables without escaped pipes or multiline cells, the pattern can be written as follows:
python
from typing import Annotated
import pandas as pd
from pydantic import (
BaseModel,
BeforeValidator,
InstanceOf,
PlainSerializer,
WithJsonSchema,
)
def parse_markdown(value: str | pd.DataFrame) -> pd.DataFrame:
if isinstance(value, pd.DataFrame):
return value
rows = [
[cell.strip() for cell in line.strip().strip("|").split("|")]
for line in value.strip().splitlines()
if line.strip()
]
if len(rows) < 2:
raise ValueError("Expected a Markdown header and separator row.")
header, separator, *data = rows
if len(separator) != len(header) or any(
not cell or set(cell) - set("-:") or "-" not in cell
for cell in separator
):
raise ValueError("Invalid Markdown separator row.")
if any(len(row) != len(header) for row in data):
raise ValueError("Every row must match the header width.")
return pd.DataFrame(data, columns=header)
MarkdownDataFrame = Annotated[
InstanceOf[pd.DataFrame],
BeforeValidator(parse_markdown),
PlainSerializer(
lambda frame: frame.to_markdown(index=False),
return_type=str,
),
WithJsonSchema({
"type": "string",
"description": "A Markdown table with a header and separator row.",
}),
]
class Table(BaseModel):
caption: str
data: MarkdownDataFrame
table = Table.model_validate({
"caption": "Products",
"data": "| Product | Quantity |\n| --- | --- |\n| Pen | 2 |",
})
csv_text = table.data.to_csv(index=False)
Here the small product table illustrates the conversion. Once validation returns a Table, its data field already exposes to_csv; downstream code does not need to repeat the model-output parsing. Markdown serialization through pandas requires its tabulate dependency.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reliability depends on useful feedback
The same approach can generate date ranges, relationships, knowledge graphs, DAGs, workflows and tables. The design work lies in choosing a useful response model and writing validators for it. As models improve, Liu expects those interfaces to require less intervention.
Liu reports that one retry is often enough with OpenAI and Anthropic models when validation errors are informative. He presents this as practical experience rather than a measured success rate. A good error message now serves two readers: the developer diagnosing the failure and the model attempting a correction. Faster models also make the extra round trip easier to afford; Liu refers to newer “3.5” and “4.0” models while discussing that latency tradeoff.
In consulting work involving complex validations, Liu reports that function-calling fine-tuning reduced failure rates from 4–5% to zero on models such as Mistral and GPT-3.5. The slide describes the result as near-zero; no evaluation size, model versions or held-out conditions are supplied. The observation supports testing fine-tuning when failures persist, but does not establish a general zero-failure guarantee.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate reports and plans for execution
Vision, RAG and agents all need types that the application can program against. Prompt construction, the machinery implementing a response model, and constrained sampling in systems such as LlamaCPP, Ollama or Outlines sit beneath that application contract. Their implementation can change while the programmer continues to consume useful objects.
That separation opens up two applications beyond a chat answer:
- Enterprise RAG: Generate reports that support decisions, rather than treating retrieval only as a question-answering feature.
- Agents: Generate workflows and directed acyclic graphs, then pass those plans to an execution engine that performs the computation.
A generated DAG is a plan, not an executed outcome. The application owns its execution instead of depending on an open-ended ReAct loop to terminate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep ownership of the program
The destination is familiar software engineering: types the IDE understands, field access it can check, and errors it can underline. Generative AI becomes a producer of data structures. Developers own the objects they define, the functions they implement and the control flow that connects them. They also retain the prompt through the messages array.
Typed boundaries make generated data usable by existing software. Liu describes this as making software 3.0 backward compatible with what developers already build. The model can remain powerful and flexible without making the surrounding program mysterious: its output enters a world of objects, validation and explicit control flow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
A structured-generation library for constraining language-model output formats.
Further reading
- Instructor 1.0 announcementArticle
Jason Liu explains the 1.0 client interface, typing improvements and validation support. Embedded examples have since been updated.
Build a DataFrame type that parses Markdown, serializes back to Markdown and supplies a model-facing JSON schema.
- PEP 593: Annotated typesDocumentation
The Python proposal defining metadata attached to type annotations through Annotated.
Define typed models and understand what Pydantic validation guarantees.
Updates since the talk
Current examples of incremental structured output, including limitations on validators during streaming.
Configure retries and handle errors in current Instructor applications.
Read the complete timestamped transcript
- 0:00
[upbeat music] So the context from last year's talk was Pydantic All You Need.
- 0:15
It was a very popular talk. You know, it kinda like kicked off my Twitter career. And, uh, today I'm coming back a year later to basically say the same thing again.
- 0:25
Uh, Pydantic is still all you need. And really my goal is to share with you sort of what I've learned for the past year. And, and, and the problem has always been the fact that if I had hired an intern to write an API for me, and that API returns a string that I have the JSON loads
- 0:39
into a dictionary, and then just pray that the data was still there to begin with, I would be pretty pissed off. I would probably just fire them, replace them with Devin, and just prompt it to use FastAPI and Pydantic.
- 0:51
Because, you know, I'm really tired of writing code like this, right? And this is the kind of code that we wrote when we had to work with things like ChatGPT-- uh, GPT-3, uh, and stuff like that.
- 0:59
But there's a lot of, uh, good tools that we have in the Python ecosystem, and the ecosystem in all of these languages, whether it's Ecto and Elixir, Active Record, or anything like that, that can make our lives, uh, much, much easier.
- 1:11
And so the problem is that by not having schemas and structured responses, we tend to lose compatibility, composability, and reliability when we build tools and write code that interact with external systems.
- 1:23
But it seems that we're very happy with using LLMs for the exact same reason. And so last year we mostly talked about how Pydantic and function calling was a great alternative for how we can use structured output to do a lot of additional benefits, right?
- 1:37
We, we, we are able to have nested objects and nested models for modular structures, and then we can also use validators to improve the reliability of the systems that we build.
- 1:45
And I'll talk about some of these examples. And so it's been about a year and a half. I think the big question is, what's new in Pydantic? What's new in the library?
- 1:54
And the answer is basically nothing. Um- [laughing] I'm basically coming back to say that I was right, and it feels really, really good. [laughing] It's still pip install instructor, right? And since then, we've rel-released 1.0.
- 2:07
Uh, we've launched in five languages, Python, dot-- uh, TypeScript, Ruby, Go, Elixir. We just built out a version in Rust. And again, it's mostly because this is just the exact six hundred lines of code that you do not want to write.
- 2:20
And at least in the Python library, we've seen, you know, forty percent growth month over month. And, you know, we've only had about two percent of the coverage of the OpenAI download.
- 2:28
So there's still tons of room to grow in terms of how we can, you know, make these APIs a lot more ergonomic. And so you saw 1.0, you know, you might be going like, "Jason, like, what did we break in the API?"
- 2:39
Uh, I renamed a method, and now we support things like O- Ollama and LlamaCPP, and along with a bunch of other APIs. So we, we support things like Anthropic, Cohere, Gemini, Groq, everything that you need.
- 2:52
And as long as language models support more function calling capabilities, this API will pretty much stay standard.
- 2:58
And if you haven't seen the talk last year, the general API looks like this. You define a Pydantic object. You can then, you know, patch the OpenAI client or any client that you want.
- 3:08
And all you have to do is you gotta pass in response model equals user, right? This is basically it. This is very similar to how FastAPI works. And, you know, it took a, a little bit of, of hackiness, but now we can also leverage some of the new Python tooling to also infer the return type.
- 3:23
And so here, because response model is a user, the object is inferred as a user object. You get nice red squiggly lines if you've messed up your code. The same thing happens when you wanna create an iterable.
- 3:34
Here you see that I have a single response model as a user, but I wanna extract two objects. And as long as you set stream equals true, you're gonna get each object as they return.
- 3:43
And this is kind of a way of using streaming to improve the latency while having a little bit more structured output.
- 3:50
We also have partials, right? The difference here is that instead of just returning a partially correct or validated JSON object, we can validate the entire object. And this means that if you have things like generative UI that use a structure, you can render that while streaming without having to write this like very evil like JSON stack code
- 4:07
to figure out how to render this in real time.
- 4:11
And so, yeah, nothing's really changed. You have one noun, which is the client, and you have three verbs. You can create, create_with_iterable, and create_with_partial based on whether or not you wanna use streaming.
- 4:20
And everything you-- else you think about is gonna be around the response model, the validators that you have to build, and the messages array that you pass in. So if, if OpenAI supports some new weird API call, as long as it fits within messages, there's not gonna be any break in code.
- 4:35
And that's why I think Pydantic is still all you need.
- 4:39
And so the rest of this talk is basically gonna be about, you know, two, really three things. I'm gonna cover some examples of generation, in particular, uh, around RAG and extraction.
- 4:49
Then I'm just gonna cover what we learned this year. And, and it's really not that much, right? Uh, validation errors are very important, and usually they can fix any errors that we have.
- 4:57
Uh, not all language models can really support retry logic right now. I think that's something we're gonna work towards. And ultimately, whether you use vision or text or RAG or agents, they all benefit from structured outputs, right?
- 5:09
Because the real idea here is we're gonna be programming with data structures, which is something everyone knows how to do, rather than trying to like beg and pray to the LLM gods.
- 5:17
And really, again, the theme of this talk is the fact that nothing really has changed. The language did not change. All we learned to do is relearn how to program.
- 5:27
And so the first concept that I think many people might not have seen in Pydantic is the validators, right? Here you can define a validator on any kind of attribute and add additional logic that tells you what correct looks like.
- 5:38
And so you see in my prompt, I don't really ask the language model to uppercase all the names, but I can actually write Python code to verify that something is correct and throw an error message.
- 5:48
And if I want to, I can turn retrying on, and that error message is caught by the language model and then used to correct the outputs. And so in this example, it is the error message that is part of the prompt, but conditionally added to the language model.
- 6:00
And as you can see, you know, after one API call, Jason is now all caps. Pretty nice.
- 6:06
We can also do model-level validation. This is a very simple example, you know, you, that you might see something like Ramp where you're processing receipts. You might want to use a vision language model to extract the receipt data.
- 6:18
There's a total cost, and the price is a list of products, and the validator does, does something a little bit more interesting. It says, "Make sure that the price and the quantity add up to the total cost."
- 6:27
Right? Again, this basically doesn't really happen for ninety-nine percent of the cases, but when it does happen, you see like a red bar in Datadog, and that's really what I care about.
- 6:36
And if I wanna ask re-asking, I wanna make sure that, again, everything's done correctly.
- 6:41
So let's jump into generation, right? Why should I use structured outputs? Well, it turns out if you don't use structured outputs, the structure you get is just response has a content string, right?
- 6:51
You still get an object back out, but you're just hoping that you don't have to call JSON loads yourself and, you know, eat, eat whatever cost, uh, you have in terms of, uh, parsing.
- 7:01
And so a really simple example of a RAG application is not only having a content, but having a list of follow-up questions, right? The follow-up questions can be informed by the existing context, but now you're gonna let the user feel like, hey, like there's other questions that you can answer based on the context that I've put in
- 7:15
the prompt. A really funny example that I've actually done in production quite a bit is just making sure that the links we return are valid. And so here I have a very simple validator.
- 7:26
I just have a regular expression parse all the URLs, and I use post to figure out if the URL returns a two hundred. And now I can make sure very easily that no URLs are, you know, hallucinated.
- 7:37
And in my instructions, I just say, "Well, if it's not real, just throw it out next time," right? "Don't try to-- Don't try too hard." The same thing happens with retrieval-augmented generation.
- 7:48
We all kind of know at this point that embeddings won't really solve all the problems you have in search, right? For example, if I ask the questions like, "What is the latest news from Z?"
- 7:56
Like, latest news isn't something that embeddings can capture, right? The source of that may, maybe that is relevant if you use BM25, but really there might be separate indices that we wanna query.
- 8:07
And we can use something very simple in the structured output world that makes this very reasonable, right? Here I can define a search object. I say it has a query, a start date, an end date that is optional.
- 8:18
Maybe there's a limit in case I wanna see the top five results. And then a source that allows the language model to choose which backend I wanna hit.
- 8:26
And then, you know, well, how you actually search the endpoint is kind of an implementation detail that we don't care about. And now you just define a very simple function, you know, create search.
- 8:34
It takes in a string, returns the object. And even the API call itself now is an implementation detail, right? As long as I get the search query out and it's correct, I can do a lot more.
- 8:43
And in particular, like even the validations themselves, you know, I can figure out whether or not the date ranges are zero days, one day, and, and figure out even distributions based on the structured output.
- 8:54
Then if I ask a question like, "What is the difference between X and Y?" I can just turn on interval mode. Now, if I ask this question, I'm gonna have a search question query for Y, a search query for X, and again, my RAG application can figure out that I can do two parallel search queries, collect them
- 9:09
together, and continue on. And so this means that you can build a fairly sophisticated RAG application in two functions and two models. First, you have the model for how you respond with the data and then how you process a search query, right?
- 9:22
As you can see here. And then you define two functions that return those objects.
- 9:29
And then this is basically your advanced RAG application, right? You make a search query, you return multiple searches, you search each one, and then you pass the context into the answer mo-- answer question function.
- 9:39
Right? This is very, very straightforward code. But what this means is you get to render something very structured, and then whether or not this endpoint is used by OpenAPI, is parsed by a React model, you know, again, these are all just implementation details.
- 9:50
The LLM is very hidden behind the, the type system that we can now guarantee to be correct.
- 9:57
And the last one I think is really interesting is this data extraction. Um, you know, if you wanna do something like labeling, it's really easy to just say, "Okay, class label is a literal of either spam or not spam."
- 10:07
You've built a classifier. If you want the accuracy to improve about fifteen percent, you can add chain of thought, right? And again, it's the structure that tells you how the language model works, but you still have good validation on whether or not you're gonna get, you know, spam or some, some bl-- like babble on like, you know,
- 10:21
here's the JSON that you care about. You can do the same thing for things like extracting like structured information out of transcripts. Like a very common example is people wanna process transcripts.
- 10:31
Now it's very structured, right? I have a, I have a classification in the meeting type. I've given myself a title, a list of action items, and a summary. Here, the owner is a string, but you could imagine having a validator that makes sure that the owners are the, at least one of the participants of the email based
- 10:46
on some Google Calendar integration. Uh, again, these are all implementation details. It's all up to you. And then lastly, you can do some really magical stuff. In this example, the type I've given is called table.
- 10:58
It has a caption string and a very weird data, markdown data frame type hint. And here what you can see is that I'm really just trying to extract images or tables out of an image.
- 11:08
But this is a bit wild. Like, don't worry if you don't understand it. But basically what we're using is we're using the new PIP, uh, PEP
- 11:16
basically to figure out how we can use annotations to create new type hints. And so this type hint is pretty advanced. It says that it is an instance of data frame, which means your IDE will now auto-complete all the data frame methods as you continue to program.
- 11:29
But the before validator says, "I know M markdown is gonna come out, but I wanna parse it to a data frame." The serializer says, "I know it's a data frame, but when I serialize it, I want it to be markdown."
- 11:39
And then lastly, you can a-add additional JSON schema information, which becomes the prompt that you would use to send to a language model. But the idea here is, you know, it's really just a type system that we've defined that can be used by a language model.
- 11:53
And then you can get pretty interesting outputs out of this, right? And because of the data frame, you can instantly call to CSV or something like that without worrying about other implementation details.
- 12:03
And so what we've seen is that we can now just generate things like date ranges, relationships. We can generate knowledge graphs that we've shown last year. And generally just think about DAGs and workflows and tables.
- 12:12
And again, all we really care about is just coming up with a creative response model, s-having a good set of validators. And as models get smarter, we're only gonna have to do less and less, right?
- 12:23
This is fairly bulletproof. And so for the last, like, five minutes, I really just wanna share what I've learned in the past year, right? The first thing is that often one retry for models like OpenAI and Anthropic are basically enough, and really all you care about is having good, well-written, informative error messages, which has been hard for
- 12:42
all time, but now you're more incentivized to build this out because this not only makes the code more readable to you, but to the language model. Then lastly, for the new models from, like, 3.5 and, you know, 4.0, they're so much faster now that we can actually eat the cost of latency for performance.
- 12:57
And so again, you know, as these models get far smarter and faster, you're still fairly bulletproof.
- 13:03
One thing I've noticed in a lot of, like, consulting that I've done is that we see, like, 4% to 5% failure modes in very complex validations, but just by fine-tuning language models on function calling, we can get them down to zero for even simple models like Mistral or GPT 5-- Sorry, GPT 3.5.
- 13:19
And lastly, structured output is here to stay, mostly because even in domains like vision or RAG or agents, really what I care about is defining the type system that I wanna program with on top of how I wanna use language models, right?
- 13:31
Prompting is an implementation detail. The response model is an implementation detail. And whether or not we use something like constrained sampling that's available in Llama CBP or Ollama or Outlines, again, the benefits I get as a programmer is sort of on a different level of abstraction.
- 13:46
And then even with things like RAG and agents,
- 13:49
right now we think of RAG as much more like question answering systems, but in larger enterprise s-situ-situations, I see a lot of report generation as a step to make, you know, better decision-making, right?
- 14:00
In agents, a lot of it now becomes generating workflows and DAGs to then go send to an execution engine to do the computation ourselves rather than having some kind of react loop and hope that these things terminate.
- 14:12
And so really there's no new abstractions, right? Everything that we've done today is just reducing language models back to very classical programming, right? What I care about is that my ID understands the types, and we just get red squiggly lines when things are unhappy.
- 14:25
And what we've done is we've turned generative AI just to becoming generating data structures, right? You can now own the objects you define. You own the functions that you implement.
- 14:34
You own the control flow, and most importantly, you own the prompt because we just give you this messages array, and you can do anything that you want.
- 14:41
And what this means to me, and I think what this means to everyone else here, is that we are actually turning software 3.0 and making it backwards compatible with existing software, right?
- 14:50
We're allowing ourselves to demystify the language models and go back to a much more classical structure of how we program.
- 14:57
And that's why I still think Pydantic is basically all we need. Thank you. [audience applauding] [upbeat music]