# No more bad outputs with structured generation

Rémi Louf, .txt (Outlines)

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/aNmfvN6S_n4-no-more-bad-outputs-structured-generation) · [Video](https://www.youtube.com/watch?v=aNmfvN6S_n4)

## Making model outputs dependable with structured generation

A flight-extraction failure leads into Outlines, token masking, and the ways output constraints can improve validity, reduce generation work, and change task performance.

Familiarity with Python, JSON, and the idea that language models generate text one token at a time will help.

### A JSON error is an interface failure

[0:00](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=0s)

Suppose you want to extract flight details from emails. The next piece of software expects a JSON object containing an origin, a destination, and other flight information. You prompt the model carefully, insist on JSON, and try function calling—yet sometimes the result still produces a JSON decoding error. This is the opening problem Rémi Louf, co-author and co-maintainer of [Outlines](https://github.com/dottxt-ai/outlines) and co-founder and CEO of .txt, brings to structured generation.

![Black slide displaying json.decoder.JSONDecodeError above an ASCII-art cow.](https://ai.engineer/api/talk-essay-media/0b849f8d9cec9fa25062187d3c14b5a1db09fe91b6226af6549d33ffaf27f45a)

A JSON decoding error illustrates the opening failure case.

**Output structure is part of a software interface.** Modular systems work because one component can rely on another component’s contract. If a model sometimes returns an object and sometimes returns something the caller cannot parse, every downstream consumer inherits that uncertainty. Louf connects this failure to the difficulty of building dependable agents: the concern is not merely an untidy answer, but an unreliable boundary between components.

Structured generation moves the required format into the generation process itself. Instead of asking the model to respect an interface and checking afterward, the decoder restricts what the model can emit. That directly addresses output structure; whether it also improves the answer is a separate question, which the later experiments test.

### Turn an extraction pattern into a generation constraint

[2:04](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=124s)

Outlines is a Python library intended to fit into an existing workflow, rather than a framework that requires the workflow to fit around it. At the time of the talk, Louf reports that vLLM and TGI use Outlines underneath their function-calling support. He also credits a contributor community of roughly 87–88 people for building the library.

The basic workflow has three steps:

1. **Choose and instantiate a model.** The demonstration focuses on open models. Louf reports six provider integrations, including Transformers, llama.cpp, and the recently added MLX-LM; the OpenAI integration is used primarily for comparisons.
2. **Create a generator.** In the introductory example, `generate.text` creates a text generator.
3. **Call it with a prompt.** The prompt asks for the benefits of structured generation in one sentence, with a period used as the stopping condition.

The recording uses the historical `generate.*` API. Current Outlines documentation uses a different model/output-type interface, so the API names here describe the demonstrated workflow rather than a current installation recipe.

Now ask for the IP address of Google’s public DNS servers. Unconstrained generation can surround the address with a long explanation. An application might then use a regular expression to recover the address from that text. Outlines lets you move that same kind of pattern earlier: give it to `generate.regex`, then call the resulting generator with the question. **The pattern defines the allowed answer, rather than searching an answer after generation.** Louf’s demonstration uses **Mistral 7B v0.1** and returns the correct address directly.

### Specify the structure inside the fields

[4:59](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=299s)

Regular expressions are one way to describe an output; JSON Schema and Pydantic models are another. Returning to flight extraction, a schema can require an object with `origin` and `destination` fields. But declaring both fields as arbitrary strings leaves useful information out of the contract. Louf instead uses a custom airport-code type: each value must contain three uppercase letters.

This distinction is easy to express in a Python schema definition. The following teaching example encodes the field constraint described in the talk; it is not a transcription of the custom Outlines type:

```python
flight_schema = {
    "type": "object",
    "properties": {
        "origin": {
            "type": "string",
            "pattern": "^[A-Z]{3}$",
        },
        "destination": {
            "type": "string",
            "pattern": "^[A-Z]{3}$",
        },
    },
    "required": ["origin", "destination"],
    "additionalProperties": False,
}
```

Three uppercase letters are a stronger contract than a string, but the pattern alone does not establish that the value names a real airport. The practical principle is to express the structure you actually know, while keeping format validity distinct from factual correctness.

The same schema-first workflow extends to vision. In a recently merged capability, Louf supplies a photograph of a dish, specifies the expected JSON, instantiates a generator, and passes both the image and the prompt. The demonstration returns valid JSON. He closes the usage introduction with the installation command `pip install outlines`.

### Mask invalid continuations before sampling

[6:28](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=388s)

A language model does not directly return text. Given a prompt represented as token IDs, it produces **logits**: scores from which next-token probabilities are derived. Generation then processes those scores and samples a token. Temperature changes the distribution; top-k and top-p restrict which candidates remain available. The sampled token is appended to the context, and the model runs again.

Structured generation intervenes between the model’s scores and token selection. For each candidate token, the constraint processor asks whether appending it to the current output would violate the required structure. If it would, the processor masks that token so it cannot be sampled. The remaining candidates still compete according to the model and sampling configuration; the constraint determines which continuations are admissible, not which admissible answer is true.

For a concrete illustration, suppose generation is inside the `origin` string from the flight schema and has already emitted `SF`. A candidate token whose decoded text is `O` preserves the three-uppercase-letter pattern. A candidate `o` violates it, and a closing quote would end the field too early. The latter two candidates must be masked. These are illustrative token texts, not a claim about a particular tokenizer’s vocabulary.

Candidate continuations inside an airport code
Filtering changes eligibility; no next token has been sampled yet.
Constructed example: The prefix SF and candidate texts O, o, and a closing quote are invented teaching values. They do not assert a particular tokenizer vocabulary or reproduce a recorded token trace.
Current output prefix — unchanged: {"origin":"SF
Operation: Apply the three-uppercase-letter constraint to candidate token texts before sampling.
- Output prefix
  - Before: Before constraint filtering: {"origin":"SF
  - After: After constraint filtering: {"origin":"SF
- Candidate token text: O
  - Before: Before constraint filtering: Candidate for sampling
  - After: After constraint filtering: Allowed: completes three uppercase letters
- Candidate token text: o
  - Before: Before constraint filtering: Candidate for sampling
  - After: After constraint filtering: Masked: lowercase violates the pattern
- Candidate token text: "
  - Before: Before constraint filtering: Candidate for sampling
  - After: After constraint filtering: Masked: would close a two-letter code
Filtering changes eligibility; no next token has been sampled yet.

The difficult engineering problem is making those decisions efficiently across a model’s vocabulary at every step. Louf presents that efficiency as .txt’s distinguishing contribution relative to other structured-generation libraries, including Guidance and LMQL. The simple rule—exclude invalid continuations—does not by itself explain how to apply it cheaply.

### Structure reaches beyond JSON

[7:43](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=463s)

Even text that looks like ordinary reasoning can contain a useful formal structure. Louf uses a GSM8K math example: a question marker, question text ending in a question mark, and arithmetic within the solution. Arithmetic expressions can be described with a **context-free grammar**. That lets the constraint extend beyond a JSON envelope into the shape of the reasoning text and its final answer—36 in the illustrated problem.

The most direct benefit remains valid output. Prompting tricks attempt to persuade a model to emit JSON; constrained sampling limits generation to the specified structure. Louf illustrates the difference with a Predibase experiment using Mistral 7B v0.1 on a version of CoNLL modified to require JSON output.

| Generation method | Reported valid JSON |
|---|---:|
| Unconstrained Mistral 7B v0.1 | 17% |
| With structured generation | 99.9% |

Louf says the constrained result used no prompt optimization. These are reported JSON-validity rates for that experiment, not extraction-accuracy scores or a universal success rate for every schema.

### Separate constraint overhead from generation work

[9:26](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=566s)

Checking constraints adds work, but when that work occurs matters. Louf shows a comparison with Guidance in which overhead is plotted against generated-token count. In the displayed comparison, he describes Outlines’ incremental inference overhead as approximately zero, with an upfront compilation cost. His suggestion that the difference could be imperceptible even in a Groq integration is a prospective claim, not a reported Groq deployment measurement.

Constraints can also reduce how much generation is necessary, through two different mechanisms. First, some output text is already determined by the structure. If the next part of a JSON object must be a bracket or a known field name, there may be no decision for the model to make. In Louf’s toy JSON example, only five of ten tokens require generation. Skipping model calls along deterministic paths is an additional optimization, not an automatic consequence of masking invalid tokens; it also requires care about tokenization and subsequent probabilities.

Second, a narrow output format eliminates unnecessary prose. Louf revisits the Google DNS question: in his comparison, ChatGPT produces 50 tokens, while constrained generation produces eight. This saving comes from returning just the requested value. It is distinct from skipping fixed JSON text, and neither token-count illustration establishes a universal wall-clock speedup.

### What are prompt examples teaching?

[11:03](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=663s)

The next experiment measures GSM8K accuracy for Mistral 7B v0.1 while varying the number of examples in the prompt. With unstructured generation, one-shot accuracy is worse than eight-shot accuracy. With structured generation, Louf reports one-shot and eight-shot accuracy in the same ballpark. The talk does not give exact accuracy values for this comparison.

That result suggests a possible role for demonstrations beyond teaching the task itself: some of their value may come from teaching the model the required output structure. If the decoder already supplies that structure, fewer examples may suffice. Louf explicitly leaves this as a direction for further investigation, rather than a general explanation of how in-context learning works.

### A task-specific gain for Phi-3 Medium

[12:10](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=730s)

The function-calling experiment asks whether constraints can improve task accuracy, not just parseability. Louf evaluates Microsoft’s **Phi-3 Medium** on the Berkeley Function Calling Leaderboard’s simple function benchmark. He reports the following results, comparing the constrained model with the best **GPT-4** version in his comparison:

| Model and generation method | Reported accuracy in the talk |
|---|---:|
| Phi-3 Medium, unconstrained | 86% |
| Phi-3 Medium, structured | 96.5% |
| GPT-4, version unspecified | 93.5% |

These are the figures spoken and shown in the recording. The June 17, 2024 companion report, [Beating GPT-4 with Open Source](https://blog.dottxt.ai/oss-v-gpt4.html), distinguishes results between runs and reports different final figures: 96.25% for `Phi-3-medium-4k-instruct` and 94.5% for `GPT-4-0125-Preview` on 400 BFCL simple examples. The recording’s chart should therefore be read as its own reported comparison, not silently replaced with the later figures.

![Three outlined bars labeled Phi-3-medium at 86%, GPT-4 at 93.5%, and Phi-3-medium with a document icon at 96.5%; axes read Accuracy and BFCL simple function.](https://ai.engineer/api/talk-essay-media/841341ba3ac46190b1d1d3d7c99005b9005a7c895023e70a8e576890df24e198)

BFCL simple function accuracy: 86%, 93.5%, and 96.5%.

The improvement is reported without fine-tuning. Its significance is specific but useful: changing the decoding constraints can extract substantially more task performance from an existing open model. It does not establish that Phi-3 Medium is generally more capable than GPT-4. Louf’s broader optimism about open models follows from this possibility of improving their useful behavior without changing their weights.

### From valid syntax to usable outputs

[13:19](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=799s)

Louf closes by moving beyond the earlier regular-expression work, which he describes as roughly a year old. The team has extended its approach to context-free grammars, with applications including code, protein structure, and the arithmetic used in the GSM8K example. His negligible-overhead claim here concerns experimental grammar generation: the contemporaneous research describes an in-house implementation, distinct from the public Outlines CFG implementation.

The next boundary is **semantic constraints**. Text-to-SQL makes the distinction concrete: a model can produce valid SQL syntax while referring to a table or column that does not exist. Restricting identifiers to the database’s available tables and columns addresses a failure that a syntax grammar alone cannot prevent. Louf says the team’s internal work can guarantee that a query will run, while explicitly declining to guarantee that it answers the intended question. Executability and correctness remain different contracts.

A further efficiency direction moves constraints into the model architecture. Masking logits happens after the model has already computed scores, including scores for tokens that will be discarded. Preventing unnecessary computation earlier could save additional work. Louf presents this as ongoing research, with implementation details and measured gains still to come in an anticipated blog post.

For applications outside open-ended chat, Louf predicts that structured generation will become a normal part of using language models. The flight-email example explains the practical pull: once an output must cross into another software component, its structure becomes part of what that component needs to trust. He ends by directing the audience to supporting blog posts through the closing QR code.

![Large pink text on black reads: You will be using structured generation.](https://ai.engineer/api/talk-essay-media/dfa7d22b9eb6a408785d33e253f2dfd5374cfd08923ae6705829fcf282bace06)

“You will be using structured generation.”

## Resources

- [Outlines](https://github.com/dottxt-ai/outlines) — Python library for constrained outputs, with installation instructions and current typed-generation examples.
- [Beating GPT-4 with Open Source](https://blog.dottxt.ai/oss-v-gpt4.html) — The .txt team's report on Phi-3 Medium and GPT-4 evaluated on BFCL's 400-example simple function-calling subset.
- [Prompt Efficiency](https://blog.dottxt.ai/prompt-efficiency.html) — Experiments examining how structured generation changes few-shot performance on GSM8K and GPQA.
- [Efficient Guided Generation for Large Language Models](https://arxiv.org/abs/2307.09702) — Willard and Louf's formulation of efficient constrained generation through finite-state transitions and vocabulary indexing.
- [How fast can grammar-structured generation be?](https://blog.dottxt.ai/how-fast-cfg.html) — An experimental CFG implementation illustrated with SQL generation and constraint-processing measurements.
- [Coalescence: making LLM inference 5x faster](https://blog.dottxt.ai/coalescence.html) — How deterministic output structure can eliminate model calls, including the consequences of tokenization choices.
- [How guidance works in TGI](https://huggingface.co/docs/text-generation-inference/en/conceptual/guidance) — TGI's documented grammar compilation, token masking and tool-generation workflow.

## Slides

### Presenting sponsor

Microsoft is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=5s) [Microsoft presenting sponsor slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/a94f6720bf4d74cca7f89072db514d439a008de3faab534dcf27c3ff10b18c98.webp)

PRESENTING SPONSOR

Microsoft

### Innovation and platinum sponsors

[0:10](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=10s) [AWS innovation sponsor and MongoDB, Google Cloud, and Neo4j platinum sponsors.](https://assets.ai.engineer/sites/aiecode2025/sha256/4f124370b2308fdace554f9723c6e7efe4ccf592143cd6b1c57035dc1da6079e.webp)

INNOVATION SPONSOR
aws

PLATINUM SPONSORS
MongoDB®
Google Cloud
neo4j

### Rémi Louf

Rémi Louf introduces himself as a co-author and co-maintainer of the open-source library Outlines.

[0:22](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=22s) [Rémi Louf’s name beside a pixelated pattern.](https://assets.ai.engineer/sites/aiecode2025/sha256/a6708d3de4f842b525d41a72003db30c1080bcf6d67bc020b3731da12b672b7d.webp)

Rémi Louf
@remilouf

### Parsing an LLM response as JSON

Louf parses an LLM response in an example that extracts flight details from emails. The response can sometimes produce a JSON decode error.

[0:34](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=34s) [A code window showing a JSON parsing statement.](https://assets.ai.engineer/sites/aiecode2025/sha256/496c712972ab504b86c3af3aa52a4107c6447e0fa2742a505fcb70aad2550234.webp)

result = json.loads(llm_json_answer)

Interface text:
Page 4 of 40

### JSON decode error

A flight-information extraction prompt can produce a JSON decode error, illustrating the speaker’s concern that inconsistent LLM outputs make interfaces harder to trust.

[1:01](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=61s) [A JSONDecodeError message above an ASCII cow.](https://assets.ai.engineer/sites/aiecode2025/sha256/530bd0e20845f411250f08951598b1ab200eea7b7c89053b9148348508543340.webp)

 ______________________________
< json.decoder.JSONDecodeError >
 ------------------------------
       \   ^__^
        \  (oo)/______
           (__)/      )\/\
               ||---w |
               ||     ||

### A broken connection

A red error symbol breaks the connection between documents and a computer. The speaker argues that inconsistent LLM outputs undermine the trusted interfaces needed to build complex systems.

[1:05](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=65s) [Pixel-art documents and a computer separated by a red error symbol.](https://assets.ai.engineer/sites/aiecode2025/sha256/a5c8b23f5853e451ebeefa5d0580c49db02a35be6fb34e325980d8c6927521cd.webp)

### Structured generation fixes this

The speaker presents structured generation as a way to guide model outputs into a specific structure, and says it can beat GPT-4 as a byproduct.

[1:36](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=96s) [Text says structured generation fixes this and beats GPT-4 as a byproduct.](https://assets.ai.engineer/sites/aiecode2025/sha256/f9d5989d4a21db3f2b5ca3a207537d05903568ef435719f5f18c9b5f03d3af10.webp)

Structured generation
fixes this (and beats GPT-4 as a byproduct)

### Goals for today

The talk introduces the open-source Outlines library, briefly explains how it works, makes the case for using it in workflows, and looks at its near future.

[1:46](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=106s) [Agenda: introduce Outlines, explain how it works, discuss why to use it, and look at the near future.](https://assets.ai.engineer/sites/aiecode2025/sha256/4f8b6a9a4072198682bccff1d39f42df10a9bb9c671efe94370b042039a1e13b.webp)

Goals for today
Introduce Outlines
How does it work?
Why should I use it?
The (near) future

### Outlines is a library

Outlines is a Python library that fits into existing workflows. The speaker says vLLM and TGI use it under the hood for function calling.

[2:10](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=130s) [Outlines is a library, alongside several project logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/0c5837efa322513f65ccdb7f23aa714bbb8522a1ef59343be3684a273050f8a0.webp)

Outlines is a library

vLLM
SGL

### 87 contributors

The Outlines co-author thanks its contributors, pictured in a grid of profile images. The slide counts 87; he says there may be 88.

[2:34](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=154s) [A slide thanking 87 contributors, illustrated with a grid of profile images.](https://assets.ai.engineer/sites/aiecode2025/sha256/514ba16b6d6df7669b2754ed649f29173d75f3eacceb5792bceb1e563581757c.webp)

87 contributors

### 77k downloads per week

Outlines reports 77k downloads per week. Its GitHub star history rises sharply before October, then continues upward past 6,000 stars.

[2:50](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=170s) [A 77k downloads per week headline above a chart showing rising GitHub stars.](https://assets.ai.engineer/sites/aiecode2025/sha256/dbe07acaa0ab5254f09551d06c40f936db2f151a71affa4804748292f416eff2.webp)

77k downloads / week

Star History
Legend: outlines-dev/outlines
Y-axis: GitHub Stars
Y-axis ticks: 1.0k, 2.0k, 3.0k, 4.0k, 5.0k, 6.0k
X-axis: Date
Timeline: April, July, October, 2024, April
star-history.com

### Choose your model

Outlines starts text generation by instantiating a model. The slide lists six open-source integrations and an OpenAI integration used to compare results; the model can be changed later.

[3:12](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=192s) [Code examples show how to choose among several model integrations in Outlines.](https://assets.ai.engineer/sites/aiecode2025/sha256/ebbf058b6a06782089fef85cf0da45330a098b4f6ab4e745382cddaac47a7999.webp)

Choose your model

from outlines import models

# Open Source
model = models.transformers("meta-llama/Meta-Llama-3-8B")
model = models.llamacpp("TheBloke/Mistral-7B-Instruct-v0.1-GGUF", "mistral-7b-instruct-v0.1.Q5_0.gguf")
model = models.mlxlm("mlx-community/phi-2-hf-4bit-mlx")
model = models.vllm("meta-llama/Llama-2-7b-hf")
model = models.exl2("path-to-model")
model = models.mamba("state-spaces/mamba-2.8b")

# APIs
model = models.openai("gpt-3.5")

(you can easily change later)

### Generate text

The example creates a text generator that stops at a period, then prompts it to describe structured generation in one sentence.

[3:48](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=228s) [Code creates a text generator and prompts it for one sentence.](https://assets.ai.engineer/sites/aiecode2025/sha256/b19be1b6780deaecc8495546c1f563b2e5a7a9cf4fdba3a6ee609953bcfcf337.webp)

Generate text

from outlines import generate, models

model = models.transformers("meta-llama/Meta-Llama-3-8B")
generator = generate.text(model, stop_at=".")

result = generator("Describe the benefits of structured generation in one sentence.")
print(result)
# You'll have to wait for 10 more minutes

### Install Outlines

Install Outlines for structured generation with pip install outlines.

[6:17](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=377s) [The command pip install outlines appears on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/5ca2b07a5c018eb2bcc8abd2763901f25b5b9300fb4bad7d50d247799eb5ddec.webp)

pip install outlines

### How does it work?

The speaker begins explaining how Outlines works, starting with the logits a model produces from a prompt.

[6:23](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=383s) [The question “How does it work?” appears on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/0b46d37c5d2f4e79acb2bd6442fc513eca79e192b04d3ad5f832592ae23d0d72.webp)

How does it work?

### Logit processing and sampling

The LLM produces logits, which .txt’s logit processor biases by masking tokens that would violate the desired structure. A sampler then selects the next token.

[7:01](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=421s) [A prompt enters an LLM; its logits pass through a logit processor and sampler to produce a token.](https://assets.ai.engineer/sites/aiecode2025/sha256/5f1782db4dfe13e1889c5516c369889b9aaddee9500aacb57462da8c34c39d17.webp)

Prompt
LLM
logits
logit processor
biased logits
sampler
token

Legend:
Model providers
.txt

### Use structured generation

The speaker argues for structured generation, starting from the observation that most text is structured.

[7:43](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=463s) [The slide states, “There is no reason not to use structured generation.”](https://assets.ai.engineer/sites/aiecode2025/sha256/e908e08168899505ffede646cef0b8d18af3f7a186b888ffec068844a9c1fcc1.webp)

There is no reason not to use structured generation

### Structured arithmetic example

A tree-planting problem separates free text from an arithmetic expression. The Outlines example uses stop markers, an arithmetic grammar and a regex to generate the parts.

[8:02](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=482s) [A tree-planting word problem is paired with a structured answer template and generation code.](https://assets.ai.engineer/sites/aiecode2025/sha256/df6710f158990e2d315cdeb05595c1e4cd9eb722fc203d5e10b02767509a8b4f.webp)

Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?
A: There are 15 trees originally. Then there were 21 trees after some were planted. So there must have been <<21-15=6>>.
The answer is 6.

Q: [text until question mark]?
A: [text until <<] <<[arithmetic operation]>>.
The answer is [result of operation].

from outlines import generate, grammars

model = ...
gen_question = generate.text(model, stop_at="?")
gen_reasoning = generate.text(model, stop_at="<<")
gen_arithmetic = generate.cfg(model, grammars.arithmetic)
gen_answer = generate.regex(model, r"[1-9][0-9]*")

prompt = "Q: "
prompt += gen_question(prompt) + "?\n"
prompt += "A: "
prompt += gen_reasoning(prompt) + "<<"
prompt += gen_arithmetic(prompt) + ">>.\n"
prompt += "The answer is "
result = gen_answer(prompt)

### Valid structure

Structured generation produces valid structure, a benefit for the structured text the speaker says is common.

[8:36](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=516s) [Most text is structured; structured generation yields valid structure.](https://assets.ai.engineer/sites/aiecode2025/sha256/68fc359bac7f75ca09ff42aab311b80c6f6aeb5c6aa4c0763fff09f53e2764a9.webp)

Most text is structured
You get valid structure

### Sample the desired structure

The meme contrasts sampling the desired structure with prompt hints, repair chains, and threats used to coax valid JSON. The speaker says those workarounds do not guarantee valid output.

[8:45](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=525s) [A meme contrasts directly sampling the desired structure with elaborate attempts to force valid JSON.](https://assets.ai.engineer/sites/aiecode2025/sha256/4f0ea9b815ee7012a52726747e772b94eb372c7631590dd9c9a8e38278cad32d.webp)

this meme made by
structured generation
gang

just sample what you want

sample arbitrary text

oh no it's not the right format

add prompt hints

install langchain and add a 7 call chain of repaired self consistency pipeline you saw on arxiv

ONLY OUTPUT JSON OR YOU WILL BE FIRED

@voooooogel

### Valid JSON comparison

In Prettybase’s experiment, Mistral-7B-v0.1 produced valid JSON 16.8% of the time. With structured generation, that rose to 99.9% without prompt optimization.

[8:57](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=537s) [A bar chart shows valid JSON rates of 16.8% and 99.9%.](https://assets.ai.engineer/sites/aiecode2025/sha256/ae3e39c7161dd8a8e27b3006655d1d6273ec8584b2d3c8fe2c739d8243176950.webp)

Y-axis: %valid JSON

Mistral-7B-v0.1
16.8%

Mistral-7B-v0.1
99.9%

### Benefits of structured generation

Structured generation provides valid structure with negligible overhead; the speaker says it need not affect inference time.

[9:26](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=566s) [The slide lists structured text, valid structure, and negligible overhead.](https://assets.ai.engineer/sites/aiecode2025/sha256/3e8da6793d1d4d0593670253b3d2807d1a10b2f39725cb23e40d4c6451d37af4.webp)

Most text is structured
You get valid structure
It adds negligible overhead

### Structured generation overhead

Guidance overhead rises steeply as generated tokens increase, while Outlines stays near zero. The speaker notes a compilation-time trade-off for Outlines.

[9:36](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=576s) [Line chart comparing generation overhead: Guidance rises sharply while Outlines stays near zero.](https://assets.ai.engineer/sites/aiecode2025/sha256/fe07cfde4f7bb1947fc29599f7703ccbfd1a59b8837fa4edbb690862ff176a56.webp)

Y-axis: overhead (s)
Y-axis ticks: 0, 120
X-axis: # Generated tokens
Legend: guidance, outlines

### Generating fewer JSON tokens

The JSON structure and field names are known in advance, so the model only needs to generate 5 of the 10 tokens in this example.

[10:11](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=611s) [A JSON example highlights generated values and states that five of ten tokens need to be generated.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a74059d2f73fb2304f4f3a83ee229e43602211ab8ecf3176f98eacc6ed8d087.webp)

{
    "name": "Remi Louf",
    "nationality": "French"
}

5/10 tokens need to be generated

## Full transcript

[0:00](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=0s) [on hold music] So yeah, my name's Rémi. I'm the co-author and co-maintainer of the open source library Outlines, uh, which some of you might know, and I'm also the CEO and co-founder of .txt or .text,

[0:25](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=25s) uh, whichever you prefer. Uh, we're more traditional machine learning people, and the motivation for work is the very simple observation that large language models are fundamentally flawed. I'll give you a very simple example.

[0:39](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=39s) Um, you're trying to extract flight information from a bunch of emails. Uh, of course, you want them to be, you know, a J-JSON object, um, you know, with origin, destination, et cetera.

[0:51](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=51s) So you go to OpenAI, you prompt the model to death, you threaten it, uh, you use function calling, and what you get sometimes as an answer is JSON decode error.

[1:01](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=61s) Um, I gave you very simple examples, but this has, like, very fundamental implications because computing rests on interfaces. We're able to build modular infrastructures and very complex infrastructure because we can trust the API over the pieces of code.

[1:17](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=77s) And here with LLM, and what you've probably witnessed, uh, you can't actually trust large language model to return consistent outputs. And, you know, in short is that the technology for agents is currently not there.

[1:30](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=90s) Um, so the good news is that structured generation, which is the ability of guiding the model to return to the specific structure, actually solves-- We'll see is that it allows you to beat GPT-4 sort of a byproduct.

[1:47](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=107s) The goals for today are first to introduce the open source library Outlines for those of you who don't know about it, then very briefly explain how it works, I won't get into the technical details, and then try to convince you that you should use it today, uh, for, you know, most of the workflows, uh, that you have

[2:04](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=124s) to deal with and sort of a very short glimpse into the near future. So Outlines is a Python library, emphasis on library. Uh, you can actually include Outlines in your workflow, and it's unlike frameworks where you have to make your workflow, you know, fit inside a framework.

[2:21](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=141s) Um, I think as a result, uh, it's been adopted by vLLM and TGI, uh, in the serving frameworks. And if you use function calling in either of these libraries, you're actually using Outlines under the hood-- uh, Outlines under the hood.

[2:36](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=156s) So I'm co-author, uh, but Outlines would be nothing without its contributors. Today, it's eighty-[REDACTED:generic_id]. Uh, I think it might be eighty-eight. I think I merged a PR this morning, I don't remember.

[2:45](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=165s) And so Outlines would be nothing with all-- without all these people, and I thank them-- uh, thank them a lot. Um, people thought we were crazy about a year ago when we were talking about structured generation.

[2:57](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=177s) Uh, but since then, uh, pretty happy because it looks like people are sort of caught up with the topic and realized that you can actually, you know, you can actually, uh, do structured output.

[3:09](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=189s) Um, so just now, just to run through-- quick run through Outlines. Um, so usually generating text happens in three stages. Uh, the first stage is that you need to choose the model and instantiate it.

[3:19](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=199s) So Outlines is purely focused on open source models. Uh, we have integration with six different model providers, uh, Transformers, llama.cpp, and also, uh, recently we added MLX, uh, MLX-LM.

[3:31](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=211s) Um, we have an integration with OpenAI, but that's mostly for us to compare the results that we get with open models with the results that are given by OpenAI.

[3:41](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=221s) The second step is to, I mean, generate text. What you do is that you instantiate a generator using generate.text. Here we just wanna, you know, return a single sentence.

[3:51](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=231s) So we're telling the generator, "Stop whenever you encounter a period." And question is described, then you call the generator, uh, with your prompt. Here is, "Describe the benefits of structured generation in one sentence."

[4:03](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=243s) And you'll have to wait for ten more minutes, uh, hopefully less. Okay, now we get into structured generation. So with Outlines-- without Outlines, if you ask what is the IP address of the public Google DNS servers, and you just generate text, you just let the LLM do its thing, then generally it will yap for a long time,

[4:24](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=264s) uh, you know, hundred tokens, five hundred tokens, and the answer will be somewhere in there. And the way you extract the answer is using regular expressions generally. Here, what you can do with Outlines is actually taking that regular expression that you use-- you would use to extract the answer and use it to guide the model, to tell

[4:42](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=282s) the model, "This is the structure that the output should follow." And as you see, you kind of remove the yapping, you print the-- you just call generate.regex, call the generator, and what you get is just the result, and it's actually the correct answer.

[4:55](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=295s) Uh, that was with Mistral, uh, [REDACTED:generic_id] BV [REDACTED:generic_id].

[4:59](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=299s) Regular expressions are not the only way to define structure. Uh, something that people need a lot in practice is a JSON. And Outlines allow you to generate, um, to generate text that,

[5:12](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=312s) you know, is a JSON object with a given structure. The way you specify the structure is using JSON Schema, or you can pass Pydantic models as well. Um, now you might notice on the flight information, so here we're-- you know, it's the example that I used at the beginning.

[5:27](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=327s) You're extracting flight information from an email. I could have used string as a type for origin and destination, but I did not. I used actually a custom type that we implemented in Outlines, and the reason is that origin and destination have way more structure than just text.

[5:41](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=341s) It's actually, you know, it's, it's an airport code that has three letters that's capitalized, and you can actually specify more and more structure, ev-- all the structure that you have in your problem, basically.

[5:52](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=352s) Uh, you can use this with vision models. Uh, that's something that we merged recently. So here we took, um, I think it's a picture from Wikipedia, uh, of a dish.

[6:02](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=362s) Uh, we

[6:03](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=363s) Tell the model what is the JSON that we expect as a, as an, as an output, and then we instantiate the generator and then pass the image and the prompt to the generator, and we get valid JSON.

[6:16](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=376s) Um, if you want to install Outlines, uh, and you think you could benefit from structured generation, then it's very simple. Just pip install outlines. Now, I'm gonna try to very quickly explain how it works.

[6:28](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=388s) Um, so models themselves, uh, what Mistral and Cohere, this one, are doing, uh, is actually training model weights. Uh, what a model does is, uh,

[6:38](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=398s) you input a prompt, you send a prompt, it's like token IDs, and what you get as an output is not text. It's logits. It's a probability distribution over the next token.

[6:46](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=406s) Now, what happens after that, when you want to generate text, the first step is that you have a logit processor that biases the logits. You probably use this every day actually without noticing it.

[6:56](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=416s) When you use temperature or when you use top-k, top-p sampling, you're actually biasing the logits. And once you have your biased logits, you use a sampling algorithm, then you get a token.

[7:04](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=424s) And once you have your token, you add it to the prompt and then feed it back to the LLM.

[7:09](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=429s) And where we fit is here. We actually wire the model...

[7:15](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=435s) Whenever the model generates logits, we look at every token and we say, "If I add this token to the current generation, is it gonna violate the structure?" If the answer is yes, we, we, like, we mask it so that it doesn't get generated.

[7:30](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=450s) Now, that story is very simple. What is really hard is doing that efficiently, and that's what we figured out at .txt, and that's what makes us different from the other libraries like Guidance or LMQL that, um, do structured generation.

[7:43](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=463s) And now I'm going to convince you, uh, that there's absolutely no reason to not use... sorry for the double negation here, to not use structured generation. Uh, the first reason is that most text is structured.

[7:58](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=478s) Um, I talked to you about JSON earlier. We talked about regular expressions. But here I just took the GSM8K dataset. Um, if you look at a-- If you're not me and don't see structure everywhere, um, what I see immediately, if you look at the right, uh, you can actually see that it's highly structured.

[8:15](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=495s) It's always Q, uh, period, text until a question mark, then et cetera, so on and so forth. Arithmetic operation, which is defined by a context-free grammar, and you could actually express this in Outlines and just get the answer at the end, which is, you know, thirty-six.

[8:30](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=510s) So there's a lot of structured text out there, not just, uh... [phone ringing]

[8:35](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=515s) Thank you. [laughing] I'll, I'll be, I'll be quick. Um, [laughs] of course, the second benefit is that, uh, you get valid structure. I mean, that's an obvious one. That's what we're doing it.

[8:46](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=526s) Uh, I like this meme, uh, at the bottom. This is what people are currently doing. Uh, it's just crazy stuff to get valid JSON as an output, and it's not even guaranteed.

[8:54](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=534s) And here with Outlines, you just sample what you want. It's as simple as this. And as an experiment, it's actually an experiment that Prettybase did. Uh, they took Mistral [REDACTED:generic_id].

[9:04](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=544s) They used a version of CoNNL that they modified so that it gives structured output JSON. What they found is Mistral [REDACTED:generic_id] only gets valid JSON, uh, s- seventeen percent of the time.

[9:14](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=554s) When you add structured generation on top of it, you get ninety-nine point nine percent, and that's without optimizing the prompt. So you can actually get, you know, you can actually get better than this.

[9:26](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=566s) The nice thing is that it also adds negligible overhead. So you actually have... You know, you don't have to fear for that affecting inference time, uh, which is the highly, you know, highly nontrivial thing.

[9:36](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=576s) Uh, here we compared, uh, the overhead introduced by Guidance when they do structured generation, uh, you know, as a function of the number of generated token. And at the bottom, it's Outlines.

[9:47](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=587s) Uh, Outlines stays approximately zero until the end. Uh, there's a trade-off. There's a compilation time. But during inference, it doesn't slow down inference. Now we're at a point where we could integrate this in Groq, and you wouldn't see the difference between structured and unstructured.

[10:00](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=600s) Um, so no overhead. But even more than no overhead, it is faster to generate text with structured generation. Um, the first is that when you take JSON, you don't need to generate the tokens that correspond to the bracket and to the field names.

[10:15](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=615s) I know that in advance. I don't need to ask the model to return, uh, those tokens. So here on this very simple example, only five out of ten tokens need to be generated, so only one half.

[10:26](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=626s) But there's an even more subtle, um, way in which it accelerates inference, and this is the, uh, example that we took at the beginning. So here I, I asked ChatGPT, like a good model, I asked ChatGPT the same question.

[10:40](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=640s) What is the, uh, public addre-- like the addre-- the... of Google's public DNS servers? And ChatGPT took fifty tokens. You know, it yapped it, yapped it, yapped and gave it up to fifty tokens.

[10:52](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=652s) It's not as bad. It could get a lot worse, uh, with lesser models. Uh, but when you use structured generation, you just generate eight tokens. So that's a subtle way in which it accelerates inference by a lot.

[11:03](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=663s) Um, then it improves efficiency, and that's probably the most, uh, actually mind-blowing result, uh, that we've had. So here what you're looking at is the accuracy on GSM8K, uh, with, again, Mistral [REDACTED:generic_id], structured and unstructured.

[11:21](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=681s) And here we look at the accuracy as a function of the number of shots, so the number of examples that you give to the model, uh, before asking the question.

[11:30](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=690s) And what we found is that, yeah, for unstructured, normal, one shot is worse than eight shots. Uh, that's completely expected. Uh, but what we found with structured is that you actually, and that's really surprised us, is that you actually get in the same ballpark in terms of accuracy with one shot as you do with eight shots, which

[11:46](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=706s) is surprising for a machine learning system. Like, you would think that examples are there to teach the model about the task, but it looks like it's actually there to teach the model about the structure of the problem.

[11:57](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=717s) There are more investigations to do in this line, but that was very mind-blowing. And the last one, which probably, you know, after faster, a lot of people care about here, is that it does improve the performance of open source models.

[12:10](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=730s) Um Here, um, what you're looking at is the Berkeley function calling leaderboard, uh, simple function benchmark, and we look at the accuracy. So first thing we did is that we took Microsoft Phi-3 Medium model, uh, which is a small model, uh, but we looked at its accuracy without structured generation.

[12:30](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=750s) It's eighty-six percent, which is pretty good for an open model. Uh, Phi-3 is actually a pretty good model. When you add structured generation, you get ninety-six point five percent.

[12:41](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=761s) And as a comparison, GPT-4, the best G- version of GPT-4 on this task get ninety-three point five percent, uh, on this benchmark. And now there are two things to note, is that ninety-six point five percent gets dangerously clo-- useful.

[12:56](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=776s) And the second thing is that we have open models that are available today that can beat, you know, larger models, um, without fine-tuning. So it's pretty huge win for, uh, open models, and that's why I'm really bullish on open models.

[13:12](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=792s) I think, you know, as a community, we can actually extract a lot more out of these models. Um,

[13:19](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=799s) and this is just a glimpse. Um, the work that I just showed you is what we did at .txt about a year ago. Since then, we've generalized from regular expression to what you call context-free grammars.

[13:32](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=812s) Context-free grammars are used to define code. They're used to define protein structure. I mean, and to define as well what I showed you earlier on the GSM8K example. So we can do the same thing, gener-- structured generation with no overhead with, um, with context-free grammar.

[13:47](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=827s) We also started working on, um, semantics, like adding some semantic constraints to the generation. And one very popular example of this is text-to-SQL. Uh, text-to-SQL most models get SQL syntax.

[14:01](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=841s) Usually what they get wrong is they hallucinate table or column names. At-- A,

[14:06](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=846s) internally, we're able to get perfect text-to-SQL, so I can't guarantee you that the query will be correct and give you the answer that you expect, but I can guarantee you that it will run.

[14:16](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=856s) So that's a pretty huge advance in text-to-SQL. And what else? Oh yeah, and we're also starting to, uh, to bubble up computations into the, uh, structured generation into the model architecture.

[14:28](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=868s) Because when you think about it, we're biasing logits. When you're biasing logits, the model is actually doing computation for nothing. And so you can gain even more in efficiency by preventing the model from doing these computations in the first place.

[14:40](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=880s) And that's all work that we'll actually publish in a blog post, I think, in the next couple of weeks. Uh,

[14:45](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=885s) so all that to say that if you're doing-- if you're not doing a chatbot, there's a really good chance that you will be using structured generation. You know, it's just a matter of time until you adopt it, I think.

[14:58](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=898s) Uh, our users are pretty, pretty, pretty happy. So

[15:02](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=902s) yeah. Thank you for your attention. And, uh, [audience applauding]

[15:06](https://www.youtube.com/watch?v=aNmfvN6S_n4&t=906s) all the, all the crazy claims that I made, you can go... The QR code, there's a link to all the blog posts. [laughs] [upbeat music]
