AI Engineer World's Fair 2024
No more bad outputs with structured generation
Read the talk
Making model outputs dependable with structured generation
A flight-extraction failure leads into Outlines, token masking, and the ways output constraints can improve validity, reduce generation work, and change task performance.
From a talk by Rémi Louf
Before you start: Familiarity with Python, JSON, and the idea that language models generate text one token at a time will help.
A JSON error is an interface failure
Suppose you want to extract flight details from emails. The next piece of software expects a JSON object containing an origin, a destination, and other flight information. You prompt the model carefully, insist on JSON, and try function calling—yet sometimes the result still produces a JSON decoding error. This is the opening problem Rémi Louf, co-author and co-maintainer of Outlines and co-founder and CEO of .txt, brings to structured generation.
Output structure is part of a software interface. Modular systems work because one component can rely on another component’s contract. If a model sometimes returns an object and sometimes returns something the caller cannot parse, every downstream consumer inherits that uncertainty. Louf connects this failure to the difficulty of building dependable agents: the concern is not merely an untidy answer, but an unreliable boundary between components.
Structured generation moves the required format into the generation process itself. Instead of asking the model to respect an interface and checking afterward, the decoder restricts what the model can emit. That directly addresses output structure; whether it also improves the answer is a separate question, which the later experiments test.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn an extraction pattern into a generation constraint
Outlines is a Python library intended to fit into an existing workflow, rather than a framework that requires the workflow to fit around it. At the time of the talk, Louf reports that vLLM and TGI use Outlines underneath their function-calling support. He also credits a contributor community of roughly 87–88 people for building the library.
The basic workflow has three steps:
- Choose and instantiate a model. The demonstration focuses on open models. Louf reports six provider integrations, including Transformers, llama.cpp, and the recently added MLX-LM; the OpenAI integration is used primarily for comparisons.
- Create a generator. In the introductory example,
generate.textcreates a text generator. - Call it with a prompt. The prompt asks for the benefits of structured generation in one sentence, with a period used as the stopping condition.
The recording uses the historical generate.* API. Current Outlines documentation uses a different model/output-type interface, so the API names here describe the demonstrated workflow rather than a current installation recipe.
Now ask for the IP address of Google’s public DNS servers. Unconstrained generation can surround the address with a long explanation. An application might then use a regular expression to recover the address from that text. Outlines lets you move that same kind of pattern earlier: give it to generate.regex, then call the resulting generator with the question. The pattern defines the allowed answer, rather than searching an answer after generation. Louf’s demonstration uses Mistral 7B v0.1 and returns the correct address directly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Specify the structure inside the fields
Regular expressions are one way to describe an output; JSON Schema and Pydantic models are another. Returning to flight extraction, a schema can require an object with origin and destination fields. But declaring both fields as arbitrary strings leaves useful information out of the contract. Louf instead uses a custom airport-code type: each value must contain three uppercase letters.
This distinction is easy to express in a Python schema definition. The following teaching example encodes the field constraint described in the talk; it is not a transcription of the custom Outlines type:
python
flight_schema = {
"type": "object",
"properties": {
"origin": {
"type": "string",
"pattern": "^[A-Z]{3}$",
},
"destination": {
"type": "string",
"pattern": "^[A-Z]{3}$",
},
},
"required": ["origin", "destination"],
"additionalProperties": False,
}
Three uppercase letters are a stronger contract than a string, but the pattern alone does not establish that the value names a real airport. The practical principle is to express the structure you actually know, while keeping format validity distinct from factual correctness.
The same schema-first workflow extends to vision. In a recently merged capability, Louf supplies a photograph of a dish, specifies the expected JSON, instantiates a generator, and passes both the image and the prompt. The demonstration returns valid JSON. He closes the usage introduction with the installation command pip install outlines.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Mask invalid continuations before sampling
A language model does not directly return text. Given a prompt represented as token IDs, it produces logits: scores from which next-token probabilities are derived. Generation then processes those scores and samples a token. Temperature changes the distribution; top-k and top-p restrict which candidates remain available. The sampled token is appended to the context, and the model runs again.
Structured generation intervenes between the model’s scores and token selection. For each candidate token, the constraint processor asks whether appending it to the current output would violate the required structure. If it would, the processor masks that token so it cannot be sampled. The remaining candidates still compete according to the model and sampling configuration; the constraint determines which continuations are admissible, not which admissible answer is true.
For a concrete illustration, suppose generation is inside the origin string from the flight schema and has already emitted SF. A candidate token whose decoded text is O preserves the three-uppercase-letter pattern. A candidate o violates it, and a closing quote would end the field too early. The latter two candidates must be masked. These are illustrative token texts, not a claim about a particular tokenizer’s vocabulary.
Candidate continuations inside an airport code
Constructed example: The prefix SF and candidate texts O, o, and a closing quote are invented teaching values. They do not assert a particular tokenizer vocabulary or reproduce a recorded token trace.
{"origin":"SFOperation: Apply the three-uppercase-letter constraint to candidate token texts before sampling.
Output prefix
{"origin":"SF{"origin":"SFCandidate token text: O
Candidate for sampling
Allowed: completes three uppercase letters
Candidate token text: o
Candidate for sampling
Masked: lowercase violates the pattern
Candidate token text: "
Candidate for sampling
Masked: would close a two-letter code
The difficult engineering problem is making those decisions efficiently across a model’s vocabulary at every step. Louf presents that efficiency as .txt’s distinguishing contribution relative to other structured-generation libraries, including Guidance and LMQL. The simple rule—exclude invalid continuations—does not by itself explain how to apply it cheaply.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Structure reaches beyond JSON
Even text that looks like ordinary reasoning can contain a useful formal structure. Louf uses a GSM8K math example: a question marker, question text ending in a question mark, and arithmetic within the solution. Arithmetic expressions can be described with a context-free grammar. That lets the constraint extend beyond a JSON envelope into the shape of the reasoning text and its final answer—36 in the illustrated problem.
The most direct benefit remains valid output. Prompting tricks attempt to persuade a model to emit JSON; constrained sampling limits generation to the specified structure. Louf illustrates the difference with a Predibase experiment using Mistral 7B v0.1 on a version of CoNLL modified to require JSON output.
| Generation method | Reported valid JSON |
|---|---|
| Unconstrained Mistral 7B v0.1 | 17% |
| With structured generation | 99.9% |
Louf says the constrained result used no prompt optimization. These are reported JSON-validity rates for that experiment, not extraction-accuracy scores or a universal success rate for every schema.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate constraint overhead from generation work
Checking constraints adds work, but when that work occurs matters. Louf shows a comparison with Guidance in which overhead is plotted against generated-token count. In the displayed comparison, he describes Outlines’ incremental inference overhead as approximately zero, with an upfront compilation cost. His suggestion that the difference could be imperceptible even in a Groq integration is a prospective claim, not a reported Groq deployment measurement.
Constraints can also reduce how much generation is necessary, through two different mechanisms. First, some output text is already determined by the structure. If the next part of a JSON object must be a bracket or a known field name, there may be no decision for the model to make. In Louf’s toy JSON example, only five of ten tokens require generation. Skipping model calls along deterministic paths is an additional optimization, not an automatic consequence of masking invalid tokens; it also requires care about tokenization and subsequent probabilities.
Second, a narrow output format eliminates unnecessary prose. Louf revisits the Google DNS question: in his comparison, ChatGPT produces 50 tokens, while constrained generation produces eight. This saving comes from returning just the requested value. It is distinct from skipping fixed JSON text, and neither token-count illustration establishes a universal wall-clock speedup.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What are prompt examples teaching?
The next experiment measures GSM8K accuracy for Mistral 7B v0.1 while varying the number of examples in the prompt. With unstructured generation, one-shot accuracy is worse than eight-shot accuracy. With structured generation, Louf reports one-shot and eight-shot accuracy in the same ballpark. The talk does not give exact accuracy values for this comparison.
That result suggests a possible role for demonstrations beyond teaching the task itself: some of their value may come from teaching the model the required output structure. If the decoder already supplies that structure, fewer examples may suffice. Louf explicitly leaves this as a direction for further investigation, rather than a general explanation of how in-context learning works.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A task-specific gain for Phi-3 Medium
The function-calling experiment asks whether constraints can improve task accuracy, not just parseability. Louf evaluates Microsoft’s Phi-3 Medium on the Berkeley Function Calling Leaderboard’s simple function benchmark. He reports the following results, comparing the constrained model with the best GPT-4 version in his comparison:
| Model and generation method | Reported accuracy in the talk |
|---|---|
| Phi-3 Medium, unconstrained | 86% |
| Phi-3 Medium, structured | 96.5% |
| GPT-4, version unspecified | 93.5% |
These are the figures spoken and shown in the recording. The June 17, 2024 companion report, Beating GPT-4 with Open Source, distinguishes results between runs and reports different final figures: 96.25% for Phi-3-medium-4k-instruct and 94.5% for GPT-4-0125-Preview on 400 BFCL simple examples. The recording’s chart should therefore be read as its own reported comparison, not silently replaced with the later figures.
The improvement is reported without fine-tuning. Its significance is specific but useful: changing the decoding constraints can extract substantially more task performance from an existing open model. It does not establish that Phi-3 Medium is generally more capable than GPT-4. Louf’s broader optimism about open models follows from this possibility of improving their useful behavior without changing their weights.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From valid syntax to usable outputs
Louf closes by moving beyond the earlier regular-expression work, which he describes as roughly a year old. The team has extended its approach to context-free grammars, with applications including code, protein structure, and the arithmetic used in the GSM8K example. His negligible-overhead claim here concerns experimental grammar generation: the contemporaneous research describes an in-house implementation, distinct from the public Outlines CFG implementation.
The next boundary is semantic constraints. Text-to-SQL makes the distinction concrete: a model can produce valid SQL syntax while referring to a table or column that does not exist. Restricting identifiers to the database’s available tables and columns addresses a failure that a syntax grammar alone cannot prevent. Louf says the team’s internal work can guarantee that a query will run, while explicitly declining to guarantee that it answers the intended question. Executability and correctness remain different contracts.
A further efficiency direction moves constraints into the model architecture. Masking logits happens after the model has already computed scores, including scores for tokens that will be discarded. Preventing unnecessary computation earlier could save additional work. Louf presents this as ongoing research, with implementation details and measured gains still to come in an anticipated blog post.
For applications outside open-ended chat, Louf predicts that structured generation will become a normal part of using language models. The flight-email example explains the practical pull: once an output must cross into another software component, its structure becomes part of what that component needs to trust. He ends by directing the audience to supporting blog posts through the closing QR code.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Python library for constrained outputs, with installation instructions and current typed-generation examples.
Further reading
The .txt team's report on Phi-3 Medium and GPT-4 evaluated on BFCL's 400-example simple function-calling subset.
- Prompt EfficiencyArticle
Experiments examining how structured generation changes few-shot performance on GSM8K and GPQA.
Willard and Louf's formulation of efficient constrained generation through finite-state transitions and vocabulary indexing.
An experimental CFG implementation illustrated with SQL generation and constraint-processing measurements.
How deterministic output structure can eliminate model calls, including the consequences of tokenization choices.
Updates since the talk
TGI's documented grammar compilation, token masking and tool-generation workflow.
Read the complete timestamped transcript
- 0:00
[on hold music] So yeah, my name's Rémi. I'm the co-author and co-maintainer of the open source library Outlines, uh, which some of you might know, and I'm also the CEO and co-founder of .txt or .text,
- 0:25
uh, whichever you prefer. Uh, we're more traditional machine learning people, and the motivation for work is the very simple observation that large language models are fundamentally flawed. I'll give you a very simple example.
- 0:39
Um, you're trying to extract flight information from a bunch of emails. Uh, of course, you want them to be, you know, a J-JSON object, um, you know, with origin, destination, et cetera.
- 0:51
So you go to OpenAI, you prompt the model to death, you threaten it, uh, you use function calling, and what you get sometimes as an answer is JSON decode error.
- 1:01
Um, I gave you very simple examples, but this has, like, very fundamental implications because computing rests on interfaces. We're able to build modular infrastructures and very complex infrastructure because we can trust the API over the pieces of code.
- 1:17
And here with LLM, and what you've probably witnessed, uh, you can't actually trust large language model to return consistent outputs. And, you know, in short is that the technology for agents is currently not there.
- 1:30
Um, so the good news is that structured generation, which is the ability of guiding the model to return to the specific structure, actually solves-- We'll see is that it allows you to be GPT-4 sort of a byproduct.
- 1:47
The goals for today are first to introduce the open source library Outlines for those of you who don't know about it, then very briefly explain how it works, I won't get into the technical details, and then try to convince you that you should use it today, uh, for, you know, most of the workflows, uh, that you have
- 2:04
to deal with and sort of a very short glimpse into the near future. So Outlines is a Python library, emphasis on library. Uh, you can actually include Outlines in your workflow, and it's more like frameworks where you have to make your workflow, you know, fit inside a framework.
- 2:21
Um, I think as a result, uh, it's been adopted by vLLM and TGI, uh, in the serving frameworks. And if you use function calling in either of these libraries, you're actually using Outlines under the hood-- uh, Outlines under the hood.
- 2:36
So I'm co-author, uh, but Outlines would be nothing without its contributors. Today, it's eighty-[REDACTED:generic_id]. Uh, I think it might be eighty-eight. I think I merged a PR this morning, I don't remember.
- 2:45
And so Outlines would be nothing with all-- without all these people, and I thank them-- uh, thank them a lot. Um, people thought we were crazy about a year ago when we were talking about structured generation.
- 2:57
Uh, but since then, uh, pretty happy because it looks like people are sort of caught up with the topic and realized that you can actually, you know, you can actually, uh, do structured output.
- 3:09
Um, so just now, just to run through-- quick run through outline. Um, so usually generating text happens in three stages. Uh, the first stage is that you need to choose the model and instantiate it.
- 3:19
So Outlines is purely focused on open source models. Uh, we have integration with six different model providers, uh, Transformers, LlamaCPP, and also, uh, recently we added MLX, uh, MLXLM.
- 3:31
Um, we have an integration with OpenAI, but that's mostly for us to compare the results that we get with open models with the results that are given by OpenAI.
- 3:41
The second step is to, I mean, generate text. What you do is that you instantiate a generator using generate.text. Here we just wanna, you know, return a single sentence.
- 3:51
So we're telling the generator, "Stop whenever you encounter a period." And question is described, then you call the generator, uh, with your prompt. Here is, "Describe the benefits of structured generation in one sentence."
- 4:03
And you'll have to wait for ten more minutes, uh, hopefully less. Okay, now we get into structured generation. So with Outlines-- without Outlines, if you ask what is the IP address of the public Google DNS servers, and you just generate text, you just let the LLM do its thing, then generally it will yap for a long time,
- 4:24
uh, you know, hundred tokens, five hundred tokens, and the answer will be somewhere in there. And the way you extract the answer is using regular expressions generally. Here, what you can do with Outlines is actually taking that regular expression that you use-- you would use to extract the answer and use it to guide the model, to tell
- 4:42
the model, "This is the structure that the output should follow." And as you see, you kind of remove the yapping, you print the-- you just call generate.regex, call the generator, and what you get is just the result, and it's actually the correct answer.
- 4:55
Uh, that was with Mistral, uh, [REDACTED:generic_id] BV [REDACTED:generic_id].
- 4:59
Regular expressions are not the only way to define structure. Uh, something that people need a lot in practice is a JSON. And Outlines allow you to generate, um, to generate text that,
- 5:12
you know, is a JSON object with a given structure. The way you specify the structure is using JSON Schema, or you can pass Pydantic models as well. Um, now you might notice on the flight information, so here we're-- you know, it's the example that I used at the beginning.
- 5:27
You're extracting flight information from an email. I could have used string as a type for origin and destination, but I did not. I used actually a custom type that we implemented in Outlines, and the reason is that origin and destination have way more structure than just text.
- 5:41
It's actually, you know, it's, it's an airport code that has three letters that's capitalized, and you can actually specify more and more structure, ev-- all the structure that you have in your problem, basically.
- 5:52
Uh, you can use this with vision models. Uh, that's something that we merged recently. So here we took, um, I think it's a picture from Wikipedia, uh, of a dish.
- 6:02
Uh, we
- 6:03
Tell the model what is the JSON that we expect as a, as an, as an output, and then we instantiate the generator and then pass the image and the prompt to the generator, and we get valid JSON.
- 6:16
Um, if you want to install Outlines, uh, and you think you could benefit from structured generation, then it's very simple. Just pip install outlines. Now, I'm gonna try to very quickly explain how it works.
- 6:28
Um, so models themselves, uh, what Mistral and Cohere, this one, are doing, uh, is actually training model weights. Uh, what a model does is, uh,
- 6:38
you input a prompt, you send a prompt, it's like token IDs, and what you get as an output is not text. It's logits. It's a probability distribution over the next token.
- 6:46
Now, what happens after that, when you want to generate text, the first step is that you have a logit processor that biases the logits. You probably use this every day actually without noticing it.
- 6:56
When you use temperature or when you use top-k, top-p sampling, you're actually biasing the logits. And once you have your biased logits, you use a sampling algorithm, then you get a token.
- 7:04
And once you have your token, you add it to the prompt and then feed it back to the LLM.
- 7:09
And where we fit is here. We actually wire the model...
- 7:15
Whenever the model generates logits, we look at every token and we say, "If I add this token to the current generation, is it gonna violate the structure?" If the answer is yes, we, we, like, we mask it so that it doesn't get generated.
- 7:30
Now, that story is very simple. What is really hard is doing that efficiently, and that's what we figured out at dottext, and that's what makes us different from the other libraries like Guidance or MQL that, um, do structured generation.
- 7:43
And now I'm going to convince you, uh, that there's absolutely no reason to not use... sorry for the double negation here, to not use structured generation. Uh, the first reason is that most text is structured.
- 7:58
Um, I talked to you about JSON earlier. We talked about regular expressions. But here I just took the GSM8K dataset. Um, if you look at a-- If you're not me and don't search everywhere, um, what I say immediately, if you look at the right, uh, you can actually see that it's highly structured.
- 8:15
It's always Q, uh, period, text until a question mark, then et cetera, so on and so forth. Arithmetic operation, which is defined by a context-free grammar, and you could actually express this in Outlines and just get the answer at the end, which is, you know, thirty-six.
- 8:30
So there's a lot of structured text out there, not just, uh... [phone ringing]
- 8:35
Thank you. [laughing] I'll, I'll be, I'll be quick. Um, [laughs] of course, the second benefit is that, uh, you get valid structure. I mean, that's an obvious one. That's what we're doing it.
- 8:46
Uh, I like this meme, uh, at the bottom. This is what people are currently doing. Uh, it's just crazy stuff to get valid JSON as an output, and it's not even guaranteed.
- 8:54
And here with Outlines, you just sample what you want. It's as simple as this. And as an experiment, it's actually an experiment that Prettybase did. Uh, they took Mistral [REDACTED:generic_id].
- 9:04
They used a version of CoNNL that they modified so that it gives structured output JSON. What they found is Mistral [REDACTED:generic_id] only gets valid JSON, uh, s- seventeen percent of the time.
- 9:14
When you add structured generation on top of it, you get ninety-nine point nine percent, and that's without optimizing the prompt. So you can actually get, you know, you can actually get better than this.
- 9:26
The nice thing is that it also adds negligible overhead. So you actually have... You know, you don't have to fear for that affecting inference time, uh, which is the highly, you know, highly nontrivial thing.
- 9:36
Uh, here we compared, uh, the overhead introduced by Guidance when they do structured generation, uh, you know, as a function of the number of generated token. And at the bottom, it's Outlines.
- 9:47
Uh, Outlines stays approximately zero until the end. Uh, there's a trade-off. There's a completion time. But during inference, it doesn't slow down inference. Now we're at a point where we could integrate this in Groq, and you wouldn't see the difference between structured and unstructured.
- 10:00
Um, so no overhead. But even more than no overhead, it is faster to generate text with structured generation. Um, the first is that when you take JSON, you don't need to generate the tokens that correspond to the bracket and to the field names.
- 10:15
I know that in advance. I don't need to ask the model to return, uh, those tokens. So here on this very simple example, only five out of ten tokens need to be generated, so only one half.
- 10:26
But there's an even more subtle, um, way in which it accelerates inference, and this is the, uh, example that we took at the beginning. So here I, I asked ChatGPT, like a good model, I asked ChatGPT the same question.
- 10:40
What is the, uh, public addre-- like the addre-- the... of Google's public DNS servers? And ChatGPT took fifty tokens. You know, it yapped it, yapped it, yapped and gave it up to fifty tokens.
- 10:52
It's not as bad. It could get a lot worse, uh, with lesser models. Uh, but when you use structured generation, you just generate eight tokens. So that's a subtle way in which it accelerates inference by a lot.
- 11:03
Um, then it improves efficiency, and that's probably the most, uh, actually mind-blowing result, uh, that we've had. So here what you're looking at is the accuracy on GSM8K, uh, with, again, Mistral [REDACTED:generic_id], structured and unstructured.
- 11:21
And here we look at the accuracy as a function of the number of shots, so the number of examples that you give to the model, uh, before asking the question.
- 11:30
And what we found is that, yeah, for unstructured, normal, one shot is worse than eight shots. Uh, that's completely expected. Uh, but what we found with structured is that you actually, and that's really surprised us, is that you actually get in the same ballpark in terms of accuracy with one shot as you do with eight shots, which
- 11:46
is surprising for a machine learning system. Like, you would think that examples are there to teach the model about the task, but it looks like it's actually there to teach the model about the structure of the problem.
- 11:57
There are more investigations to do in this line, but that was very mind-blowing. And the last one, which probably, you know, after faster, a lot of people care about here, is that it does improve the performance of open source models.
- 12:10
Um Here, um, what you're looking at is the Berkeley function calling leaderboard, uh, simple function benchmark, and we look at the accuracy. So first thing we did is that we took Microsoft 3 Medium model, uh, which is a small model, uh, but we looked at its accuracy without structure generation.
- 12:30
It's eighty-six percent, which is pretty good for an open model. Uh, Phi-3 is actually a pretty good model. When you add structure generation, you get ninety-six point five percent.
- 12:41
And as a comparison, GPT-4, the best G- version of GPT-4 on this task get ninety-three point five percent, uh, on this benchmark. And now there are two things to note, is that ninety-six point five percent gets dangerously clo-- useful.
- 12:56
And the second thing is that we have open models that are available today that can beat, you know, larger models, um, without fine-tuning. So it's pretty huge win for, uh, open models, and that's why I'm really bullish on open models.
- 13:12
I think, you know, as a community, we can actually extract a lot more out of these models. Um,
- 13:19
and this is just a glimpse. Um, the work that I just showed you is what we did at dottext about a year ago. Since then, we've generalized from regular expression to what you call context-free grammars.
- 13:32
Context-free grammars are used to define code. They're used to define protein structure. I mean, and to define as well what I showed you earlier on the GSM8K example. So we can do the same thing, gener-- structure generation with no overhead with, um, with context-free grammar.
- 13:47
We also started working on, um, semantics, like adding some semantic constraints to the generation. And one very popular example of this is to SQL. Uh, text-to-SQL most model, that's SQL syntax.
- 14:01
Usually what they get wrong is they hallucinate table or column names. At-- A,
- 14:06
internally, we're able to get perfect text to SQL, so I can't guarantee you that the query will be correct and give you the answer that you expect, but I can guarantee you that it will run.
- 14:16
So that's a pretty huge advance in text to SQL. And what else? Oh yeah, and we're also starting to, uh, to bubble up computations into the, uh, structure generation into the model architecture.
- 14:28
Because when you think about it, we're biasing logits. When you're biasing logits, the model is actually doing computation for nothing. And so you can gain even more in efficiency by preventing the model from doing these computations in the first place.
- 14:40
And that's all work that we'll actually publish in a blog post, I think, in the next couple of weeks. Uh,
- 14:45
so all that to say that if you're doing-- if you're not doing a chatbot, there's a really good chance that you will be using structured generation. You know, it's just a matter of time until you adopt it, I think.
- 14:58
Uh, our users are pretty, pretty, pretty happy. So
- 15:02
yeah. Thank you for your attention. And, uh, [audience applauding]
- 15:06
all the, all the crazy claims that I made, you can go... The QR code, there's a link to all the blog posts. [laughs] [upbeat music]