No more bad outputs with structured generation

Talk transcript

1. 0:00

   [on hold music] So yeah, my name's Rémi. I'm the co-author and co-maintainer of the open source library Outlines, uh, which some of you might know, and I'm also the CEO and co-founder of .txt or .text,

2. 0:25

   uh, whichever you prefer. Uh, we're more traditional machine learning people, and the motivation for work is the very simple observation that large language models are fundamentally flawed. I'll give you a very simple example.

3. 0:39

   Um, you're trying to extract flight information from a bunch of emails. Uh, of course, you want them to be, you know, a J-JSON object, um, you know, with origin, destination, et cetera.

4. 0:51

   So you go to OpenAI, you prompt the model to death, you threaten it, uh, you use function calling, and what you get sometimes as an answer is JSON decode error.

5. 1:01

   Um, I gave you very simple examples, but this has, like, very fundamental implications because computing rests on interfaces. We're able to build modular infrastructures and very complex infrastructure because we can trust the API over the pieces of code.

6. 1:17

   And here with LLM, and what you've probably witnessed, uh, you can't actually trust large language model to return consistent outputs. And, you know, in short is that the technology for agents is currently not there.

7. 1:30

   Um, so the good news is that structured generation, which is the ability of guiding the model to return to the specific structure, actually solves-- We'll see is that it allows you to be GPT-4 sort of a byproduct.

8. 1:47

   The goals for today are first to introduce the open source library Outlines for those of you who don't know about it, then very briefly explain how it works, I won't get into the technical details, and then try to convince you that you should use it today, uh, for, you know, most of the workflows, uh, that you have

9. 2:04

   to deal with and sort of a very short glimpse into the near future. So Outlines is a Python library, emphasis on library. Uh, you can actually include Outlines in your workflow, and it's more like frameworks where you have to make your workflow, you know, fit inside a framework.

10. 2:21

    Um, I think as a result, uh, it's been adopted by vLLM and TGI, uh, in the serving frameworks. And if you use function calling in either of these libraries, you're actually using Outlines under the hood-- uh, Outlines under the hood.

11. 2:36

    So I'm co-author, uh, but Outlines would be nothing without its contributors. Today, it's eighty-[REDACTED:generic_id]. Uh, I think it might be eighty-eight. I think I merged a PR this morning, I don't remember.

12. 2:45

    And so Outlines would be nothing with all-- without all these people, and I thank them-- uh, thank them a lot. Um, people thought we were crazy about a year ago when we were talking about structured generation.

13. 2:57

    Uh, but since then, uh, pretty happy because it looks like people are sort of caught up with the topic and realized that you can actually, you know, you can actually, uh, do structured output.

14. 3:09

    Um, so just now, just to run through-- quick run through outline. Um, so usually generating text happens in three stages. Uh, the first stage is that you need to choose the model and instantiate it.

15. 3:19

    So Outlines is purely focused on open source models. Uh, we have integration with six different model providers, uh, Transformers, LlamaCPP, and also, uh, recently we added MLX, uh, MLXLM.

16. 3:31

    Um, we have an integration with OpenAI, but that's mostly for us to compare the results that we get with open models with the results that are given by OpenAI.

17. 3:41

    The second step is to, I mean, generate text. What you do is that you instantiate a generator using generate.text. Here we just wanna, you know, return a single sentence.

18. 3:51

    So we're telling the generator, "Stop whenever you encounter a period." And question is described, then you call the generator, uh, with your prompt. Here is, "Describe the benefits of structured generation in one sentence."

19. 4:03

    And you'll have to wait for ten more minutes, uh, hopefully less. Okay, now we get into structured generation. So with Outlines-- without Outlines, if you ask what is the IP address of the public Google DNS servers, and you just generate text, you just let the LLM do its thing, then generally it will yap for a long time,

20. 4:24

    uh, you know, hundred tokens, five hundred tokens, and the answer will be somewhere in there. And the way you extract the answer is using regular expressions generally. Here, what you can do with Outlines is actually taking that regular expression that you use-- you would use to extract the answer and use it to guide the model, to tell

21. 4:42

    the model, "This is the structure that the output should follow." And as you see, you kind of remove the yapping, you print the-- you just call generate.regex, call the generator, and what you get is just the result, and it's actually the correct answer.

22. 4:55

    Uh, that was with Mistral, uh, [REDACTED:generic_id] BV [REDACTED:generic_id].

23. 4:59

    Regular expressions are not the only way to define structure. Uh, something that people need a lot in practice is a JSON. And Outlines allow you to generate, um, to generate text that,

24. 5:12

    you know, is a JSON object with a given structure. The way you specify the structure is using JSON Schema, or you can pass Pydantic models as well. Um, now you might notice on the flight information, so here we're-- you know, it's the example that I used at the beginning.

25. 5:27

    You're extracting flight information from an email. I could have used string as a type for origin and destination, but I did not. I used actually a custom type that we implemented in Outlines, and the reason is that origin and destination have way more structure than just text.

26. 5:41

    It's actually, you know, it's, it's an airport code that has three letters that's capitalized, and you can actually specify more and more structure, ev-- all the structure that you have in your problem, basically.

27. 5:52

    Uh, you can use this with vision models. Uh, that's something that we merged recently. So here we took, um, I think it's a picture from Wikipedia, uh, of a dish.

28. 6:02

    Uh, we

29. 6:03

    Tell the model what is the JSON that we expect as a, as an, as an output, and then we instantiate the generator and then pass the image and the prompt to the generator, and we get valid JSON.

30. 6:16

    Um, if you want to install Outlines, uh, and you think you could benefit from structured generation, then it's very simple. Just pip install outlines. Now, I'm gonna try to very quickly explain how it works.

31. 6:28

    Um, so models themselves, uh, what Mistral and Cohere, this one, are doing, uh, is actually training model weights. Uh, what a model does is, uh,

32. 6:38

    you input a prompt, you send a prompt, it's like token IDs, and what you get as an output is not text. It's logits. It's a probability distribution over the next token.

33. 6:46

    Now, what happens after that, when you want to generate text, the first step is that you have a logit processor that biases the logits. You probably use this every day actually without noticing it.

34. 6:56

    When you use temperature or when you use top-k, top-p sampling, you're actually biasing the logits. And once you have your biased logits, you use a sampling algorithm, then you get a token.

35. 7:04

    And once you have your token, you add it to the prompt and then feed it back to the LLM.

36. 7:09

    And where we fit is here. We actually wire the model...

37. 7:15

    Whenever the model generates logits, we look at every token and we say, "If I add this token to the current generation, is it gonna violate the structure?" If the answer is yes, we, we, like, we mask it so that it doesn't get generated.

38. 7:30

    Now, that story is very simple. What is really hard is doing that efficiently, and that's what we figured out at dottext, and that's what makes us different from the other libraries like Guidance or MQL that, um, do structured generation.

39. 7:43

    And now I'm going to convince you, uh, that there's absolutely no reason to not use... sorry for the double negation here, to not use structured generation. Uh, the first reason is that most text is structured.

40. 7:58

    Um, I talked to you about JSON earlier. We talked about regular expressions. But here I just took the GSM8K dataset. Um, if you look at a-- If you're not me and don't search everywhere, um, what I say immediately, if you look at the right, uh, you can actually see that it's highly structured.

41. 8:15

    It's always Q, uh, period, text until a question mark, then et cetera, so on and so forth. Arithmetic operation, which is defined by a context-free grammar, and you could actually express this in Outlines and just get the answer at the end, which is, you know, thirty-six.

42. 8:30

    So there's a lot of structured text out there, not just, uh... [phone ringing]

43. 8:35

    Thank you. [laughing] I'll, I'll be, I'll be quick. Um, [laughs] of course, the second benefit is that, uh, you get valid structure. I mean, that's an obvious one. That's what we're doing it.

44. 8:46

    Uh, I like this meme, uh, at the bottom. This is what people are currently doing. Uh, it's just crazy stuff to get valid JSON as an output, and it's not even guaranteed.

45. 8:54

    And here with Outlines, you just sample what you want. It's as simple as this. And as an experiment, it's actually an experiment that Prettybase did. Uh, they took Mistral [REDACTED:generic_id].

46. 9:04

    They used a version of CoNNL that they modified so that it gives structured output JSON. What they found is Mistral [REDACTED:generic_id] only gets valid JSON, uh, s- seventeen percent of the time.

47. 9:14

    When you add structured generation on top of it, you get ninety-nine point nine percent, and that's without optimizing the prompt. So you can actually get, you know, you can actually get better than this.

48. 9:26

    The nice thing is that it also adds negligible overhead. So you actually have... You know, you don't have to fear for that affecting inference time, uh, which is the highly, you know, highly nontrivial thing.

49. 9:36

    Uh, here we compared, uh, the overhead introduced by Guidance when they do structured generation, uh, you know, as a function of the number of generated token. And at the bottom, it's Outlines.

50. 9:47

    Uh, Outlines stays approximately zero until the end. Uh, there's a trade-off. There's a completion time. But during inference, it doesn't slow down inference. Now we're at a point where we could integrate this in Groq, and you wouldn't see the difference between structured and unstructured.

51. 10:00

    Um, so no overhead. But even more than no overhead, it is faster to generate text with structured generation. Um, the first is that when you take JSON, you don't need to generate the tokens that correspond to the bracket and to the field names.

52. 10:15

    I know that in advance. I don't need to ask the model to return, uh, those tokens. So here on this very simple example, only five out of ten tokens need to be generated, so only one half.

53. 10:26

    But there's an even more subtle, um, way in which it accelerates inference, and this is the, uh, example that we took at the beginning. So here I, I asked ChatGPT, like a good model, I asked ChatGPT the same question.

54. 10:40

    What is the, uh, public addre-- like the addre-- the... of Google's public DNS servers? And ChatGPT took fifty tokens. You know, it yapped it, yapped it, yapped and gave it up to fifty tokens.

55. 10:52

    It's not as bad. It could get a lot worse, uh, with lesser models. Uh, but when you use structured generation, you just generate eight tokens. So that's a subtle way in which it accelerates inference by a lot.

56. 11:03

    Um, then it improves efficiency, and that's probably the most, uh, actually mind-blowing result, uh, that we've had. So here what you're looking at is the accuracy on GSM8K, uh, with, again, Mistral [REDACTED:generic_id], structured and unstructured.

57. 11:21

    And here we look at the accuracy as a function of the number of shots, so the number of examples that you give to the model, uh, before asking the question.

58. 11:30

    And what we found is that, yeah, for unstructured, normal, one shot is worse than eight shots. Uh, that's completely expected. Uh, but what we found with structured is that you actually, and that's really surprised us, is that you actually get in the same ballpark in terms of accuracy with one shot as you do with eight shots, which

59. 11:46

    is surprising for a machine learning system. Like, you would think that examples are there to teach the model about the task, but it looks like it's actually there to teach the model about the structure of the problem.

60. 11:57

    There are more investigations to do in this line, but that was very mind-blowing. And the last one, which probably, you know, after faster, a lot of people care about here, is that it does improve the performance of open source models.

61. 12:10

    Um Here, um, what you're looking at is the Berkeley function calling leaderboard, uh, simple function benchmark, and we look at the accuracy. So first thing we did is that we took Microsoft 3 Medium model, uh, which is a small model, uh, but we looked at its accuracy without structure generation.

62. 12:30

    It's eighty-six percent, which is pretty good for an open model. Uh, Phi-3 is actually a pretty good model. When you add structure generation, you get ninety-six point five percent.

63. 12:41

    And as a comparison, GPT-4, the best G- version of GPT-4 on this task get ninety-three point five percent, uh, on this benchmark. And now there are two things to note, is that ninety-six point five percent gets dangerously clo-- useful.

64. 12:56

    And the second thing is that we have open models that are available today that can beat, you know, larger models, um, without fine-tuning. So it's pretty huge win for, uh, open models, and that's why I'm really bullish on open models.

65. 13:12

    I think, you know, as a community, we can actually extract a lot more out of these models. Um,

66. 13:19

    and this is just a glimpse. Um, the work that I just showed you is what we did at dottext about a year ago. Since then, we've generalized from regular expression to what you call context-free grammars.

67. 13:32

    Context-free grammars are used to define code. They're used to define protein structure. I mean, and to define as well what I showed you earlier on the GSM8K example. So we can do the same thing, gener-- structure generation with no overhead with, um, with context-free grammar.

68. 13:47

    We also started working on, um, semantics, like adding some semantic constraints to the generation. And one very popular example of this is to SQL. Uh, text-to-SQL most model, that's SQL syntax.

69. 14:01

    Usually what they get wrong is they hallucinate table or column names. At-- A,

70. 14:06

    internally, we're able to get perfect text to SQL, so I can't guarantee you that the query will be correct and give you the answer that you expect, but I can guarantee you that it will run.

71. 14:16

    So that's a pretty huge advance in text to SQL. And what else? Oh yeah, and we're also starting to, uh, to bubble up computations into the, uh, structure generation into the model architecture.

72. 14:28

    Because when you think about it, we're biasing logits. When you're biasing logits, the model is actually doing computation for nothing. And so you can gain even more in efficiency by preventing the model from doing these computations in the first place.

73. 14:40

    And that's all work that we'll actually publish in a blog post, I think, in the next couple of weeks. Uh,

74. 14:45

    so all that to say that if you're doing-- if you're not doing a chatbot, there's a really good chance that you will be using structured generation. You know, it's just a matter of time until you adopt it, I think.

75. 14:58

    Uh, our users are pretty, pretty, pretty happy. So

76. 15:02

    yeah. Thank you for your attention. And, uh, [audience applauding]

77. 15:06

    all the, all the crazy claims that I made, you can go... The QR code, there's a link to all the blog posts. [laughs] [upbeat music]