DSPy: The End of Prompt Engineering

Talk transcript

1. 0:00

   [on-hold music] Thanks everybody for, uh, for joining.

2. 0:23

   I'm here to talk to you today about DSPy. Um, and feel free to jump in with questions or anything throughout the talk. It's, you know, I don't spend-- I don't plan on spending the full hour and a half or so.

3. 0:34

   I know it's the last session of the day, so, um, keep it casual. Feel free to jump in. I'll start with a little bit of background. Don't wanna go through too many slides.

4. 0:43

   I'm technically a consultant, so I have to do some slides, but we will dive into the code for the, the latter half, and there's a GitHub repo that you can download to, to follow along and play around with it on your own.

5. 0:56

   Um, so how many people here have heard of DSPy?

6. 1:02

   Almost everyone. That's awesome. How many people have actually used it kind of day-to-day in production or anything like that? Three. Okay, good. So hopefully we can convert some more of you today.

7. 1:13

   Um, so high level, DSPy, this is straight from the website. Um, it's a declarative framework for how you can build modular software, and most important for someone like myself, I'm not necessarily, um, an engineer that is writing code all day, every day.

8. 1:29

   As I mentioned before, I'm a more of a technical consultant, so I run across a variety of different problems. Could be, um, an investigation for a law firm. It could be helping a company understand how to improve their processes, how to deploy AI internally.

9. 1:43

   Maybe we need to look through ten, ten thousand contracts to identify a particular clause, um, or, or paragraph. And so DSPy has been a really nice way for me personally and my team to iterate really, really quickly on building these ap- building these applications.

10. 1:59

    Most importantly, building programs. It's not, um, it's not kind of iterating with prompts and tweaking things back and forth. It is building a, a proper Python program, and, and DSPy is a really good way for you to do that.

11. 2:13

    So I mentioned before there's a repo online if you wanna download it now and kind of just get everything set up. I'll put this on the screen later on.

12. 2:21

    Um, but if you wanna go here, just kind of download some of the code. It, uh, it's been put together over the past couple of days, so it's not gonna be perfect production level code.

13. 2:30

    It's much more of utilities and little things here and there to just come and kind of demonstrate the usefulness, demonstrate the point of, of what we're talking about today.

14. 2:39

    In that, and we'll walk through all of these, these different use cases. So, um, sentiment classifier going through a PDF, some multimodal work, uh, a very, very simple web research agent detecting boundaries of a PDF document.

15. 2:53

    You'll see how to summarize basically arbitrary length text and then go into an optimizer, uh, with Jetboat.

16. 3:02

    But before we do that, just again, kind of level set, the biggest thing for me personally, DSPy is a really nice way to decompose your logic into a program that treats LLMs as a first-class citizen.

17. 3:15

    So at the end of the day, you're fundamentally just calling a function that under the hood just happens to be an LLM. And DSPy gives you a really nice, intuitive, easy way to do that with some guarantees about the input and output types.

18. 3:30

    So of course, there are structured outputs. Of course, there are other ways to do this, Pydantic and others. Um, but DSPy has a set of primitives that when you put it all together, allows you to build a cohesive modular piece of software that you then happen to be able to optimize, and we'll, we'll get into that, uh,

19. 3:49

    in a minute. So just a few reasons of why I'm such an advocate. It sits at, it sits at this really nice level of abstraction, so it's-- I, I would say it doesn't get in your way as much as a LangChain, and that's not a knock on LangChain.

20. 4:04

    It's just a different kind of paradigm, the way that DSPy is, is structured, um, and allows you to focus on things that actually matter. So you're not writing .choices, zero messages content.

21. 4:16

    You're not, you're not doing string parsing. You're not doing a bunch of stuff under the hood. You're just declaring your intent of how you want the program to operate, what you want your out- inputs and outputs to be.

22. 4:27

    Because of this, it allows you to create computer programs, as I mentioned before, not just tweaking strings and sending them back and forth. You are building a program first.

23. 4:36

    It just happens to also use LLMs. And really the, the most kind of important part of this is that in Omar that-- Kateb, the, uh, the founder of this or the, the original developer of it, had this really good podcast with a16z, I think came out just like two or three days ago.

24. 4:52

    But it, he, he put it a really nice way. He said it's, it's built with a systems mindset, and it's really about how you are encoding or expressing your intent of what you want to do, most importantly, in a way that's transferable.

25. 5:04

    So the, the design of your system, I would imagine, or your program isn't going to move necessarily as quickly as maybe the model capabilities are under the hood when we see new releases almost every single day, different capabilities, better models.

26. 5:20

    And so DSPy allows you to structure it in a way that retains the control flow, it con-- uh, retains the intent of your system, your program, um, while allowing you to bounce from model to model to the extent that you want to or, or need to.

27. 5:36

    Convenience comes for free. There's no parsing JSON, things like that. But again, it sits at a nice level of abstraction where you can still understand what's going on under the hood.

28. 5:45

    If you want to, you can go in and tweak things, but it allows you to, to kind of focus on just what you want to do while retaining the level of precision that you, that I think most of us would like to have in, in kind of building your programs.

29. 5:58

    Um, as mentioned, it's, it's robust to kind of model and paradigm shifts, so you can again, keep the logic of your program, um, but it, but keep tho-those LLMs infused in, uh- Basically inline.

30. 6:11

    Now that being said, you know, there are absolutely other great libraries out there. Pydantic AI, LangChain, I mean, there's many, many others that allow you to do similar things.

31. 6:19

    Agno is another one. Um, this is just one perspective and, um, it may not be perfect for your use case. For me, it took me a little bit to kind of grok how DSPy works, and you'll see why that is in a minute.

32. 6:33

    Um, so I would just recommend to kind of have an op-have an open mind, play with it, um, run the code, tweak the code, do whatever you need to do, um, and just see how it might work, might work for you.

33. 6:45

    And really this talk is more about ways that I found it useful. It's not a dissertation on the ins and outs of every nook and cranny of DSPy. It's more of, you know, I've run into these problems myself, now I naturally run to DSPy to solve them.

34. 7:00

    Uh, and this is kind of why. And the hope is that you can extrapolate some of this to your own use cases. So we'll, we'll go through everything, uh, fairly quickly here, but the core concepts of DSPy really comes down to arguably five or these six that you see on the screen here.

35. 7:16

    So we'll go into each of these in more detail, but high level signatures specify what you want the LL-- what, basically what you want your function call to do.

36. 7:27

    This is when you specify your inputs, your outputs. Inputs and outputs can both be typed, um, and you defer the rest of the, basically the how, the implementation of it to the LLM.

37. 7:39

    And we'll see how we, how that all kinda comes together, uh, in a minute. Modules themselves are ways to logically structure your program. They're based off of signatures, so a module can have one or more signatures embedded within it, in addition to, uh, additional logic, and it's based off of, um, PyTorch and, and some of the te--

38. 8:00

    in terms of like the methodology for how it's structured, and you'll, you'll see how that, uh, comes to be in a minute.

39. 8:07

    Tools, we're all familiar with tools, MCP and others, and really tools fundamentally, as DSPy looks at them, are just Python functions. So it's just a way for you to very easily expose Python functions to the LLM within the DSPy kind of ecosystem, if you will.

40. 8:25

    Um, adapters live in between your signature and the LLM call itself. I mean, as we all know, prompts are ultimately just strings of text that are sent to the LLM.

41. 8:40

    Signatures are a way for you to express your intent at a, at a higher level. And so adapters are the things that sit in between those two. So it's how you translate your inputs and outputs into a format, basically explodes out from your initial signature into a format that is ultimately the prompt that is sent to the

42. 8:59

    LLM. And so, you know, there's some debate or some research on if certain models perform better with XML, as an example, or BAML or JSON or others. And so adapters give you a nice easy abstraction to, to basically mix and match those at, at will, uh, you know, as you want.

43. 9:20

    Optimizers, um, are the most interesting and, for whatever reason, the most controversial part of DSPy. That's kind of the first thing that people think of, or at least when they hear of DSPy, they think optimizers.

44. 9:34

    We'll see a quote in a minute. It's not optimizers first, it is just a nice added benefit and a nice capability that DSPy offers in addition to the ability to structure your program with the signatures and modules and everything else.

45. 9:48

    Um, and metrics are used in tandem with optimizers that it basically defines how you measure success in your, in, uh, your DSPy program. So the optimizers use the metrics to determine if it's finding the right path, if you will.

46. 10:05

    So signatures, I mentioned before, it's how you express your intent, your declarative intent. Can be super simple strings, and this is the weirdest part for me initially, but is one of the most powerful parts, uh, of it now.

47. 10:17

    Or it can be more complicated class-based o- class-based objects. If you've used Pydantic, it, that's basically what it, what it runs on under the hood.

48. 10:27

    So this is exa- an example of one of the class-based signatures. Again, it, it's basically just a Pydantic object.

49. 10:35

    What's super interesting about this is that the,

50. 10:40

    the names of the fields themselves act almost as like mini prompts, as part of the prompt itself, and you'll see how this comes to life in a minute. But what's ultimately passed to the model from something like this is, oh, it will say, "Okay, your inputs are going to be a parameter called text," and it's based off

51. 10:59

    of the name of the, that particular parameter in this class. And so these things are actually passed through. And so it, it's, it's very important, uh, to be able to name your parameters in a way that is intuitive for the model to be able to pick it up.

52. 11:16

    Um, and you can add some additional context or what have you in the description field here. So most of this, if not all of this, yes, it is proper, you know, typed Python code, but it's also, it also serves almost as a prompt ultimately that feeds into the model.

53. 11:32

    Um, and that's basically translated through the use of adapters.

54. 11:36

    Um, and so just to highlight here, like these, it's the ones that are a little bit darker and bold, you know, those are the things that are effectively part of the prompt, uh, that's been sent in.

55. 11:47

    And you'll see kind of how DSPy works with all this and, and formats it in a way that, again, allow you to just worry about what you want, worry about constructing your signature instead of figuring out how best to word something in the prompt.

56. 12:00

    Go ahead.

57. 12:00

    Oh, uh, Kevin, it's okay. I just wanna jump in with one thing, 'cause like a lot of people get hung up with like, "But I have a really good prompt-

58. 12:07

    Sure

59. 12:07

    ... so then I don't want this thing." Again, prompt. For that, you literally just do it here. So we get prompt from docstring.

60. 12:14

    Yeah

61. 12:15

    And eventually start from-

62. 12:17

    That's exactly right

63. 12:19

    ... something good. A lot of people just like a version to just buy it. It comes from thinking that they're gonna get something.

64. 12:25

    Sure.

65. 12:27

    So it's like, okay, good prompt.

66. 12:28

    So the, the question from folks online is, what if I al- already have a great prompt? I've done all this work. I'm a, I'm an amazing prompt engineer. I don't want my job to go away or whatever.

67. 12:37

    Um, yes. So you can absolutely start with a custom prompt or something that you have demonstrated works really well, and you're exactly right. That's, that can be done in the docstring itself.

68. 12:49

    There's an, there are some other methods in order, uh, for you to inject basically system instructions or add additional things at certain parts of the ultimate prompt and/or of course, you can just inject it in the, in the final string anyway.

69. 13:01

    I mean, it, it's just, you know, a string that is constructed by DSPy. So, um, absolutely, this doesn't necessarily prevent you. It does, does not prevent you from adding in some super prompt that you already have.

70. 13:14

    Absolutely. Um, and to your point, it is, it can serve as a nice starting point from which to build the rest of the system.

71. 13:24

    Here's a shorthand version of the same exact thing, which to me, the first time I saw this, so this was like baffling to me. Um, but it, it... That's exactly how it works, is that you're basically, again, kind of deferring the implementation or the logic or what have you to DSPy and the model to basically figure out

72. 13:41

    what you wanna do. So in this case, if I want a super, super simple text, uh, sentiment classifier, this is basically all you need. You're just saying, "Okay, I'm gonna give you text as an input.

73. 13:52

    I want the sentiment as an integer, as the output." Now, you probably wanna specify some additional instructions to say, "Okay, your sentiment, you know, a lower number means negative, a po- you know, a, a higher number is more positive sentiment," et cetera.

74. 14:05

    But it just gives you a nice kind of easy way to, to kind of scaffold these things out in a way that you don't have to worry about like, you know, creating this whole prompt from hand.

75. 14:15

    It's like, okay, I just wanna see how this works, and then if it works, then I can add the additional instructions, then I can create a module out of it or, you know, whatever it might be.

76. 14:23

    It's these shorthand, or, or it is this shorthand that makes experimentation and iteration incredibly quick.

77. 14:32

    So modules, it's the base abstraction layer for DSPy programs. There are a bunch of modules that are built in, and these are a collection of kind of prompting techniques, if you will, and you can always create your own module.

78. 14:46

    So to the question before, if you have something that you know works really well, sure, yeah, put it in a module. That's now the kind of the base assumption, the base module that others can build off of.

79. 14:57

    And all of DSPy is meant to be composable, optimizable, [clears throat] and when you deconstruct your business logic or whatever you're trying to achieve by using these different primitives, it all, it's intended to kind of fit together and flow together.

80. 15:12

    Um, and we'll get to optimizers in a minute, but at least for me and my team's experience, just being able to logically separate the different components of a program, but basically inlining, uh, LLM calls has been incredibly powerful for us, and it's just an added benefit that at the end of the day, because we're just kind of

81. 15:31

    in the DSPy paradigm, we happen to also be able to optimize it at the end of the day.

82. 15:37

    Uh, so it comes with a bunch of standard ones built in. I ha- I don't use some of these bottom ones as much, although it's, they're super interesting. Um, the base one at the top there is just dspy.predict.

83. 15:50

    That's literally just, you know, an LLM call that's just, uh, a vanilla call. Chain of thought, uh, probably isn't a- isn't as relevant anymore these days because models have kind of ironed those out.

84. 16:02

    But, um, it is a good example of the types of, um, kind of prompting techniques that can be built into some of these modules. Um, and basically, all this does is add, um, some, some of the, uh, strings from literature to say, "Okay, let's think step by step," or whatever that might be.

85. 16:21

    Same thing for react and CodeAct. React is basically the way that you expose the tool to the model, so it's wrapping and doing some things under the hood with, um, basically taking your signatures and, uh, it's injecting the Python functions that you've given it as tools, and basically react is how you do tool calling in DSPy.

86. 16:43

    Program with thought is, uh, is pretty cool. It kind of forces the model to think in code, and then will return the result. Um, and you can give it a-- It comes with a Python interpreter built in, but you can give it some custom one, some type of custom harness if you wanted to.

87. 17:00

    Um, I haven't played with that one too, too much, but it is super interesting if you have like a highly technical problem or workflow or something like that where you want the model to inject reasoning in code at certain parts of your pipeline.

88. 17:13

    That's a, that's kind of a really easy way to do it. And then some of these other ones are basically just different methodologies for comparing outputs or running things in parallel.

89. 17:23

    So here's what one looks like. Again, it's, it's fairly simple. It's, you know, it, it is a Python class at the end of the day. Um, and so you do some initial initialization up top.

90. 17:34

    In this case, you're seeing the, uh, uh, the shorthand signature up there. So I'm-- This module, uh, just to give you some context, is an excerpt from, um, one of the, the Python, um, files that's in the repo, is basically taking in a bunch of time entries and making sure that they adhere to certain standards, making sure

91. 17:58

    that things are capitalized properly, or that there are periods at the end of the sentences or whatever it might be. That's from a, a real client use case where they had hundreds of thousands of time entries, and they needed to make sure that they all adhere to the same format.

92. 18:11

    This was one way to, to kind of do that very elegantly, at least in my opinion, was- Taking up top, you can define the, the signature. It's adding the-- some additional instructions that were defined elsewhere, and then saying for this module, the, the change tense, um, call is going to be just a vanilla predict call.

93. 18:32

    And then when you actually call the module, you enter into the forward function, which you can inter-- basically intersperse the LLM call, which would be the first one, and then do some kind of hard-coded business logic beneath it.

94. 18:47

    Uh, tools, as I mentioned before, these are just vanilla kind of Python functions. It's the DSPy's tool interface. So under the hood, DSPy uses LiteLLM, and so there needs to be some kind of coupling between the two.

95. 19:01

    But fundamentally, um, any type of tool that would u- that you would use elsewhere, you can also use in, in DSPy. And this is probably obvious to most of you, but here's just an example.

96. 19:13

    You have two functions, get weather, search web. You include that with a signature. So in this case, I'm saying the signature is, "I'm gonna give you a question. Please give me an answer."

97. 19:24

    I'm not even specifying the types. It's just gonna infer what that means. Uh, I'm giving it the get weather and the search web tools, and I'm saying, "Okay, do your thing, but only go five rounds," just so it doesn't spin off into something crazy.

98. 19:38

    And then a call here is literally just calling the ReACT agent that I created above with the question, "What's the weather like in Tokyo?" We'll see an example of this in the code session, but basically what this would do is give the model the prompt, the tools, and let it do its thing.

99. 19:57

    So adapters, before I cover this a little bit, they're basically prompt formatters, if you will. So the description from the docs probably says it best. It's, you know, it takes your signature, the inputs, other attributes, and it converts them into some type of message format that you have specified or that the adapter has specified.

100. 20:16

     And so as an example, the JSON adapter taking, say, a Pydantic object that we defined before, this is the actual prompt that's sent into the LLM. And so you can see the input fields.

101. 20:28

     So this would have been defined as, okay, clinical note type string, patient info as a patient details object, which, which would have been defined elsewhere. And then this is the definition of the patient info, basically a JSON dump of that Pydantic object.

102. 20:45

     Go ahead.

103. 20:46

     So this idea, there's like a base adapter or default-

104. 20:49

     Yeah

105. 20:49

     ... that's good for most cases, and this is if you want to tweak that to do something more specific?

106. 20:52

     That's right.

107. 20:53

     Okay.

108. 20:53

     Yeah. The question was if, if there's a base adapter and would this be an example of where you want to do something specific? Answer is yes. So, um, there's a guy, Prashanth, who is, um, I have his Twitter at the end of this presentation, but he's been great. [clears throat]

109. 21:08

     He did some testing comparing the JSON adapter with the BAML adapter. Um, and you can see just intuitively, even us, even for us humans, the way that this is formatted is a little bit more intuitive.

110. 21:18

     It's probably more token efficient too. Just considering, like, if you look at the messy JSON that's here versus the, I guess, slightly better formatted BAML that's here, um, can actually improve performance by, you know, five to ten percent depending on your use case.

111. 21:34

     So it's a good example of how you can format things differently. The, the rest of the program wouldn't have changed at all. You just specify the BAML adapter, and it totally changes how the information is presented under the hood to the LLM.

112. 21:50

     Multimodality, I mean, this obviously is more at the model level, but DSPy supports multiple modalities by default, so images, audio, some others. Um, and the same type of thing.

113. 22:00

     You kind of just feed it in as part of your signature, and then you can get some very nice clean output. This allows you to work with them very, very, very easily, very quickly.

114. 22:10

     And for those, uh, eagle-eyed participants, you can see the first, uh, line up there is attachments. It's probably a lesser-known library. Another guy on Twitter is awesome, uh, Maxime, I think it is.

115. 22:23

     Uh, he created this library that just is basically a catchall for working with different types of files and converting them into a format that's super easy to use with LLMs.

116. 22:34

     Um, he's a big DSPy fan as well, so he made basically an adapter that's specific to this. But that's all it takes to pull in images, PDFs, whatever it might be.

117. 22:45

     You'll see some examples of that. Uh, it just makes, at least has made my life super, super easy.

118. 22:52

     Here's another example of the same sort of thing. So this is a PDF of a form four form, some, you know, public SEC form from NVIDIA.

119. 23:02

     Um, up top, I'm just giving it the link. I'm saying, "Okay, attachments, do your thing, pull it down, create images, whatever you're gonna do. I don't need to worry about it.

120. 23:10

     I don't care about it." This is super simple RAG, but basically, okay, I wanna do RAG over this document. I'm gonna give you a question. I'm gonna give you the document, and I want the answer.

121. 23:21

     Um, and you can see how simple that is, literally just feeding in the document how many shares were sold. Interestingly here, I'm not sure if it's super easy to see, but you actually have two transactions here, so it's gonna have to do some math, likely under the hood.

122. 23:37

     And you can see here the thinking and the, the ultimate answer. Go ahead.

123. 23:42

     Is it, on the RAG step, is it creating a vector store of some kind or creating embeddings and then searching over those? Is there a bunch going on in the background there, or what's happening?

124. 23:51

     This is poor man's RAG. I should have clarified. This is, this is literally just pulling in the document images, and I think attachments will do some basic OCR under the hood.

125. 24:03

     Um, but it doesn't do anything other than that. That's it. All we're feeding in here, the, the actual document object that's being fed in, yeah, is literally just the text that's been OCR'd, the images, the model does the rest.

126. 24:17

     All right, so optimizers. Uh, let's see how we're doing. Okay. Um, optimizers are super powerful, super interesting concept. There's been some research, um, that argues, I think, that it's just as performant, if not in search-- in certain situations, more performant than fine-tuning would be for certain models, for certain situations.

127. 24:39

     There's all this research about in-context learning and such, and so

128. 24:43

     whether you want to go fine-tune and do all of that, nothing stops you. But I would recommend at least trying this first to see how far you can get without having to set up a bunch of infrastructure and, you know, go through all of that.

129. 24:55

     See how the optimizers work. Um, but fundamentally, what it allows you to do is DSPy gives you the primitives that you need and the organization you need to be able to measure and then quantitatively improve that performance.

130. 25:10

     And I mentioned transferability before. This-- T-t-transferability is enabled arguably through the use of optimizers because if you can get, okay, I wanna-- I have a classification task, works really well with Four One, but maybe it's a little bit costly 'cause I have to run it a million times a day.

131. 25:29

     Can I try it with Four One Nano? Okay, maybe it's at seventy percent, whatever it might be, but I run the optimizer on Four One Nano, and I can get the performance back up to maybe eighty-seven percent, and maybe that's okay for my use case.

132. 25:43

     But I've now just dropped my cross-- my cost profile by multiple orders of magnitude, and it's the optimizer that allows you to do that type of model and kind of use case transferability, if you will.

133. 25:55

     But really, all it does at, at the end of the day under the hood is iteratively prompt-- uh, iteratively optimize or tweak that prompt, that string under the hood.

134. 26:05

     And because you've constructed your program using the different modules, DSPy kind of handles all of that for you under the hood. So if you compose a program with multiple modules, and you're optimizing against all that, it, it by itself, DSPy, will optimize the various components in order to improve the input and output performance.

135. 26:27

     And we'll, we'll take it from the man himself, Omar. You know, DSPy is not an optimizer. I've said this multiple times. It's, it's just a set of programming abstractions or a way to program.

136. 26:38

     You just happen to be able to optimize it. Um, so again, the value that I've gotten and my team has gotten is mostly because of the programming abstractions. It's just this incredible added benefit that you are also able to, to-- should you choose, to optimize it afterwards.

137. 26:55

     And I was listening to this-- to, uh, Dwarkesh and, and, uh, Karpathy the other day, and this kind of-- It w-- I was, like, prepping for this talk, and this, like, hit home perfectly.

138. 27:05

     I was thinking about the optimizers. And someone smarter than me can, can plea-- you know, please correct me, but I think this makes sense because he, he was basically talking about L-- using LLM as a judge can be a bad thing because the model being judged can find adversarial examples and degrade the performance or basically, um, create

139. 27:30

     a situation where the judge is not, uh, not scoring something properly. Um, because he's saying that the model will find these little cracks. It'll find these little spurious things in the nooks and crannies of the giant model and find a way to cheat it, basically saying that LLM as a judge can only go so far until the

140. 27:47

     other model, uh, finds those adversarial examples. If you kind of invert that and flip that on its head, it's this property that the optimizers for DSPy are taking advantage of to optimize-- to find the nooks and crannies in the model, whether it's a bigger model or, model or smaller model,

141. 28:04

     to improve the performance against your dataset. So that's what the optimizer is doing, is finding these look-- finding these nooks and crannies in the model to optimize and improve that performance.

142. 28:15

     So a typical flow, I'm not gonna spend too much time on this, but fairly logical. Construct your program, which is decomposing your logic into the modules. You use your metrics to define basically the contours of how the program works, and you optimize all that through, um, to, to get your, your, uh, your final result.

143. 28:37

     So another talk that this guy, Chris Potts, just had maybe two days ago, um, where he made the point, this is what I was mentioning before, where GEPA, which is, uh, you probably saw some of the, the talks the other day, um,

144. 28:51

     where the optimizers are on par or exceed the performance of something like GRPO, a-another kind of fine-tuning method. So pretty impressive. I think it's an active area of research.

145. 29:02

     People a lot smarter than me, like Omar and Chris and others are, are leading the way on this. But, uh, point being, I think prompt optim-prompt optimization is a pretty exciting place to be, and if nothing else, is worth exploring. [clears throat]

146. 29:17

     And then finally, metrics. Again, these are kind of the building blocks that allow you to define what success looks like for the optimizer. So this is what it's using, and you can have many of these, and we'll see examples of this where, again, at a high level, your program works on inputs, it works on outputs.

147. 29:36

     The optimizer is going to use the metrics to understand, okay, my last tweak in the prompts, did it improve performance? Did it, did it degrade performance? And the way you define your metrics, uh, provides that direct feedback for the optimizers to work on.

148. 29:53

     Uh, so here's another example, a super simple one from that time entry example I mentioned before. Um, so they can be-- the metrics can either be, like, fairly rigorous in terms of, like, does this equal one or, or, you know, some type of a quality check, or a little bit more subjective.

149. 30:09

     We're using LLM as a judge to say whatever. Was this generated, um, string, you know, does it adhere to these, you know, various criteria, whatever it might be. But that itself can be a metric

150. 30:22

     And so all of this is to say, it's a very long-winded way of saying, in my opinion, this is probably most, if not all of what you need to construct arbitrarily complex workflows, data processing pipelines, business logic, whatever that might be, different ways to work with LLMs.

151. 30:39

     If nothing else, DSPy gives you the primitives that you need in order to build these modular composable systems.

152. 30:48

     So if you're interested, some people online, um, there's many, many more. There's a Discord community as well. Um, but usually these people are, are on top of the latest and greatest, and so would recommend giving them a follow.

153. 31:02

     You don't need to follow me, I don't really do much. But, uh, the others on there are, are really pretty good.

154. 31:08

     Okay, so the fun part. We'll actually get into, to some code. So if you haven't had a chance, now's your last chance to get the repo.

155. 31:18

     Um, but I'll just kind of go through a few different examples here of what we talked about maybe.

156. 31:27

     Yeah. Okay. Okay, so I'll set up Phoenix, which is from Arize, uh, which is basically an observ-- an observability platform. Uh, I just did this today, so I don't know if it's gonna work or not, but we'll, we'll see.

157. 31:42

     We'll give it a shot. Uh, but basically what this allows you to do is have a bunch of observability and tracing for all the calls that are happening under the hood.

158. 31:50

     We'll see if this works. We'll give it, like, another five seconds.

159. 31:55

     Um, but it should, I think, automatically do all this stuff for me.

160. 32:00

     Yeah. So let's see. Yeah, all right, so something's up. Okay, cool. So I'll just-- I'm just gonna run through the notebook, which is a collection of different use cases, basically putting into practice a lot of what we just saw.

161. 32:14

     Feel free to jump in, any questions, anything like that. We'll start with this notebook. There's a couple of other, uh, more proper Python programs that we'll walk through afterwards.

162. 32:24

     Uh, but really the intent is a rapid-fire review of different ways that DSPy has been useful to me and others. So

163. 32:32

     load in the .env file. Usually, I'll have some type of config object like this where I can very easily use these later on. So if I'm like call, like, model mixing, so if I have like a super hairy problem or, like, some workload I know will need the power of a reasoning model like GPT-5 or something else

164. 32:51

     like that, I'll define multiple LLMs. So, like, one would be four one, one would be five, maybe I'll do a four one nano, um, you know, Gemini two point five Flash, stuff like that.

165. 33:01

     And then I can kind of intermingle or intersperse them depending on what I think or what I'm reasonably sure the workload will be, and you'll see how that comes into play in terms of classification and others.

166. 33:15

     Um, I'll pull in, in a few others here. I'm u- I'm using OpenRouter for this. So if you have an OpenRouter API key, would recommend plug- plugging that in.

167. 33:24

     So now I have three different LLMs I can work with. I have Claude, I have Gemini, I have four one mini. And then I'll ask basically for each of them who's best between Google, Anthropic, OpenAI.

168. 33:37

     All of them are hedging a little bit. They say, "Subjective, subjective, undefined." All right, great. It's not very helpful. But because DSPy works on Pydantic, I can define the answer as a literal, so I'm basically forcing it to only give me those three options, and then I can go through each of those, and you can see each

169. 33:55

     of them, of course, chooses their own organization. Um, the reason that those came back so fast

170. 34:02

     is that DSPy has ca- uh, caching automated under the hood. So as long as nothing has changed in terms of your py-- uh, your signature definitions or basically if nothing has changed, this is super useful for testing, it will just load it from the cache.

171. 34:17

     Um, so I ran this before, that's why those came back so quickly. Uh, but that's another kind of super useful, um, piece here. Let's see.

172. 34:32

     Okay. Make sure we're up and running. So if I change this to hello with a space,

173. 34:41

     you can see we're making a live call. Okay, great. We're still up. So super simple cla-- sentiment classifier. Obviously, this can be built into something arbitrarily complex. Make this a little bit bigger.

174. 34:52

     Um, but I'm basically ha-- I'm giving it the text, the sentiment that you saw before, and I'm adding that additional specification to say, okay, lower, uh, is more negative, higher is more positive.

175. 35:04

     I'm gonna define that as my signature. I'm gonna pass this into just a super simple predict object.

176. 35:11

     And then I'm gonna say, "Okay, well, this hotel stinks." Okay, that's probably pretty negative. Now, if I flip that to, "I'm feeling pretty happy"... Whoops.

177. 35:24

     Good thing I'm not in a hotel right now. Uh, you can see I'm feeling pretty happy. Comes out to eight. And this might not seem that impressive and, you know, it's, it's not really, but, uh, the, the, the important part here is that it just demonstrates the use of the shorthand, um, signature.

178. 35:43

     So I've, I have the string, I have the integer, I pass in the custom instructions, which would be in the doc string if I use the class-based me-- class-based, uh, method.

179. 35:52

     The other interesting part or, or useful part about DSPy, it comes with a bunch of usage information built in. So, um, because it's cached, it's gonna be an empty object.

180. 36:03

     But when I change it, you can see that I'm using Azure right now. But for each call, you get this nice breakdown. I think it's from LiteLLM, but allows you to very easily track your usage, token usage, et cetera, for observability and optimization a- and everything like that.

181. 36:18

     Just nice little tidbits, uh, that are part of it here and there. Let me make this smaller.

182. 36:24

     Uh, we saw the example before in the slides, but I'm gonna pull in that Form Four, uh, off of online. I'm gonna create this doc objects using attachments. You can see some of the stuff it did under the hood, so it pulled out, um, PDF plumber, it created markdown from it, pulled out the images, et cetera.

183. 36:43

     Again, I don't have to worry about all that. Attachments makes that super easy. I'm gonna show you what we're working with here. In this case, we have the Form Four, and then I'm gonna do that poor man's RAG that I mentioned before.

184. 36:55

     Okay, great. How many shares were into-- were sold in total? It's gonna go through that whole chain of thought and bring back the response. That's all well and good, but the power, in my mind, of DSPy is that you can have these arbitrarily complex data structures.

185. 37:11

     That's fairly obvious because it uses Pydantic and, and everything else. But you can get a little creative with it. So in this case, I'm gonna say, okay, a different type of document analyzer signature.

186. 37:21

     I'm just gonna give it the document, and then I'm just gonna defer to the model on defining the structure of what it thinks is most important from the document.

187. 37:29

     So in this case, I'm defining a dictionary object, and so it will hopefully return to me a series of key value pairs that describe important information in the document in a structured way.

188. 37:40

     And so you can see here, again, this is probably cached, uh, but I passed in-- I did it all in one line in this case, but I'm saying I wanna do chain of thought using the document analyzer signature, and we're gonna pass in the input field, which is just the document here.

189. 37:58

     I'm gonna pass in the document that I got before. And you can see here, it pulled out a bunch of great information in this super structured way, and I didn't have to really think about it.

190. 38:07

     I just kind of deferred all this to the model to DSPy for how to do this. Now, of course, you can do the inverse in saying, "Okay, I have a very specific business use case.

191. 38:18

     I have something specific, uh, in terms of the formatting or the content that I wanna get out of the document." I define that as just kind of your typical Pydantic classes.

192. 38:28

     So in this case, I wanna pull out the-- if there's multiple transactions, the schema itself, important information like the filing date.

193. 38:37

     Gonna define the document analyzer schema signature. Uh, again, super simple input field, which is just the document itself, which is parsed by attachments. It gives me the text and the images.

194. 38:49

     And then I'm passing in the document schema parameter, which has the document schema type, which is defined above. And this is the-- this is effectively what you would pass into structured outputs, um, but just doing it the DSPy way, where it's going to give you, um,

195. 39:08

     basically the, the output in that specific format. So you can see, pulled out things super nicely, filing date, form date, uh, form type, transactions themselves, and then the ultimate answer. [clears throat]

196. 39:21

     And it's nice because it exposes it in a way that you can use dot notation, so you can just very quickly access the, the resulting objects.

197. 39:29

     So looking at adapters, um, I'll use another little tidbit from DSPy, which is the inspect history. So for those who wanna know what's going on under the hood, inspect history will give you the raw dump of what's actually going on.

198. 39:41

     So you can see here the system message that was, uh, constructed under the hood was all of this. So you can see input fields are document, output fields are reasoning in the schema.

199. 39:55

     It's gonna pass these in, and then you can see here the actual document content that was extracted and put into the tech-- in, into the prompt, uh, with some metadata.

200. 40:04

     This is all generated by attachments. And then you get the response, which follows the specific format, so you can see the different fields that are here, and it's this kind of relatively arbitrary response, um, basically format for the, for the names, which is then parsed by DSPy and passed back to you as the user.

201. 40:24

     Um, so I can do, okay, response.document schema a-and get the, the actual result. [clears throat]

202. 40:31

     To show you what the BAML adapter looks like, we can basically do two different calls. So this is an example from, uh, my buddy Prashanth, uh, online again. So what we do here is define Pydantic model, super simple one, patient address, and then patient details.

203. 40:48

     Patient details has the patient address, uh, object within it. And then we're gonna say we're gonna create a super simple DSPy signature to say, taking a clinical note, which is a string, the patient info is the output type.

204. 41:02

     And then note, so I'm gonna run this two different ways. The first time with the Smart LLM that I mentioned before, and just use the, the built-in adapter, so I don't specify anything there.

205. 41:14

     And then the second one will be using the BAML adapter, which, which is defined there. Um, so the-- if-- I guess a few things going on here. One is the ability to use Python's, uh, context, which is the, the lines starting with with, which allow you to basically break out of what the global LLM, um, has been

206. 41:35

     defined as, and use a specific one just for that call. So you can see, in this case, I'm using the same LLM, but if I want to change this to, like, LLM Anthropic or something,

207. 41:47

     I think that that should work. Um, but basically what that's doing is just offloading that call to the other-- whatever LLM that you're defining for that particular call. And something happened.

208. 41:58

     And I'm on a VPN, so let's kill that.

209. 42:03

     Sorry, AlixPartners. Okay. [coughs] Okay, great. So we had two separate calls. One was to the Smart LLM, which is I think four one. The other one was to Anthropic. Sa-- Everything else is the exact same.

210. 42:19

     The note's the exact same, et cetera. We got the same exact output. That's great.

211. 42:24

     But what I wanted to show here is the adapters themselves. So In this case, I'm doing inspect history equals two, so I'm gonna get both of the last two calls, and we're gonna see how the prompts are gonna be different.

212. 42:38

     And so you can see here the first one, this is the built-in JSON schema, this crazy long JSON string. Yeah, LLMs are good enough to, to handle that, but, um, you know, probably not for super complicated ones.

213. 42:52

     Um, uh, and then you see here for the, the second one, it uses the BAML notation, which as we saw in the slides, a little bit easier to comprehend, um, and on super complicated use cases can actually have a measurable im- uh, improvement.

214. 43:08

     Multimodal example, same sort of thing as before. I'll pull in the image itself.

215. 43:13

     Let's just see what we're working with. Okay, great. We're looking at these various street signs,

216. 43:18

     and I'm just gonna ask it super simple question. It's this time of day. Can I park here now? When, when should I leave? And you can see I'm just passing in, again, the super simple, um, shorthand for defining a signature, which then I get about the, the vari- uh, the Boolean in this case and a string of

217. 43:38

     when I can leave. Um, so modules themselves, it's again, fairly simple. You just kind of wrap all this in a class. Question?

218. 43:48

     So does it return reasoning by default always?

219. 43:51

     Oh, good question. Yeah. So when you do-

220. 43:54

     Can you repeat the question?

221. 43:55

     Yes. So for those online, the question was, does it always return reasoning by default? When you call DSPy.chainofthought as part of the module where it's built in, it's adding the reasoning, um, automatically into your response.

222. 44:10

     You're not defining that. It's a great question. It's not defined in the signature, as you can see up here, uh, but it will add that in and expose that to you, um,

223. 44:20

     to the extent that you wanna retain it for any, you know, any reason. Uh, but that's-- So if I ju- if I change this to predict,

224. 44:29

     you wouldn't get that same response, right? You just, you literally just get that part.

225. 44:36

     Um, so that's actually a good segue to the modules. Um, so module is basically just wrapping all that into some type of replica- replicable, uh, logic. Um, and so we're just-- we're giving it the signature here.

226. 44:50

     We're saying self.predict. We're, in this case, it's just a demonstration of how it's be-being used as a class, so I'll just add this module identifier and so-some sort of counter.

227. 45:00

     But this can be any type of arbitrary business logic or control flow or any database act or whatever it might be. When this image analyzer class is called, this function would run.

228. 45:11

     Um, and then when you actually invoke it, this is when it's actually gonna run the, the core logic. And so you can see I'm just passing in the... So I'm instantiating it, the analyzer of [REDACTED:generic_id], and then I'll call it.

229. 45:25

     Great, it called that, and you can see the counter incrementing each time I actually make the call. So super simple example. Um, we don't have a ton of time, but I'll, I'll show you some of the other modules and how that kinda works out.

230. 45:38

     In terms of tool calling, fairly straightforward. I'm gonna define two different functions, perplexity search and get URL content, creating a bio agent module. So this is going to define Gemini two five as this particular module's, um, LLM.

231. 45:54

     It's gonna create an answer generator object, which is a react call. So I'm gonna basically do tool calling whenever this is called, and then the forward function is literally just calling that answer generator with the parameters that are provided to it, and then I'm creating an async version of that function as well.

232. 46:13

     So I can do that here. I'm gonna say, okay, identify instances where a particular person has been at their company for more than ten years. It needs to do tool calling to do this to get the most up-to-date information.

233. 46:25

     And so what this is doing, I'm basically looping through, um, and it's gonna call that bio agent, which is using the tool calls in the background, and it will make a determination as to whether their background is applicable per my criteria.

234. 46:38

     In this case, Satya is true. Brian should be false. Um, but what's interesting here while that's going, in a, uh, similar to the reasoning, uh, param or the reasoning object that you get back for chain of thought, you can get a trajectory back for things like react.

235. 46:55

     So you can see what tools it's calling, the arguments that are passed in, um, and the observations for each of those calls, which is nice for debugging and, and other, obviously other uses.

236. 47:07

     Um, I wanna get to the other content, so I'm gonna speed through the rest of this. This is basically an async version of the same thing, so you would run both of them in parallel.

237. 47:15

     Same idea. Um, I'm gonna skip the GEPA example here just for a second. Um, I can show you what the output looks like, but basically what this is doing is creating a data set.

238. 47:29

     It is showing you what's in the data set. It's creating a variety of signatures. In this case, it's going to create a system that categorizes and classifies different, basically help messages, um, that is part of the data set.

239. 47:43

     So my sink is broken, or my light is out, or whatever it is. They wanna classify whether it's positive, neutral, or negative and the, uh, the urgency of the actual message.

240. 47:54

     It's gonna categorize it, and then it's gonna pack all this stuff, all those different modules into a single support analyzer module. And then from there, what it's going to do is define a bunch of metrics, which is based off of the data set itself.

241. 48:09

     So it's going to say, "Okay, how do we score the urgency?" This is a, a very simple one where it's okay, it either matches or it doesn't. Um, and there's other ones where it can be a little bit more subjective, and then you can run it.

242. 48:24

     This is gonna take too long. Probably takes twenty minutes or so. Um, but Uh, what it will do is basically evaluate the performance of the base model and then apply those metrics, uh, and iteratively come up with new prompts to, uh, to create that.

243. 48:41

     Now, I wanna pause here just for a second because there's different types of metrics, and in particular for GEPA, it uses feedback from the teacher model in this case.

244. 48:52

     So it can work with the same level of model, but in particular, when you're trying to use, say, a smaller model, um, it can actually provide textual feedback. So it says, "Not only did you get this classification wrong," but it's gonna give you some additional, um, information or feedback, as you can see here, for why it got

245. 49:12

     it wrong and what the answer should have been, which allows it... You can just u- you can read the paper, but it basically allows it to, um, iteratively find that kind of Pareto frontier of how it should, uh, tweak the prompt to optimize it based off that feedback.

246. 49:27

     It basically just tightens that iteration loop. Um, you can see there's a bunch here, um, and then you can run it and see how it works. Um, but kind of just to give you a concrete example of how it all comes together, so we took a bunch of those examples from before, and we're basically going to, basically

247. 49:47

     going to do a bit of, um, categorization. So I have things like contracts. I have images. I have different things that one DSPy program can comprehend and do some type of processing with.

248. 50:04

     So this is something that we see fairly regularly in terms of we might run into a client situation where they have just a big dump of, of files. They don't really know what's in it, and they wanna find something of, uh...

249. 50:16

     They wanna maybe find SEC filings and process them a certain way. They wanna find contracts and process those a certain way. Maybe there's some images in, in there, and they wanna process those a certain way.

250. 50:28

     Uh, so this is an example of how you would do that, where if I start at the bottom here, this is a regular Python file, um, and it uses DSPy to do all those things I just mentioned.

251. 50:40

     So we're pulling in the configurations. We're setting the regular LLM, the small, and one we use for an image. As an example, Gemini might-- Gemini models might be better at image recognition than others, so I might wanna defer or use a particular model for a particular workload.

252. 50:58

     So if I detect an image, I will route the request to Gemini. If I detect something else, I'll route it to a four point one or whatever it might be.

253. 51:09

     So I'm gonna process s- a single file,

254. 51:13

     and what it does is use our handy attachments, um, library to put it into a format that we can use. And then I'm gonna classify it, and it's not super obvious here, but I'm getting a file type from this classify file, uh, function call, and then I'm doing some different type of logic depending on what type of

255. 51:36

     file it is. So if it's an SEC filing, I do certain things. If it's a certain type of SEC filing, I do something else. Uh, if it's a contract, maybe I'll summarize it.

256. 51:47

     If it's something that looks like city infrastructure, in this case, the image that we saw before, I might do some more visual interpretation of it. Um, so if I dive into classify file super quick,

257. 52:00

     it's running the document classifier, and all that is is basically doing a predict on the image from the file and, um, making sure it returns a type.

258. 52:17

     Where is this? Returns a type which would be document type. And so you can see here, at the end of the day, it's a fairly simple signature. And so what we've done is basically take the PDF file, in this case, take all the images from it, and take the first image or first few images, in this case,

259. 52:36

     a list of images, as the input field. And I'm saying, "Okay, just give me the type. What is this?" And I'm giving it an option of these document types.

260. 52:46

     So obviously, this is a fairly simple use case, but it's basically saying, given these three images, the first three pages of a document, is it an SEC filing? Is it a patent filing?

261. 52:56

     Is it a contract, city infrastructure? Pretty different things, so the model really shouldn't have an issue with any of those, and then we have a catch-all bucket for other.

262. 53:05

     And then, as I mentioned before, um, depending on the file type that you get back, you can process them differently. So I'm using the small model to do the same type of form four extraction that we saw before, um,

263. 53:20

     and then asserting basically, in this case, that it is what we think it is. Um, a contract, in this case, we're saying, um... Let's see.

264. 53:30

     I have like ten more minutes. So we can go-- we'll, we'll stop after this, uh, after this file. But

265. 53:35

     for the particular contract, we'll go, we'll create this summarizer object. So we'll go through as many pages as there are, and we'll do some, uh, basically recursive summarization of that using a separate DSPy function, and then we'll detect some type of boundaries of that document too.

266. 53:53

     So we'll say, "I want the summaries, and I want the boundaries of the document," um, and then we'll print those things out. So let's just see if I can run this.

267. 54:02

     It's going to classify it. It should, as a contract.

268. 54:12

     So is-- You're just relying on the model itself to realize that it's a city infrastructure?

269. 54:18

     Yeah. The question was am I-- I'm just relying on the model to determine if it's a city infrastructure. Yes, I mean, this is more just like a workshop, quick and dirty example.

270. 54:28

     It's only because there's one picture of the street signs. Um, and if we look in the data folder, I have a contract- Some image that's irrelevant, the form for SEC filing, and then the parking two.

271. 54:42

     Um, they're pretty different. The model should have no problem out of those categories that I gave it to categorize it properly. In some type of production use case, you would want much more stringent or maybe even multiple passes of classification, maybe using different models to do that.

272. 54:58

     Um, but yeah, given those options, at least the many times I've run it, I had no problem.

273. 55:04

     So in this case, I gave it, um, one of these contract documents, and it ran some additional summarization logic under the hood. So if I go to that super quick, um, you can find all this in the code.

274. 55:17

     But basically, what it does is use three separate signatures to basically decompose the contents of the, the, um, the contract and then summarize them up. So it's basically just iteratively working through each of the chunks of the document to create a summary that you see here at the bottom.

275. 55:37

     And then just for good measure, we're also detecting basically the, the boundaries of the document to say, okay, here's out of the thirteen pages, you have the main document, and then some of the exhibits or the schedules that are a part of it.

276. 55:51

     So let me just bring it up super quick

277. 55:58

     just to show you what we're working with. This is just some random thing I found online. And you can see, so it said

278. 56:06

     the main document was from page zero to six,

279. 56:11

     and the way-- and so we have zero, one, two, three, four, five, six. Seems reasonable. Now we have the start of schedule one.

280. 56:21

     Schedule one, it says it's the next two pages. That looks pretty good. Schedule two is just the one page, nine to nine.

281. 56:31

     That looks good. And then schedule three through to the end of the document.

282. 56:36

     And that looks pretty good too. And so the way we did that under the hood was basically take the PDF, convert it to a list of images, and then for each of the images, pass those to classifier, um, and then use that to...

283. 56:52

     Well, let's just look at the code. But basically take the list of those classifications, give that to another DSPy signature to say, given these classifications of the document, give me the structure and basically give me a key pair of, you know, name of the section and two integers, a tuple of integers that detect or that, uh, determine

284. 57:13

     the, um, you know, the boundaries, essentially. Um, so that's what that part does.

285. 57:20

     Um, [coughs] if we go back... So city infrastructure, I'll do this one super quick just 'cause it's pretty interesting on how it uses tool calls.

286. 57:30

     And while this is running, I should use the right one. Hold on.

287. 57:39

     Sorry.

288. 57:40

     Yeah.

289. 57:40

     A good question-

290. 57:41

     Yeah

291. 57:41

     ... for the second part. Like, when you generated the list of, like, my documents from zero to six, did you have, like, original document as an input or no?

292. 57:49

     No. Uh, so let, let's just go to that, uh, super quick. So

293. 57:55

     that should be boundary detector. So there's a blog post on this that I published probably in, in August or so that goes into a little bit more detail. The code is actually pretty crappy in that one.

294. 58:07

     It's, it's gonna be better here. Uh, but basically what it does is,

295. 58:14

     this is probably the main logic. So for each of the images in the PDF, we're gonna call classify page.

296. 58:23

     We're gonna gather the results. So it's doing all that asynchronously, pulling it all back, saying, okay, all these, you know, all the different page classifications that there are. And then I pass the output of that into a new signature that says, given tuple of pa-- I don't even define it here.

297. 58:40

     Given tuple of page and classification, give me this, I don't know, relatively complicated output of a dictionary, of a string, tuple, integer, integer. And I give it this set of instructions to say, just detect the boundaries.

298. 58:57

     Like, this is all very, like, non-production code obviously. But the point is that you can do these types of things super, super quickly. Like, I'm not specifying much, not giving it much context, and it worked, like, pretty well.

299. 59:10

     Like, it, it's worked pretty well on, on most of my testing. Now, obviously, there is a ton of low-hanging fruit in terms of ways to improve that, optimize it, et cetera.

300. 59:19

     Um, but all this is doing is taking that signature, these instructions, and then I call react, and then all I give it is, uh, the ability to basically self-reflect and call, um, get page images.

301. 59:35

     So it says, okay, I'm gonna look at this boundary. Well, let me get the, the page images for these three pages to-- and make sure basically that the boundary is correct.

302. 59:45

     And then it uses that to construct the final answer. And so it's really-- this is a, a perfect example of, like, the tight iteration loop that you can have, both in, um, building it, but then the-- you can kind of take advantage of the model's introspective ability, if you will, to use function calls against the data itself,

303. 1:00:04

     the data it generated itself, et cetera, to kind of keep that loop going. Question.

304. 1:00:10

     S-so under the hood, the, the beauty of DSPy then is that it enforces kind of structured output only as on model?

305. 1:00:20

     I mean, yes. I think that's probably reductive of, of, like, its full potential, but generally that's, that's correct. I mean, yes, you can use structured outputs, but you have to do a bunch of crap basically to coordinate, like, feeding all of it, feeding it into the rest of the program.

306. 1:00:38

     Maybe you wanna call a model differently or use XML here or use a different type of model or whatever it might be.

307. 1:00:45

     Um, to, to do that. So absolutely, yeah. I'm not saying this is the only way, obviously, to kinda create these applications or that you shouldn't use Pydantic or shouldn't use structured outputs.

308. 1:00:54

     You absolutely should. Um, it's just a way that once you kind of wrap your head around the, the primitives that DSPy gives you, you can start to very quickly build these types of arguably, uh...

309. 1:01:08

     I mean, uh, these are like prototypes right now, but, like, if you want to take this to the next level to production scale, you have all the pieces in front of you to be able to do that.

310. 1:01:18

     Um, any other questions? I've probably got about five minutes left. Go ahead.

311. 1:01:22

     Can you talk about your experience with using prompt optimization with DSPy and with GEPA as a prompt optimization tool?

312. 1:01:29

     Yeah. Yeah. So GEPA-- And actually, I'll pull up, uh, I, I, I ran one right before this. Um, this uses a s- a different algorithm called MiPPro. But basically, um, the optimizers, as long as you have well-structured data, so for the machine learning folks in the room, which is probably everybody, obviously the quality of your da- of

313. 1:01:50

     your data is very important. Um, you don't need thousands and thousands of examples necessarily, but as long as you have enough, maybe ten to a hundred of inputs and outputs and it-- if you're constructing your metrics in a way that is relatively intuitive and, and that, you know, accurately describes what you're trying to achieve, the improvement can

314. 1:02:12

     be pretty significant. Um, and so that time entry corrector thing that I mentioned before, uh, you can see the output of here. It's kind of iterating through. It's measuring the output metrics for each of these, and then you can see all the way at the bottom, once it goes through all of its optimization stuff,

315. 1:02:31

     you can see the actual performance on, um,

316. 1:02:38

     the basic versus the optimized model. In this case, it went from eighty-six to eighty-nine. And then interestingly, this is still in development, this one in particular. But you can break it down by metrics, so you can see where the model's optimizing better or performing better across certain metrics.

317. 1:02:53

     And this can be really telling as to whether you need to tweak your metric. Maybe you need to decompose your metric. Maybe there's other areas within your dataset or the, the, basically the structure of your program that you can improve.

318. 1:03:07

     Um, but it's a really nice way to understand what's going on under the, under the hood. And if, if you don't care about some of these and the optimizer isn't doing as well on them, maybe you can, maybe you can throw them out too.

319. 1:03:18

     So it's, it's a very kind of flexible system, flexible way of kinda doing all that.

320. 1:03:23

     Yeah. What's the output of the optimization? Like, what do you get out of it, and then how do you use that-

321. 1:03:28

     Yeah

322. 1:03:28

     ... object, whatever it is?

323. 1:03:29

     Yeah, yeah. So the output of the optimizers is basically just another, um... It's almost like a compiled object, if you will. So DSPy allows you to save and load programs as well.

324. 1:03:40

     So the output of the optimizer is basically just a module that you can then serialize and save off somewhere, or you can call it later, uh, as you would any other module.

325. 1:03:51

     And it's just manipulating the phrasing of the prompts or, like, what is it actually like? You know, what's its solution space look like?

326. 1:03:57

     Yeah, yeah. Under the hood, it's literally just iterating on the actual prompt itself. Maybe it's adding additional instructions. It's saying, "Well, I keep failing on this particular thing, like not capitalizing the names correctly.

327. 1:04:09

     I need to add in my upfront criteria in the prompt an instruction to the model to say, you must capitalize names properly." And Chris, uh, who I mentioned before, has a really good way of putting this, and I'm gonna butcher it now, but, like, the optimizer is basically finding latent requirements that you might not have specified initially

328. 1:04:27

     up front. But based off of the data, it's kind of like a poor man's deep learning, I guess. But, like, it's learning from the data. It's learning what it's doing well, what, what it's doing not so well, and it's dynamically constructing a prompt that improves the performance based off of your metrics.

329. 1:04:42

     And is that, like, LLM guided? Like, is there, like-

330. 1:04:44

     It's all-- Yeah

331. 1:04:45

     ... thinking about, like, capitalization and-

332. 1:04:47

     Yeah, yeah. Question being, is it all LLM guided? Yes. It's p- particularly for GEPA, it's using LLMs to improve LLM's performance. So it's using the LLM to dynamically construct new prompts, which are then fed into the system, measured, and then it kind of iterates.

333. 1:05:03

     So it's using AI to build AI, if you will.

334. 1:05:06

     Thank you.

335. 1:05:07

     Yeah.

336. 1:05:08

     Can I ask you this question? Why is the solution object not just the optimized prompt?

337. 1:05:12

     Why is the solution object not what?

338. 1:05:14

     Not just the optimized prompt. Why are you using things like-

339. 1:05:17

     Oh, it absolutely is. You can get it under the hood. I mean, you can... The question was, why don't you just get the optimized prompt? You can, absolutely. Um-

340. 1:05:26

     But what else is there besides the optimized prompt?

341. 1:05:30

     The, the-- So what else is there other than the prompt? The DSPy object itself, so the module, the way things, um...

342. 1:05:39

     Well, we can probably look at one if we have time. Um.

343. 1:05:43

     If I could see a dump of what gets, you know, what is the optimized state, that would be interesting to look at.

344. 1:05:47

     Yeah, yeah. Sure. Let me see if I can find one quick. Um, but fundamentally, at the end of the day, yes, you get an optimized prompt, a string that you can dump somewhere if you, if you want to.

345. 1:05:57

     Um, actually, um...

346. 1:06:03

     There's a lot of pieces to the signature, right? So it's, like, how you describe your fields-

347. 1:06:07

     Yes

348. 1:06:07

     ... and the doc string.

349. 1:06:08

     This is a perfect segue, and I'll, I'll conclude right after this. I was playing around with something I was-- Well, I was playing around this thing called DSPy Hub that I kind of created to create a repository of optimized programs.

350. 1:06:22

     So basically, like, if you're an expert in whatever, you optimize an LLM against this dataset or have a great classifier for city infrastructure images or whatever, kind of like a Hugging Face, you can download something that has been pre-optimized.

351. 1:06:39

     And then what I have here, this is the actual loaded program. This would be the output of the optimized process, or it, it is. And then I can call it As I would any- anything else.

352. 1:06:51

     And so you can see here, this is the output, and I used the optimized program that I downloaded from, from this hub. And if we inspect maybe the loaded program,

353. 1:07:04

     you can see under the hood it's a predict object with a string signature of time and reasoning. Here is the optimized prompt ultimately.

354. 1:07:16

     This is the output of the optimization process, this long string here.

355. 1:07:21

     Um, and then the various, uh, specifications and definitions of the inputs and outputs. So-

356. 1:07:28

     Have you found specific uses of those? Like, uh, to his question, like, what is it? What can you do with that?

357. 1:07:33

     It's up to your, it's up to your use case. So if I, if I have a ... So a document classifier might be a good example. If in my business I come across, whatever, documents of a certain type, I might optimize a, a classifier against those, and then I can use that somewhere else on a different project

358. 1:07:52

     or something like that. So out of 100,000 documents, I wanna find only the pages that have an invoice on it, as an example. Now, sure, 100% you can use a typical ML classifier to do that.

359. 1:08:05

     That's great. Uh, this is just an example. But you can also theoretically train or optimize a model to do that type of classification or some type of generation of text or what have you, which then you have the optimized state of which then lives in your data processing pipeline, you know, a- and you can use it for

360. 1:08:26

     other types of purposes or give it to other teams or whatever it might be. So it's just up to your particular use case. Um, something like this, like hub, who, uh, maybe it's not useful because each individual's use case is so hyper-specific, I don't really know.

361. 1:08:41

     But, um, yeah, you can do with it kind of whatever you want. Probably last question. Yeah.

362. 1:08:49

     Um, is generally, you know, like using D- DSPy something where people kind of do replays just to optimize their prompt, or is there a way to sort of do it like in real time given delayed feedback loop?

363. 1:09:02

     Like what I mean by delayed is, okay, ChatGPT gives you your answer, and you can thumbs up or thumbs down.

364. 1:09:09

     Mm-hmm.

365. 1:09:09

     Maybe, you know, that thumbs up comes, you know, like ten minutes later, 30 minutes later, a day later, maybe.

366. 1:09:15

     So is the question more about like continuous learning-ish? Like how would you do that here?

367. 1:09:20

     To do like kind of more continuous learning.

368. 1:09:22

     You can be the judge, so you can-

369. 1:09:26

     Well, but how are you feeding back delayed metrics to optimize it?

370. 1:09:31

     Why, why would it need to be delayed?

371. 1:09:33

     Because, you know, usually maybe the feedback is from the user, right? Like, it could be delayed.

372. 1:09:39

     I mean, you reconstruct your data set.

373. 1:09:42

     Yeah, well, then you-

374. 1:09:43

     Yeah, that's right. It, you, it would basically be added to the dataset, and then you would use the latest optimize and just keep, keep optimizing off of that.

375. 1:09:51

     You have your own truth dataset.

376. 1:09:52

     That's right.

377. 1:09:53

     You will collect the outputs of your optimization and feed it back, and the, the loop repeats.

378. 1:10:00

     Yeah, but at that point you're kind of doing offline optimization, right?

379. 1:10:03

     Yeah. Yeah.

380. 1:10:03

     Yes.

381. 1:10:04

     Yes.

382. 1:10:04

     But I'm, I'm asking can you do this online where with delayed metric feedback?

383. 1:10:11

     If you're a good, if you're a good enough engineer, you probably do it, but-

384. 1:10:14

     Hacks into-

385. 1:10:14

     Uh, I'm not, I'm not recommending replacing ML models with like optimized DSPy programs for particular use cases. Maybe like classification's a terrible example, I recognize that. But for other, other areas, other th...

386. 1:10:27

     In theory, yes, you know, you could do something like that. Yes.

387. 1:10:32

     But for, for particular LLM tasks, I'm sure we all have interesting ones. If you have something that is relatively well defined where you have known inputs and outputs, it might be a candidate for something worth optimizing.

388. 1:10:46

     If nothing else, to transfer it to a smaller model to preserve the level of performance at a lower cost. That's really one of the biggest benefits I see.

389. 1:10:56

     All right, last, last question.

390. 1:10:59

     I've heard that, uh, DSPy is can be kind of expensive because you're doing all these LLM calls.

391. 1:11:04

     Mm-hmm.

392. 1:11:05

     Um, so I was curious your experience with that, and maybe relatedly, like if you have any experience with like large context in your optimization dataset and some recommended ways of shrinking those.

393. 1:11:17

     Yeah. So the question was, d- can DSPy be expensive? And then for large context, kind of how have you seen that? How have you managed that? The expensive part is totally up to you.

394. 1:11:27

     If you call a function a million times asynchronously, you're gonna generate a lot of costs. I don't think DSPy necessarily... Maybe it makes it easier to call things, but it, it's not inherently expensive.

395. 1:11:40

     It might, to your point, add more content to the prompt. Like sure, the signature is a string, but the actual text that's sent to the model is much longer than that.

396. 1:11:51

     That's totally true. I wouldn't say that it's a large cost driver. I mean, it, again, it's ultimately, it's more, more of a programming paradigm, so you can write your compressed adapter if you want that like, you know, reduces the amount that's sent to, to, uh, to the model.

397. 1:12:08

     Um, in terms of large context, I, I, it's kind of the same answer I think in terms of if you're worried about that, maybe you have some additional logic either in the program itself or in an adapter or a part of the module that keeps track of that.

398. 1:12:21

     Maybe you do some like context compression or something like that. There's some really good talks about that the past few days obviously. I have a feeling that that will kind of go away at some point, w- either context windows get bigger or context management is abstracted away somehow.

399. 1:12:38

     I don't really have an answer, just that's more of an intuition. Um, but DSPy again, kinda gives you the tools, the primitives for you to do that should you choose, um, and kind of track that state, check that management over time.

400. 1:12:50

     So I think that's it. We're gonna get kicked out soon, so thanks so much for your time. I really appreciate it. [audience applauding] [upbeat music]