Optimizing LLMs in Insurance with DSPy: Beyond Manual Tuning

Talk transcript

1. 0:00

   [upbeat music] Welcome to, uh, Optimizing LLMs with DSPy: Beyond Manual Tuning.

2. 0:19

   And, um, I hate to break it to you, but we're all bad AI engineers. [laughs]

3. 0:26

   Yes, you are. [laughs] So why, why, why is that? Um, because we don't care enough about solving real world problems. Um, instead we all love keep tinkering around with tools and hotbot QA data sets.

4. 0:42

   But let-- bear with me, there's a way out. So let's start with first principles thinking again and reminding ourselves what we are. We are engineers, you know? We have to reconsider the thing we are doing these, these days, which is like prompt engineering, and start programming AI systems again.

5. 1:01

   I mean, maybe you've seen this, uh, already, like left we have this huge neural network model, but usually neural networks on themselves and being like language models or any other, uh, AI model are not useful on their own, but are only useful if you're considering them, uh, in a huger system.

6. 1:19

   So the struggle is real when it comes to use cases in the LLM space. We're all thrilled by the excitement around capabilities of ever newer models. Uh, but we have only a vague idea of what we really want to achieve.

7. 1:33

   We throw a bunch of tools of them at the not well-defined problems. We write some handwritten prompts, use some prompt libraries in the hope that that will lead us to a magical solution of our problems.

8. 1:43

   I know myself how much I love to tinker around with all these tools and see how they work. But here I'm telling you to stop hoping that using API just works and solves all your problem.

9. 1:55

   So let's imagine you're an AI engineer, and you start your day by brewing some specialty coffee, sitting down, delving into focus mode. And you fiddle around with a bunch of prompts and, uh, try the next tool to guide system output to-- for JSON.

10. 2:11

    So you evaluate the system with your most loved metric. The answers are all like, "Looks good to me" at ten. So I have just look at all of the outputs, and I was like, "Yeah, that's fine enough.

11. 2:23

    I can, I can do this." But is this really the end? And, uh, is it really that we fiddle around with our prompts until they seem stable enough? So the question is: How do we become like great AI engineers again?

12. 2:36

    I think we should stop, start, uh, stop working on toy problems, all these hotpot QA data sets and everything, and start working on real problems again. So why am I telling you this?

13. 2:50

    So a company dude like me is going to explain to you how to solve real world problems. I mean, yes, uh, we have to. We have to solve real world problems every day because we can, and it's really about the data and helping our customers every day.

14. 3:06

    So we are data-driven company since, like ever. We don't have any assets. I mean, insurance is what are they? They are selling you a good feeling. You give us money, and, uh, we give you money back in case something happens to you.

15. 3:18

    Uh, so there's no real asset, but we always have been a data, data-focused, um, company and only providing non-touchable assets. So, um, to, to cape, to keep with the ever-- So one of the huge problems we are facing today is climate change, and with that, the ever-increasing like incoming, uh, claims, for example, and more problems

16. 3:43

    that we are seeing with our customers. So there's a huge rise of, um, of labor we have to do. And so that's the reason like why AXA Germany started the Data Innovation Lab in two thousand and seventeen.

17. 3:56

    And we are like a group of machine learning engineers working with the data scientists of the business units, and we are trying like to work cr-cross-colla- uh, collaboratively with the business units to, to really make an impact on the customer service agents to help them make the customers happy every day.

18. 4:15

    So I know you think you, you are like in Silicon Valley and, uh, you're like these amazing speedboats. You're like super fast. You can change your ideas quickly every day.

19. 4:25

    And we are like rather tankers. And in a way, you are right. So we are these tankers, but more like icebreakers often. So it's like slowly, but we are like making progress forward and, um, like pushing things forward.

20. 4:39

    And you can see this also in, uh, things that just made the announcement on Monday. So in the forefront already in two thousand twenty-three, the beginning of the year, we started collaborating with OpenAI and created this Secure GPT GenAI platform internally, uh, which is hosted on Azure platform.

21. 4:57

    And even now, we are now like announcing, uh, on Monday that we're working together with Mistral AI, so our internal GenAI platform gets like more usable every day and like especially not comprising our data security or anything our customers don't want to.

22. 5:14

    So that's super cool. So that's the reason like the tanker, uh, has some impact and, uh, makes fun. But

23. 5:23

    w-what does it, what does it take us to, to get bad-- good engineers? So h-how do we make this path, um, better? So let's me, let's take me, let me take you on a path on imaginary real world problem insurance.

24. 5:38

    So we're creating a customer-facing chatbot to help them navigate our insurance fair terms and conditions. So how would we tackle this? I mean, yeah, we kind of have like everything, but the main issue is like you really need to define your problem really clearly.

25. 5:57

    So we went to the domain experts and let us guide through the problem space, especially crafting examples. And this is especially tr-- important for prompt engineering, which are either super simple or like super hard to solve, so we can understand the, the domain space better.

26. 6:14

    And, um, that-

27. 6:16

    Guided us to, to learn a lot about what is expected in our production system in the end. So we came up with a dozen of examples and immediately started prompting our internal Secure GPT.

28. 6:26

    And, um, uh, we, we tried to come up with some useful prompts. But prompt engineering is like quite of black magic, so we didn't use phrases like, "If you don't output JSON, uh, I'm going to quit," or, "Please, please, please output JSON."

29. 6:41

    And, uh, [chuckles] there are these prompts where you fiddle around, you try to make your output better, cleaner, more structured. And, um, one, one way, uh, to do this is, uh, what Jason Lu already proposed last year, um, was to, to hijack the prompt and like say, "Okay, I'm an assistant here," and we start the prompt by just

30. 7:06

    giving in triple back ticks and JSON, and that then the, the model, uh, push forward into, uh, just creating JSON. But, uh, there are also other, a dozen other of prompting techniques like chain-of-thought, zero- few-shot learning, uh, tools like LangChain, Guardrails, DSPy, Instructor, and to help make use of these techniques,

31. 7:31

    um, to make the output safer, more deterministic, more structured, more resilient, and optimized for maybe even arbitrary metrics. And so we, we fiddled around with all these tools and played around, and to be honest, that was a lot of work.

32. 7:47

    Like finding a good prompt is hard and error-prone, as these prompt templates are often hard-coded and the usage is quite intransparent. So please don't fall into these tools will solve the problem for me trap.

33. 8:02

    So get your hands dirty, inspect what these tools are really doing for you, and I highly recommend, um, this amazing blog post from Hamel Hussein, so

34. 8:14

    which is called, uh, Please Show Me the Prompt. [chuckles] It has another title, but bear with me. So, um, uh, the idea is just to use MITM proxy and to really inspect like what are all these tools using I've been mentioning before, what are re- they really sending?

35. 8:32

    So this is about first principles thinking. I'm not advocating here to, to use tools this and that and there, and I will talk definitely about DSPy, but

36. 8:44

    please do first principles thinking. Look at what these tools create, what, what kind of prompt templates they use, what, what they send to your LLMs. Don't just believe that it will happen or it will do something great.

37. 9:01

    So one issue we had been experiencing a lot was sensitivity to minor prompt changes. So like very small prompt changes led to very high problems in the end. So what we, what we had was, what we tried to do was using chaining libraries to, to fix these issues and throwing more and more error handling code at that.

38. 9:22

    So, um, we tried to make this a bit more achievable and but this lead led into the end to an overly complex and fragile system. So we had the gut feeling that was by no means something we want to put in production, but we could show that the task was at least achievable, um, or could serve as

39. 9:43

    a baseline. So before continuing further, we knew we were missing some key pieces to fully understand what's happening. And, uh, apart from inspecting our calls, that was we were not logging any traces or something.

40. 9:56

    Um, for that, we started using an open source library called Arise Phoenix, and, uh, that definitely helped a lot, um, in understanding like which calls are bundled together, which helped to see what,

41. 10:12

    what, what is doing, what is happening behind the curtains.

42. 10:17

    So a- again, here, once again, I'm advocating for first principles thinking. You cannot improve anything what you don't measure. And we were still, at that point in time, we were still at the level looks good to me at 10, right?

43. 10:34

    Which was our main driver. So we need to definitely write some basic evaluations. So preparing the data to match. So we-- If you want to evaluate something, you need to not only have your input data, but also to have some labels.

44. 10:48

    And that was the hard part, to be honest. That took us some several months to really get scrape the production data we were already getting and, uh, looking at it and preparing it, so we had like clean input-output pairs.

45. 11:00

    And one main thing is, please don't stumble upon this, data leakage is still a thing. So it doesn't matter if you have like the coolest APIs. If you're using data which is, has compromised data leakage in it, and you already can deduce the output label from your input data, that's really bad.

46. 11:22

    So if you're creating your, uh, your evaluation data sets, really, really, really pay attention to, uh, to not, um, to not, uh,

47. 11:37

    destroy your, uh, evaluation. So there's another blog post I would definitely recommend from, uh, Hamed Hussein, uh, which is, uh, evals, called E- Evals is all you need or evals for your, um, for your, uh, LLMs.

48. 11:53

    Please get into that. That is amazing and will so much help you. So feeling quite confident. So this is the base setting, and I encourage you, please don't do anything with DSPy or any other tools when you don't have like this base setting, when you don't understand your data, when you don't have enough examples.

49. 12:12

    Because otherwise, like what? You, you're just saying prompt engineering, DSPy, profit? So what, what's, what's the case? The please, please don't do this. And To, to be fair, I, I just went into DSPy because I met Omar last year at, uh, [REDACTED:location] in, uh, in November.

50. 12:29

    And I was like, "Okay, cool. So we are now giving this a chance." But

51. 12:35

    please bear with me. This has a steep learning curve. So I'm-- I love the idea of DSPy. I love how, how it enabled us to...

52. 12:48

    Now we have this overly complex system with these huge prompts and huge error handling code and everything, and we all thought, "Okay, we're going to throw, uh, DSPy at it, and it's going to be super easy."

53. 12:58

    But to be honest, uh, the s- the learning curve is quite steep, and it's maybe a bit too much overall. So this is not a 100% recommendation, go to DSPy.

54. 13:08

    But, uh, please give it a try if you can.

55. 13:13

    Okay. So without DSPy, we had to break down our problems into step. We have to prompt well. Like, each step had to work well in isolation. We had to tweak the steps to work together.

56. 13:25

    We had to generate examples to tune each step and to use these examples to maybe even fine-tune some smaller models. Uh, and what we now could do with DSPy was taking this base idea, and you need to do that.

57. 13:39

    I encourage you, please f- uh, is-- tear apart your problem into separate modules, and we're talking about modules in a second. Um, otherwise, you won't be able to take advantage of DSPy.

58. 13:54

    And also, DSPy is not going to take-- to be able to take advantage of your, uh, of your DSPy program because it doesn't get any hints, any, any signal on how to improve, uh, the program you're providing it.

59. 14:10

    So don't come up with one huge, uh, ReAct when we did this, and it's bad. Uh, but rather split. Try to understand, um, how your flow of your program is.

60. 14:21

    Like, do you, do you have a retriever step first? Uh, do you have, uh, do you want to have, like, multi-hop, uh, question answering? Do you want to

61. 14:32

    bring together different pieces and answer them in one question in the end? This, this is really important. So what do you do? You have to separate your program into modules, which are kind like the prompts and weights.

62. 14:46

    You have to add optimizers to tune your prompts and weights, giving a metric. And for us, the metric was quite clear. Um, for the chatbot, we needed to find, like, the right answer.

63. 14:58

    And, um, DSPy compiles the same program then into different instructions, few-shot prompting, and fine-tuning. So instead of prompting one language model with a huge prompt, as I told you, uh, please break it apart.

64. 15:12

    So there's this analogy to neural networks for maybe for you to better understand how DSPy works. Um, we have this separation into an init and, and forward step, where you define in your init, um, how like in a, in a DSP-- um, how you find like a convolution or dropout layers in PyTorch.

65. 15:34

    You use then things like chain-of-thought or ReAct, and, uh, you describe in your forward pass how you do the things you want to do, like retrieving and other stuff.

66. 15:47

    Okay. So what were the key learnings? Um,

67. 15:53

    so if you want to do this for, like, German texts, answer exact match, answer passage match only work for English text. So metrics are really not working out of the box.

68. 16:02

    We have-- You have to come up with your own metrics. You have to create your own metrics. Um, if you want to use, um, something like ReAct, so there's nothing like a typed ReAct.

69. 16:12

    So we-- DSPy comes with these typed, um, typed things like type predictor, um, or type chain-of-thought. But if you want to have something more specific for your case, you need, uh, you need to come up with your own, uh, module and improve that.

70. 16:31

    And evalu-evaluation is also quite fixed on English language, to be honest. Uh, in case you want to use DSPy for German optimization, you need also to create like your own evaluators, especially for the exact and passage match.

71. 16:46

    Okay. So we tried to come up with one big model, and, uh, we found out like this is really not, not working.

72. 16:56

    Okay. Whoops. Sorry. Okay. So what are your key takeaways? Uh, you should definitely write prompts by hand and start with that.

73. 17:12

    Otherwise, uh, you cannot say if your task is achievable at all. Uh, you can't improve what you don't measure. Avoid metrics like looks good to me 10 without any annotated data or clear goal metric.

74. 17:24

    There's no clear objective to validate if the results are getting even better. You definitely should write br- basic evaluations. Start with some small evaluations like regex or string comparisons, and then only then use techniques like LMS Judge.

75. 17:38

    Please don't overcomplicate things. It's already super complicated and, uh, just make very, very small adjustments.

76. 17:50

    So, okay. Um, what do you take home from this? Um, so am I really thinking that you've learned how to use DSPy like from a twenty minutes corporate ti- guide talk?

77. 18:03

    If yes, cool. You're amazing. Wow. [laughs] Uh, but to be honest, um, if you want to learn DSPy, go to all these learning resources from Connor Shorten, from the, uh, DSPy website it-itself.

78. 18:16

    Um, there are all these people like presenting it in a lot better fashion than I did. But what, what I wanted to do is like I wanted to spark some interest in you.

79. 18:27

    The same spark like I got when I, when I saw this conference last year where I learned all the people, uh, where I learned all the guys who created these toolings.

80. 18:37

    And I want to spark that same curiosity in you and to see if you can get like better AI engineers by using not only the tools, but also first principles thinking.

81. 18:51

    And I think we are meant to become like great AI engineers. So this conference is about you getting inspired. Get out and find like some good problems where you can try this out and which are like worth solving with AI that wasn't possible before. [upbeat music]