Productionizing GenAI Models – Lessons from the world's best AI teams

Talk transcript

1. 0:00

   [upbeat music] I guess as we're getting started, this talk is about productionizing AI.

2. 0:18

   But I'm curious, um, I keep asking this question and it keeps changing. How many of you folks have, um, LLM applications in production in your company?

3. 0:28

   Wow. That's probably the highest I've ever seen, more than seventy percent.

4. 0:34

   So why did you come to this talk? You know, [laughs] you figured it out, right?

5. 0:38

   It's not going great.

6. 0:39

   It's not going great? [laughs]

7. 0:42

   Yeah.

8. 0:46

   All right. You wanna put it in slideshow mode?

9. 0:51

   It is in slideshow mode.

10. 0:51

    Oh, it is in slideshow mode? Okay. Hey, you're showing my IP to the whole, uh, audience. [laughs]

11. 1:02

    Um...

12. 1:05

    Yeah. You might ask if they're custom solutions or, or purchased, uh, LLM solutions.

13. 1:09

    Oh, good question. Yeah. How many, uh, is, uh, custom solutions?

14. 1:15

    Interesting. That's maybe thirty percent. And how many purchased solutions?

15. 1:20

    Wow. Way, way less. [laughs] What are you, what are you purchasing?

16. 1:24

    Uh, well, I put like Copilot, uh, for from GitHub in that category of using, uh-

17. 1:28

    Oh, okay. Well, if you put Copilot in that category, how many have purchased solutions?

18. 1:33

    It's gotta be most, right? Interesting. Interesting. Thanks for the... [laughs] Any more... Should we crowdsource some more questions? No, we can begin. Awesome. Um, all right. So I wanted to talk, um, briefly about our experience with our customers productionizing GenAI models.

19. 1:51

    But I want to give you a little bit of context, first of all. Um, this is clearly preaching to the choir, but it's interesting to see that democratization of AI is here.

20. 2:00

    I don't know, how many people were working in AI ten years ago?

21. 2:04

    Nice. Fifteen years ago?

22. 2:06

    ML counts.

23. 2:06

    So yeah, ML totally counts. Hundred percent ML counts. Um, ML definitely counts as a type of AI, absolutely. Um, you know, I think people talked about the democratization of AI, you know, back when AI was kind of a lame, um, term to use.

24. 2:20

    But, uh, I think we thought that it might take the form of, you know, something like AutoML or more, you know, graphical user interfaces. But clearly democratization of AI has happened in a way that we didn't expect, clearly through these, these kind of chat and conversational interfaces and LLMs.

25. 2:34

    We now see AI in literally every company that we talk to in some way. And not just purchasing solutions, although there's, I think, probably every company in this room purchases what I would call an AI solution.

26. 2:45

    We also see quite a lot of custom solutions, and probably every Fortune five hundred right now is investing in custom AI solutions. More LLMs, more GenAI, but I would say most also some traditional ML.

27. 2:59

    And it's moving into production, right? You see it in this audience, you see seventy percent of this audience. But everywhere I go to give a talk, even with companies that you wouldn't necessarily think of as the most forward-thinking, they are starting to have GenAI applications in production.

28. 3:11

    So people keep asking me, "When is GenAI gonna move into production?" It really has moved into production, right? There's a lot more that could happen, but GenAI is in production right now in a lot of applications that you use today.

29. 3:24

    But I think the challenge everybody has, and people keep saying it, important to say, AI is so easy to demo and so hard to productionize. You see this with a lot of AI apps getting released.

30. 3:33

    There's something about AI that makes CEOs stupid. It makes them stop listening to their customers. It makes them willing to put things into production that are absolutely terrible. And I think the fundamental reason is that it's so easy to make an incredibly compelling demo.

31. 3:49

    I get these all the time from my own team. I get them from startups, you know, pitching me on potentially angel investing. You can make just an absolutely astonishing demo, and it's so much further from production than we're used to.

32. 4:00

    Even with software, which has this characteristic, AI is so much more easy to demo and so much harder to productionize. So let's talk about how to productionize it. I'll, I'll state I come from Weights & Biases.

33. 4:12

    This is a company I started that I feel very proud of. We've built an AI developer platform that helps develop AI engineers with ML, uh, applications or with GenAI applications.

34. 4:24

    And we started this company because we saw this massive tools gap for AI ops.

35. 4:31

    We now see that we have three types of customers. We see foundation model builders, which we've long served since the beginning. We see a bigger group of AI engineers that are doing a blend of ML applications and GenAI applications.

36. 4:44

    And we see a new type of audience for our software in software developers that are now getting used to the fact that AI applications are so easy to demo and so hard to productionize, and trying to figure out how to productionize those applications.

37. 4:58

    What's exciting for us, and exciting for everyone, I think, is that the software developers are so much more numerous than ML engineers and foundation model builders, right? Now that companies can use software developers to do what traditionally would've taken a custom model, it opens us up to do so many more AI applications. [clears throat]

38. 5:18

    And from our perspective, this is a really exciting development because the market's so much bigger, and that's why we see a whole LLMOps tools track at this conference and so many companies popping up to help with this problem.

39. 5:29

    I feel really proud, and I always wanna say, we work with nearly all the foundation model companies. They have special needs, but they do push our scale. So a- almost all the LLMs that you would've used, from OpenAI's GPT to Mistral to Llama, these, these were built using the Weights & Biases platform.

40. 5:44

    So we have years of experience dealing with, with large scale. But it's also important to say almost all of our customers, and we, we have thousands of customers, are not foundation model builders, right?

41. 5:53

    They're, they're companies in, in nearly every industry. And the really fun part about being in the AI space right now- Is that there's applications to every industry. So we see healthcare, we see ag tech, we see it in manufacturing.

42. 6:04

    I remember when I, when I came out of school and wanted to get into the AI industry, it was really kind of ranking search results or, or going to Wall Street or targeting ads, right?

43. 6:12

    Which are, aren't the most exciting applications. Now, it's things that you can touch and feel. It's things that you can explain to your parents. This is, this is incredibly exciting development.

44. 6:21

    But what makes this hard, and I, I've thought a lot about this, what makes it hard, and, and, and what makes kind of AI engineering a different type of process with a different set of tools needed, because every software company has now launched an AI tool version of itself, right?

45. 6:35

    They, they call me up every day and they say, "You know, what do, what do you think? We're, we're launching, you know, AI version of observability. We're launching AI version of CI/CD.

46. 6:43

    We're launching AI version of, um, you know, code versioning."

47. 6:48

    Why don't those take, right? Why do we need a new set of tools? Why is Weights & Biases able to compete against companies that are, you know, much bigger and have lots of experience?

48. 6:55

    And I think it's fundamentally that the software development process, which I love, is a linear process.

49. 7:01

    You, you, you add features, you add code, and generally things improve over time. And the AI process is experimental. Almost all the things that you do when you're using an LLM is, is trying it, seeing what happens, right?

50. 7:15

    And it's, it's fundamental non, fundamentally nondeterministic. You don't know what's gonna happen. You're never gonna have a CI/CD test that's meaningful where 100% passes. So instead of kind of adding things all day long and, like what you do when you write code, you're doing lots of experiments.

51. 7:28

    I think this is a fundamentally different workflow, and what that means is that, you know, when you develop software, your code is your IP. Companies recognize this, and they take a lot of, uh, pain to protect their IP.

52. 7:39

    But when you're developing AI, it's the learning that's your IP. It's not the model that you build, right? The model's kind of an outcome, right? The prompt that you discover is kind of an outcome.

53. 7:48

    But the things that you learned along the way, all the prompts that you tried, all the workflows that you tried that didn't work, that's really the learning. That's really the IP that you have.

54. 7:56

    And if you're not saving that, when the person that figured out walks out the door, your IP walks out the door with them because you're not gonna be able to iterate further from there.

55. 8:04

    And so what that means is reproducibility matters, right? You don't really save your IP, you don't save your learnings if you can't reproduce your learnings, right? And this is a obvious thing.

56. 8:12

    I think everybody asks for reproducibility. We've always talked about a reproducibility crisis in AI. But reproducibility is incredibly hard. It's incredibly painful to keep track of everything so you can reproduce your stuff, especially with these nondeterministic endpoints.

57. 8:25

    And you see a reproducibility crisis in many fields, not just software.

58. 8:30

    And so my perspective is what you should be doing is tracking everything, and the tracking needs to happen with outside processes. You can't rely on humans to write everything down.

59. 8:39

    No matter how much they want to, they're gonna forget. You need to track everything in the background passively to have real reproducibility. And the real reason that you wanna do this is not just protecting your IP, but because now you can collaborate, right?

60. 8:50

    Now someone else can pick up the learnings that some engineer had.

61. 8:55

    And when you can collaborate well, you can iterate faster. And the iteration time is really the key to bringing things to market, right? If you wanna go from demo to production quickly, you need to bring down that iteration time, you need to have good collaboration, and you need speed.

62. 9:09

    So that's the real ROI, I think, of all these tools that we might be building. Now, we work with lots of enterprises and I, I, I wanted to give you an enterprise example, but more than that, I wanted to give you a real world example that, that mattered to me.

63. 9:22

    So this is very similar to what many of our enterprise customers are doing, but this is actually a project that I did, um, in a, in a, a few weeks that, that I took off after my second child was born.

64. 9:34

    Um, and actually, this is my first kid, um, Matilda, and, um, she was really demanding my attention after my, um, baby was born, and we were talking a lot, and she was actually, um, asking Alexa to play her favorite song.

65. 9:46

    And it was really interesting. I mean, Alexa's an interesting product. I, I really love it. My daughter is kind of connected to it, um, talks to, talks to her a lot.

66. 9:53

    But she was really kind of shocked when she asked Alexa to play her favorite song, and Alexa didn't know what her favorite song is because she asks Alexa to play "Baby Shark," like literally like five times a day.

67. 10:01

    So it's like there's no data science needed to know what the favorite song is. [laughing] But Alexa, um, couldn't f- couldn't, didn't know this. And I think there's kind of data privacy reasons that Alexa doesn't keep this, but it's kind of remarkable with all these amazing applications out there to see Alexa not able to do it.

68. 10:15

    And so, uh, my daughter was asking me, "Hey, could you build an Alexa?" And I was like, "I think maybe I could, actually, with, with LLMs today." So I kind of looked around with the open source Alexa alternatives where there's something called Mycroft that's, um, that's out there, but, but, um, kind of out of date.

69. 10:29

    Doesn't really use LLMs. And then my neighbor actually was showing me that he got, um, a seven billion, um, model, uh, I think Llama 2, running on a Rock Pi, which is like a $200 device, similar price to Alexa.

70. 10:40

    So I thought, "Okay, if you have that, maybe we could build an architecture that would, um, kind of do the things that Alexa does." So built a library of skills, um, things that we ask Alexa to do, like play music, um, ask what the weather is, do math problems, um, check the news, things like that.

71. 10:56

    And these skills are really simple, right? So you know, like a- getting what the weather is is like a five line thing that actually, um, Copilot can generate instantly and works phenomenally well.

72. 11:05

    There's APIs here. Uh, the challenge is s- is actually understanding the speech and then getting the natural language into an API call. So kind of a simple architecture here, right?

73. 11:14

    So the weather comes in. You ask what's the weather in Boston. It goes through an on-device LLM to translate it to kind of Python looking code that says, "Weather location equals Boston."

74. 11:23

    Has to look like that because it's gonna call that function on the weather skill that I wrote that calls a, a weather API.

75. 11:30

    Um, yeah, I think Llama, I think, and Whisper is a fantastic open source stack here. Llama.cpp is one of the many options to, to run Llama on, um, cheap hardware.

76. 11:39

    There's vLLM and others, um, and lots of companies that'll do it for you. I kind of wanted to do it myself, so Llama.cpp was a great choice. Um, and you can run large LLMs on a MacBook in, in real time.

77. 11:50

    Um, maybe I'll skip the demos here. Not sure the audio is working. Um, the... What you will, if you, if you actually do try to do this yourselves, and a lot of our customers do try to do this themselves and they discover the same thing, latency really matters a lot, right?

78. 12:04

    Because you're actually transcribing audio and then you're running the aud- you're running the transcription through an LLM- To, um, make this function call. And so if that takes longer than a couple hundred milliseconds, it's incredibly painful.

79. 12:16

    So you need to go kind of small model here, and you need to, to run it fast on, on hardware. So that's why we went with, um, Llama 2 at the time, and put it in a default prompt, and it didn't work.

80. 12:26

    0% accuracy. And so every kind of customer goes through this journey to try to improve the accuracy or whatever the real kind of user-facing metric is. I'll tell you briefly about my journey to improve this.

81. 12:37

    Um, you know, first thing I did was I did some prompt engineering and just kind of common sense to make the prompt better, kinda lay out what functions are available.

82. 12:45

    Um, the model kind of gets closer, in a sense. Now it's saying, "Call colon weather location equals Boston," but still actually not in any kind of format that I can run.

83. 12:52

    Here, I need really exact accuracy because I'm generating, um, code, function calls. Switched to Llama 2 Chat. Um, there's so many models to try these days. Um, Llama 2 Chat was trained on conversations, performed better in this case, and got the accuracy up to 11%.

84. 13:09

    Still not possible to ship this, um, but I kind of looked at the errors it was making. Looking at the errors, incorporating feedback, got the accuracy up to 75%.

85. 13:17

    So pretty satisfying, pretty iterative process, probably typical to what you're doing or what your engineers are doing. Now it's kind of a cool demo. My daughter is, like, mildly impressed if she says the, the thing just right.

86. 13:29

    But actually, 75% accuracy is incredibly annoying. It-it-it's, it's still not-

87. 13:34

    Okay, so-

88. 13:35

    Um, it's still not gonna, gonna work so well. Switched to Mistral. Um, Mistral came out mid-project here. Um, and Mistral's model for free, exact same API to Llama 2, got the accuracy up to 79%, right?

89. 13:46

    So this is sort of happening in real time, probably happening to you also. If you're ready to switch the models quickly, you can get free accuracy improvements as, as new models come out.

90. 13:56

    77% accuracy is still really annoying, trust me. Um, [laughs] so I thought, "Okay, I need to, I need to improve this." Still not impressing my daughter. Um, and so I thought, "Okay, let's try some fine-tuning."

91. 14:07

    And now, um, you know, fine-tuning used to be a hard thing to do, especially on these really large models. Um, LoRA, QLoRA has made this a lot more tractable.

92. 14:16

    Now there's probably 20 companies here that will actually do this, um, for you for, for super cheap. So, you know, no excuse for not fine-tuning once you get to a point where, um, you wanna make that kind of last improvement of accuracy.

93. 14:27

    Um, I was able to run QLoRA on my GPU in my [REDACTED:location]. This runs on kind of standard hardware, I think it's a 4080, um, and, and ran a dataset in, in a few hours.

94. 14:38

    But you need the dataset. So I manually created a couple examples here, you know, using a larger model, in this case ChatGPT, to create more examples. Worked phenomenally well.

95. 14:46

    I think 95% of them were good. You know, I kinda went through by hand and, and filtered some out. But that, you know, in a, in, you know, 15 minutes, I have, um, a couple thousand examples to fine-tune on.

96. 14:56

    Um, I gave it a schema to format the answer in, and once it saw more examples, it was actually ge- able to generate more interesting examples kind of from that, that dataset.

97. 15:07

    Um, fine-tuning plus Mistral actually led to 98% accuracy. So this is super satisfying project. 98% accuracy actually starts to feel like, "Hey, maybe we could, we could ship this thing."

98. 15:17

    Um, and as a surprising byproduct, it actually worked for other languages. So I was using Whisper model that, that works really well in other languages. Um, Japanese is the one that I tested on.

99. 15:27

    I think this is, uh, French here. Um, but, but the, the LLM's also work in other languages. So we've kind of built this amazing multilingual Alexa as kind of a fun side project.

100. 15:37

     Which by the way, I think is exciting because I think there was a time when these models got really big, and there's always been kind of a vibrant hacker culture, I think, in AI, but started to wonder if, if you could only do really expensive projects to get anything interesting done.

101. 15:50

     I think now there's just an incredible amount of, um, things that, that people can do.

102. 15:56

     Um, so you can find this project at, uh, github[REDACTED:url]. Um, and the, and the lessons learned, stepping back, are exactly the things that our customers are discovering. It's exactly the things that you're probably discovering.

103. 16:07

     Prompt engineering works really well, especially when you sto- start with a terrible prompt. And then fine-tuning improves the performance at the end. People often talk about, "Hey, should I be using RAG?

104. 16:16

     Should I be using fine-tuning? Should I be using prompt engineering?" I think almost always in the end, to get something shipped into production and working well, you end up using a whole bunch of these different, um, methods.

105. 16:26

     Everything is making iterative improvement. Gradually, we get to the accuracy that we need. Um, the other thing that I kind of hid from you in this project is this did take me a couple weeks to get done, and most of the experiments didn't work, right?

106. 16:38

     So, you know, in, in the real world, you know, if I gave this to a colleague and they tried to iterate o- on top of it,

107. 16:45

     they would-- if they just saw this talk here, they wouldn't realize all the things that I tried that failed, and they'd probably start to do them over and kind of have to experience the same failures to catch up to where I was.

108. 16:56

     And so, um, we actually kept track in WandB of, of all the, um, the failures and experiments, and you can go in there and, and look at them yourself.

109. 17:03

     But I think this is generally best practice.

110. 17:06

     Um, so what, what are the lessons? You know, people come to me now e-every day asking, "Hey, you know, how do I get LLM apps from demo into production?

111. 17:16

     What are the kind of broader lessons besides using, um, Weights & Biases?" which you obviously are already doing. Um, and I think the, um... I think that there's kind of four things that we see our customers do where they actually get things into production, they actually make them successful.

112. 17:32

     Um, the, the first is that they build an evaluation framework, and they take this really seriously. We've had people-- I wrote a-- I do a podcast called Gradient Dissent, where I talk to a lot of people.

113. 17:41

     Um, and, and one prominent CEO of a well-known company, um, came on the podcast, and I asked him, you know, "How do you do testing? How do you test your GenAI applications in production?"

114. 17:51

     And he said, "I, I, I test these applications by vibes." [laughs] [laughs]

115. 17:56

     And, um, I asked him, you know, "Hey, do you want me to remove that, you know, from the, the pod?" Like, we're not trying to, like, embarrass people on, on our podcast, you know?

116. 18:03

     And he's like, "No, no, it's great. I, I love it," you know? Um, and, and I think that there actually is a kind of a value to testing by vibes.

117. 18:10

     Like, I think, um, if more CEOs tested by vibes, you might actually not have some of the failures that, that you see. I mean, some of these things that get released are, are so embarrassing that I think a vibes check would've actually, um, caused better decision-making.

118. 18:23

     But When you're only testing by vibes, the problem is you can't release a V2, right? Because what happens is, unlike in software development where you're typically adding features and mostly moving forward,

119. 18:34

     any of these new things that you try are gonna make some things better and some things worse, and the vibes check is not gonna tell you the difference between a 75% accuracy and a 79% accuracy.

120. 18:43

     So he never would've been able to make that switch from Llama 2 to Mistral because it just wouldn't have been clear if it's better or worse. And so taking time to build an evaluation framework is really the, the foundation of getting these things into production and working well.

121. 18:56

     I actually checked back with that CEO a year later, and he was doing quite a lot of evaluation because he kind of realized that thing where you wanna get that second model out, and you, you don't know how to do it.

122. 19:05

     Um, the other thing is starting with a lightweight prototype. We see typically, this is kind of enterprise anti-pattern, but we see it a lot at Weights & Biases, where people kinda wanna build the first step, and kinda nail that, and then build the second step, and they kinda never get something into the hands of the end user,

123. 19:19

     which might be an internal audience or it might be a customer. I think this is kind of basic agile product development, but somehow people forget this with GenAI, so it's worth mentioning because we see this failure mode so frequently.

124. 19:30

     And then, of course, incorporating end user feedback in lots of different ways is, is critical.

125. 19:38

     Um, and then iterate. So I'll say, you know, evaluation best practices because I think it's such a cornerstone, and I think... Well, actually, let me ask you this. How many, how many people in the audience feel like they're doing solid evaluations of their GenAI, um, models?

126. 19:53

     GenAI applications? Interesting. So, so a lot less. It seems like you feel people are, you all are not feeling great about your, um, [laughs] evaluation frameworks. Um, I think, I think what we see, where people kind of end up, um, is lots of different evaluation sets and techniques.

127. 20:09

     Um, so we see a lot of our customers will do some things like, "Okay, these are the things we can never get wrong, and we wanna have, like, 100% on this simple test."

128. 20:15

     And there's lots of tricks to getting that right. Then people wanna have, you know, something that's sort of very quick to run, so they can, they can kind of run it in real time.

129. 20:23

     They can test something new in 20 or 30 seconds. And then our customers typically will have much bigger evaluation sets that run nightly and give feedback. That's where people end up, and I think that's really important to do all the different kinds of evaluations that might matter for your application.

130. 20:37

     And then, you know, the other thing that's very hard to do, but I mean, MLOps has been doing this for decades, is trying to make sure that the metrics correlate with the user experience or the value that you actually provide for your customer.

131. 20:48

     Totally application dependent, um, very hard to do. I mean, when we started Weights & Biases, we kind of thought people might wanna track, I don't know, like 10 or 20 metrics per m- model that they build or per application, and we see with, with people that are really in production, like a typical, you know, self-driving car company

132. 21:04

     or, like, a typical bank with a mission critical application, we see thousands of metrics. We see tens of thousands of metrics. We get requests like, "Hey, can I do regex search across my metrics?

133. 21:13

     Because I have, like, so many metrics, I can't even [laughs] I can't find, you know, the, the, the metrics." And, and I think that's because, um, you really wanna get these metrics working and, and connected to your user experience, and there's so many different ways that something could fail.

134. 21:25

     But then the most important thing is to actually do it, right? It seems like a lot of people in this room maybe haven't started to do it yet, but doing evaluation is so critical because you can't, you can't improve any of these other things.

135. 21:36

     And when people come to me and they ask me questions like, you know, "Should I be doing fine-tuning? Should I be doing RAG?" You know, "Should I be doing XYZ?"

136. 21:45

     What that tells me is they don't have a good evaluation system in place, right? Because it shouldn't take you very long to have an answer for yourself, for your own application, um, if one of these techniques works or not.

137. 21:56

     Um, so look, you know, we're here at this conference, and this is a lot of fun. Um, if you wanna come by our booth, the Weights & Biases booth, we can show you, um, Weave, which we can help you with some of your problems getting, um, getting AI applications into production, but, but not all of them today.

138. 22:10

     But we'd love to talk to you about what we're doing, and we'd love to hear from you about how your experience has been. Thank you. [audience applauding] [upbeat music]