What We Learned From A Year of Building With LLMs

Talk transcript

1. 0:00

   [upbeat music] You're about to experience something of a strange talk, and not just because Bryan and I are strange, but because something kind of strange happened.

2. 0:24

   Over the last year, a bunch of us were posting things on Twitter. We were, uh, writing blog posts complaining about LLMs. And we formed a little group chat, and we were, you know, continuing to complain about LLMs to each other, uh, and sharing what we were working on when we realized we were all about to write the

3. 0:42

   exact same blog post: what we learned in the last year. So we, we got together, and we, uh, we turned what was initially a couple of short blog posts into a long whitepaper on O'Reilly, uh, combining our lessons across strategic, operational, and tactical levels of building LLM applications.

4. 1:04

   And the response to that whitepaper was overwhelmingly positive. We got, uh... We heard from everybody from people who contribute to Postgres to venture capitalists to tool builders saying, "We loved what you wrote in that article."

5. 1:21

   Um, uh, like, "I've felt that pain, too." And we were invited on the strength of that to give this keynote address. And so we faced a kind of funny challenge, which is part of the appeal of this blog post, uh, of this article, was that the six of us all came together to write it.

6. 1:38

   As, uh, Scott Conran put it, it was like the, an Avengers team-up. Uh, so we had to figure out a way to deliver, uh, one keynote talk from six people.

7. 1:49

   Uh, so we, we pulled the Avengers together [laughs] for, uh, one night only, uh, to sort of ex- like, deliver some of the most important insights from that 30-page article, uh, to add some of our spicy extra takes that ended up on the cutting room floor, and to respond to the allegations.

8. 2:11

   I'd like to state unequivocally that we are not, in fact, crypto bros who just found out that GPT-4 was the new Web3. Um, uh, we all trained our first neural networks back when you had to write the gradients by hand.

9. 2:27

   So, uh, we split the article up into three pieces, so we split the talk into three pieces. First, you're gonna hear from me and Bryan talking about the strategic considerations for building LLM applications.

10. 2:38

    How do you look to the future? How do you see around corners? How do you make big decisions?

11. 2:44

    Then we're gonna hand the clickers and the stage over to Hamel Husain and Jason Liu, who are gonna share the operational considerations. How do you put together processes? How do you put together teams?

12. 2:57

    How do you think about workflows around delivering LLM applications?

13. 3:02

    And then they will hand over the clickers and the stage to, uh, Shreya Shankar and Eugene Yan, who will talk about the tactical considerations for building LLM applications. What are the specific techniques, tactics, and moves that have stood the test of one year's time for building LLM applications?

14. 3:23

    All right. So Bryan, how do you build an LLM application without getting outmaneuvered and wasting everybody's time and money?

15. 3:31

    Ah, yes. Yes. Well, many of you may be thinking that there's really only one way to win in this new, exciting, dynamic, and very scary industry. And that, of course, is to train your own custom model.

16. 3:46

    Pre-training, fine-tuning, a little RLHF here and there. You better start from scratch, buddy. Eh, not quite. The model is actually not your moat.

17. 3:58

    For almost no one in this audience, the model is the moat.

18. 4:04

    You all, as AI engineering devotees, should be building in your zone of genius. You should be leveraging your product expertise or your existing product, maybe you've got one, and you should be finding your niche and digging into that niche, exploiting it.

19. 4:25

    You should be building what the model providers are not.

20. 4:29

    There's a high likelihood that the model providers have to build a lot of things for all of their customers. Don't waste your calories on building these things. The Sam Altman phrase of steamrolling is appropriate here.

21. 4:44

    And you should be treating the models like any other SaaS product. You should be quickly dropping them when there's a competitor that's clearly better. No offense to GPT-4.0, but Sonnet 3.5 looking, looking pretty sharp. [laughs]

22. 5:02

    It's important to keep in mind that a model with high MMLU scores, that's not a product. 87% on SpiderSQL, that doesn't automate all data requests or even 87% of them.

23. 5:16

    You can't sell HumanEval pass at 67. At least my GTM team doesn't know how.

24. 5:24

    An excellent LLM-powered application is an excellent product.

25. 5:30

    It's well-designed. It solves a job to be done.

26. 5:36

    And it enhances your user. Why are we so excited about AI? Human enhancement.

27. 5:45

    So what should you build, if not all these things?

28. 5:50

    Things that generalize to smarter and faster models.

29. 5:55

    Things that help you maintain your product's quality bar under uncertainty.

30. 6:02

    And things that help you continuously improve.

31. 6:06

    Whoa, Bryan. Continuous improvement. That's, uh, that's my trigger phrase. [laughs]

32. 6:14

    The idea of continuous improvement has been brought to the world of LLM applications by, like, this shift in focus that we've all felt since the previous AI engineer summit to focus on evaluation and data.

33. 6:32

    It's nicely synecdochilized by this diagram from our co-author Hamel Husain showing this virtuous cycle of improvement. It has evals and data at the center, but the core reason to create those evals, the core reason to collect that data, is to drive forward this loop of continuous improvement.

34. 6:52

    And despite what your expensive consultants or, uh, your... the, many of the, uh, LinkedInfluencers posting about LLM apps might say, this is not actually the first time that engineers have tried to tame a complex system and make it useful and valuable.

35. 7:11

    This same loop of iterative improvement was also at the core of MLOps at the operationalization of machine learning models before LLMs. This figure from our co-author Shreya Shankar's paper, uh, had that same loop of iterative improvement centered also on evaluation and on data collection.

36. 7:34

    MLOps was also not the first time that engineers faced this problem, the problem of complexity, the problem of nondeterminism and uncertainty. This-- The DevOps movement that gave MLOps its name also focused on this kind of iterative improvement and on monitoring, uh, information in production to turn into improvements to

37. 7:59

    products. But dear reader, DevOps was not the first time that engineers tackled this problem of uncertainty

38. 8:09

    and solved it with iterative improvement. DevOps built on the ideas of the lean startup movement, uh, from Eric Ries that s- that was focusing not just on building an application, not just on building an, a machine learning model or an LLM agent, but on building the entire business.

39. 8:28

    And it used this same loop centered on measurement and data, uh, to, to drive the improvement of and building of a business.

40. 8:38

    This idea itself was not invented in Northern California, despite what, uh, some people might say. It has its roots in the Toyota Production System and in the idea of Kaizen, or continuous improvement.

41. 8:53

    Genchi Genbutsu is one of the core principles from that movement that we can take forward into the development of LLM applications. It means real things, real places, and at Toyota, that meant sending executives out to factory floors, getting their khakis a bit dirty.

42. 9:08

    For LLM applications, the equivalent is looking at your data.

43. 9:13

    Look at-- The, that data is the real information about how your LLM application is delivering value to users. There's nothing that is more valuable than that.

44. 9:25

    Finally, it's-- there's lots of people selling tools at this conference, including myself. It's easy to get overly excited about the tools and the construction of this iterative loop of improvement and to forget where value actually comes from.

45. 9:38

    And there's a great pithy, earthy statement from the Toyota Production System from Shigeo Shingo that I really like: "Value is only created when metal gets bent." So we have to make sure that we don't get lost just building our evals and calculating concept drift, and we instead make sure that we continue to get out there and bend

46. 9:58

    metal and create value for our users.

47. 10:01

    Not gonna lie, I might have misunderstood earlier when you said, "Let's get bent." Okay. [laughs] So right off the bat, we need to spin that data flywheel, Bob. Oh, wait, sorry.

48. 10:11

    Wrong, wrong game show. Point is, we need to get this moving. We need to get this in front of users and human beings. We need to express the goals for our system.

49. 10:20

    And how do we do that? With evals. Remember, evals are not convenient, weird, bespoke, uh, metrics. Evals are objectives. They're what we want our system to do.

50. 10:32

    Any system for capturing this behavior is good enough. I don't have an evals framework to sell you, but what I do have to sell you is this idea that you should be getting out there.

51. 10:44

    You should be getting started. But wait, Bryan, I'm really nervous. What if this isn't good enough for my customers? Fear is the mind killer. [laughs]

52. 10:56

    Put it out there in beta. If it's good enough for these incredible companies like Apple Intelligence, Photoshop, and Hex, that's me, it's good enough for you. You need to collect this data.

53. 11:07

    You need to put something in the wild. You need to start looking at your user interactions, the real user interactions. LLMs responses deserve human eyes. You can give it some AI eyes too, but definitely look at it with your human eyes.

54. 11:25

    Binary human feedback is valuable. It's nice to add some rich feedback too. That can be interesting. But start with binaries. And finally, user requests will reveal the PMF opportunities that lie below your product substrate.

55. 11:41

    Where is your PMF? Everybody wants to know. It's in your user interactions. What are they asking your chatbot that you haven't yet implemented? That's a really nice direction to skate if that's where the puck's going.

56. 11:57

    And despite the focus on the user interactions that you can have today, the things that you can ship right now, it's important to also think about the future. The best way to predict the future is to look at the past, find people predicting the present, and copy what they did.

57. 12:17

    In designing the, many of the components of the personal computing revolution, Alan Kay and others at PARC adopted as a core technique projecting Moore's Law out into the future.

58. 12:29

    They built expensive, unmarketable, slow, and buggy systems themselves so they could experience what it was like and build for that future and it, and create it.

59. 12:41

    We don't have quite the industrial scaling, uh, information that, uh, that Moore had when he wrote down his predictions, but we do have the beginnings of those same laws.

60. 12:54

    There's been an order-of-magnitude decrease every twelve to eighteen months at three distinct levels of capability. At the capability of davinci, the original GPT-3 API that brought-- that excited a lot of us about the idea of building on foundation models.

61. 13:10

    The capabilities of text-davinci-002, the model lineage underlying ChatGPT that brought the rest of the world t-to excitement about this technology, and the latest and greatest level of capabilities with GPT-4 and Sonnet.

62. 13:25

    In each case, around fifteen months is enough time to drop the cost by an entire order of magnitude. This is faster than Moore's Law.

63. 13:35

    And so the appropriate way to plan for the future is to think what this implies for what applications that are not economical today will be economical at the time that you need to raise your next round.

64. 13:49

    Uh, so in twenty twenty-three, it cost about six hundred and twenty-five dollars an hour to run a video game where all the NPCs were powered by a chatbot. That's pretty expensive.

65. 14:00

    In nineteen eighty, it cost about six dollars an hour to play Pac-Man, inflation adjusted. That suggests that if we just wait for two orders of magnitude reduction or about thirty months from mid twenty twenty-three, it should be possible to deliver a compelling video game experience with chat-chatbot NPCs at about six dollars an hour, and people will probably

66. 14:21

    pay for it. So you can't sell it now, but you could live it and you can design it, and you can be ready when the time comes.

67. 14:30

    So that's how to think about the future and how to think strategically when building LLM applications. I'd like to call to the stage my co-authors, Jason Liu and Hamel Husain, to talk about the operational aspects.

68. 14:42

    Let's give them a hand. [audience applauding]

69. 14:48

    All right. Thank you, man.

70. 14:52

    Cool.

71. 14:52

    All right. So Hamel and I have basically been doing a lot of AI consulting in the past year, right? We've worked with about twenty companies so far, and, you know, we've done something from pre-seed all the way to public companies, and I'm pretty bored of giving generic good advice, especially because there's such a range of operators here.

72. 15:10

    And so instead, I'm gonna invert. My goal today is to tell you how to ruin your business. [laughing]

73. 15:17

    First of all, everyone knows that in the gold rush, you sell shovels. And so if you wanna get gold, you gotta buy shovels too, right? You know, if you wanna find more gold, keep buying shovels.

74. 15:29

    Where do I dig? Keep buying shovels. How do I know when to stop digging? The shovel will tell you. And how do I dig one deep hole versus making investments in a plenty of shallow holes?

75. 15:39

    Again, the answer is more shovels, clearly, right? And this might be generic, so I'll give you some more specific advice. If your RAG app doesn't work, try a vector database, a different vector database.

76. 15:51

    If the methodology doesn't work, implement a new paper. And maybe if you update the embedding model, you'll finally find product market fit. [laughing]

77. 16:02

    Because truth be told, success does not lie in developing expertise or processes. Try more tools. There's no need to balance between exploring and exploiting the mechanisms that work for you.

78. 16:14

    Change the tools. And the processes and the decision-making frameworks don't matter. The right tool will solve everything.

79. 16:23

    Number two, find a machine learning engineer who can fine-tune as quickly as possible. A two thousand dollar per month OpenAI bill is very expensive. And instead, hire someone for a quarter of a million dollars, give them one percent of their company to fight CUDA build errors and figure out server cold starts, right?

80. 16:41

    Because what's the point of growing your company if you're just a wrapper? And if your margins are too low, try fine-tuning. It's much easier than figuring out how to build something worth charging for.

81. 16:53

    It's really-- I cannot, I cannot reiterate this enough. It's very important to hire a machine learning engineer as quickly as possible, right? Even if you have no data, generating products.

82. 17:04

    They love fixing Vercel TypeScript build errors. And generally,

83. 17:10

    if you hire a full stack engineer who's really caught the LLM bug, they, they're gonna lack real experience.

84. 17:17

    And this is because Python is a dead language, right? Machine learning engineers, research engineers can easily pick up TypeScript, and the ecosystem that exists in Python could be quickly re-implemented in a couple of weekends, right?

85. 17:30

    The people who wrote Python code over the past ten years doing data analysis, they're gonna easily be able to transition their tools. And if anything, it's really easy to teach things like product sense and data literacy to the JavaScript community.

86. 17:45

    And most important of all, in order to find this kind of magic talent, we need to create a very catch-all job title. Thus use words like ninja and wizard or data scientist or prompt engineer, or even the AI engineer.

87. 18:00

    In the past ten years, we've known that this works really well, right? Every time we know exactly who we want, as long as we catch a very wide net of skills, it doesn't really un- matter whether or not we don't know what outcomes we're looking for.

88. 18:15

    Anyways, to dig me out of this hole, I'll, uh, have Hamel explain and, uh, you know, take a deep breath, think out loud, step by step.

89. 18:24

    Thank you, Jason. [audience applauding] So that was really good. I mean, let's just step back from the cliff a little bit, and let's kind of linger on the topic of AI engineer.

90. 18:40

    I heard, heard some booing in the audience. Um, and so I love the term AI engineer. Like much props to Swyx for kind of popularizing this term. Allows us all to get together and have conversations like this.

91. 18:53

    But I think that there's a misunderstanding of the skills of AI engineer. What is-- what skills you need to be successful, and there's a lot of inflated expectations.

92. 19:05

    As a founder or engineering leader, the talent is the most important lever that you have.

93. 19:11

    And so what I'm gonna do is I'm gonna talk about some of the problems and perhaps some solutions when it comes to this talent, uh, misunderstanding.

94. 19:21

    So just to review, what is an AI engineer? So this is a diagram that everyone has probably seen. Uh, there's a spectrum of skills in the AI space, and there's this API dividing line in the middle.

95. 19:33

    And kind of to the right of the API dividing line, we have AI engineer. The AI engineer skills are focused on things like chains, agents, tooling, and infra. And auspiciously missing from the AI engineer are tools like evals and data.

96. 19:52

    And I think a lot of people have taken this diagram too literally and taken it to heart and say, "Hey, we don't really need to know about evals," for example.

97. 20:02

    The problem is, is that you can go from zero to one really fast. In fact, you can go to zero one faster than ever before with all the great tools out there just by using vibe checks and implementing the tools that we talked about.

98. 20:16

    However, without evals, you can't make progress. It quickly leads to stagnation because if you can't measure what you're doing, you can't make your system better, and you can't go beyond zero to one.

99. 20:29

    So what can we do about this? About this evals skill set and data literacy? So Jason and I have found that you can actually get really good at writing evals and data literacy with just four to six weeks of deliberate practice.

100. 20:46

     In fact, like very effective. And we think that these skills, evals and data, should be brought more into the core of AI engineer. And it really, it like helps solve this problem, and it's something that we see over and over again.

101. 21:00

     So the next thing I want to talk about is the AI engineer job title itself. And so vague job titles can be problematic. What we see over and over again in our consulting is that this kind of catch-all role have very inspla- inflated expectations.

102. 21:19

     Um, this-- anytime anything goes wrong with the AI, people look towards that role to fix it, and sometimes that role doesn't have all the skills they need to move forward.

103. 21:28

     And we've seen this before with the role of data scientists. Titles and names really matter.

104. 21:36

     Um, and what I wanna emphasize is I think AI engi-engineer is very aspirational, and you should keep learning, and it's a good thing to strive towards. But you need to have reasonable expectations.

105. 21:48

     And just to kind of bring it back to data science, we've seen this before in data science as well, where we had kind of a decade ago when this role was coined.

106. 22:00

     It was a unicorn that had all these skills, software engineering skills, statistics, math, domain expertise. And we found out as an industry that we had to unroll that into many other different roles, such as decision scientists, machine learning engineer, data engineer, so on and so forth.

107. 22:17

     And I think similar things may be happening with the role of AI engineer, and it's good to keep that in mind. And what I see, or what we both see in consulting, is that it's helpful to be more specific, to be more deliberate about what skills you need and at what time.

108. 22:33

     And depending on your maturity, it's very helpful to not only specify what the skills are, but what kinds of products you'll be working on. So these are some job titles from GitHub Copilot, um, that kind of are very specific about the skills you need at that time.

109. 22:49

     And really, it's important to hire the right talent at the right time on the maturity curve. So when you're first starting out, you only need application development, software engineering, and/or AI engineering to go from zero to one.

110. 23:02

     Then you need platform and data engineering to ca-capture that data. And then only after that, you should hire a machine learning engineer. Do not hire a machine learning engineer without having any data.

111. 23:13

     But again, you can get a lot more mileage out of your AI engineer with deliberate practice on evals and data. We usually find four to six weeks practice does the job.

112. 23:24

     So in recap, one of the biggest failure modes is talent. We think the AI engineer is often over-scoped but under-specified, but we can fix that by learning evals. Next, I wanna give it over to Shreya Shankar and Eugene Yan to talk about-- to dive into this evals and data literacy. [audience applauding]

113. 23:52

     Thanks. Question.

114. 23:54

     Thank you, Jason. Thank you, Hamel. Next up, Shreya and I are gonna share with you about the tactical aspects of building with LLMs in production, specifically evals, monitoring, and guardrails.

115. 24:05

     So here's a Hacker News quote: "How important evals are to the team is a differentiator between teams shipping out hot garbage and those building real products." I would agree.

116. 24:16

     I think here's an example of LLMs-- uh, of Apple's recent LLM, where they shared about how they actually collected seven hundred and fifty summaries of push notification and email sum-summarizations because these are datasets, they are representative of their actual use case.

117. 24:32

     So how do we build evals for our own products? Well, I think the same thing-- the simple thing is to just make it simpler. For example, if you're trying to extract product attributes from a product description, break it down into title, price, rating, and then you can just simple do-- simply do assertions.

118. 24:50

     Sim-similarly, for summarization, instead of trying to eval that amorphous blob of a summary, break it down into dimensions such as factual inconsistency, relevance, and informational density. And once you've done that, assertion-based tests can go a long way.

119. 25:05

     Are we extracting the correct price? Are we extracting the correct title? Or if you're doing natural language to SQL generation, is it using the expected table? Is it using the expected columns?

120. 25:15

     These are very simple to eval and reiterates what Hamel has mentioned about keeping it simple. Lastly,

121. 25:23

     assertions can do everything, but they can only go so far. So therefore, consider evaluator models. May, may be training a classifier for factual inconsistency or reward model for relevance.

122. 25:34

     This is easier if your evals are classification and regression based. But that said, I don't know how I feel about LLM-as-a-judge.

123. 25:41

     What do you mean you don't like LLM-as-a-judge? I, I personally am super bullish on LLM-as-a-judge, and I'm curious how many of you are exploring LLM-as-judge or have implemented it.

124. 25:56

     No.

125. 25:56

     Yeah. There's a judge right here. You wanna stand up?

126. 25:59

     No.

127. 26:00

     Actual ju- uh, LLM judge here. Yeah. Anyways, we're gonna go through some points on what to consider when deploying LLM-as-judge. First of all, there-- it's no brainer. LLM-as-judge is the most easy to prototype.

128. 26:14

     You just have to write a prompt to check for the criteria or metric that you want, and you can even align this towards your own preferences by providing few shot examples of good and bad for that criteria.

129. 26:26

     On the other hand, fine-tune models or LLMs that, you know, you have to collect a lot of data and set up a pipeline to train as your evaluator are not super easy to prototype and have a lot of upfront investment.

130. 26:40

     Yeah. But that said, LLM-as-a-judge is pretty difficult to align it to your specific criteria in the business. Who here has not had e-- Who here has not had any difficulty aligning the LLM-as-a-judge to your criteria?

131. 26:53

     No.

132. 26:53

     Anyone? Okay, we gotta talk later, Shreya. [laughs] Um, I think that if you just have a few hundred to a few thousand samples, it's very easy to fine-tune a simple model who can do it more precisely.

133. 27:04

     Second, if you wanna do LLM-as-a-judge and have it fairly precise, you sort of need to use chain of thought, and chain of thought is gonna be, I don't know, five to eight seconds long.

134. 27:14

     On the other hand, if you have a simple classifier or reward model, every request is maybe ten milliseconds long. That's two orders of magnitude lower and would improve throughput.

135. 27:26

     Next, we wanna think about technical debt. Okay? When we're implementing our validators in production, even if they run asynchronously or they run in the critical path, how much effort do we need to put in to keep these up to date?

136. 27:39

     With LLM-as-judge, if you don't make sure your few shot examples are dynamic or some way of making sure your judge kind of prompt aligns with your definition of good and bad, then you're toast.

137. 27:52

     And s- kind of-- The effect is not as pronounced for fine-tune models, but if you don't continually fine-tune your validators on new data, on new production data, then they will also be susceptible to drift.

138. 28:04

     So overall, when do you wanna use LLM-as-judge? It's honestly a resources question and where you are in your application development. If you're starting to prototype it, um, you need quick evals with minimal dev effort and need something...

139. 28:19

     You have low-ish volume of evals, start with LLM-as-a-judge and kind of invest in the infrastructure to align that over time. If you have more resources or you know that your product is gonna be sticky, go for a fine-tune model.

140. 28:34

     Next, I'm gonna talk about looking at the data. Eugene mentioned, you know, you should create evals on your custom or bespoke criteria, but how do you know what criteria you want?

141. 28:44

     Simple answer, look at your data. Great AI researchers, but we changed that to engineers. Great AI engineers look at their data. So how do we do this? The first question actually before how is when do you look at this?

142. 28:58

     I know people who never look at their data at all or people who look at it initially after deployment. Wrong answer. You wanna look at it regularly. I work with a startup that, you know, whenever they ship a new LLM agent, they create a new Slack channel with all of the agent's outputs that come in real time.

143. 29:15

     After a couple of weeks, they transition this to kind of daily batch jobs, um, and make sure that, you know, they're not running into errors that they didn't anticipate.

144. 29:24

     Second thing is what specifically are you looking for? You wanna find slices of the data that are pretty simple or easy to characterize in some way. For example, data that comes from a particular source or data that has a certain keyword or phrase or is about a certain topic, right?

145. 29:39

     Simply just saying all of these are bad, but having no way of characterizing them and then improving your pipeline based on that is not gonna help. Finally, some things to keep in mind throughout this whole kind of looking at your data experience is that your code base is very rapidly changing over time probably, your prompts, components of

146. 29:58

     the pipeline, and et cetera. So when you're inspecting traces, it's super helpful to be able to know, you know, what GitHub commit or what model version or prompt version did this correspond to.

147. 30:08

     I think this is one of the very successful things that traditional MLOps tools did, like MLflow, for example. They made it very easy to trace back, and then hopefully you could replay something.

148. 30:18

     Well-[laughs] I, I see the judge shaking his head, but [laughs]

149. 30:22

     great. Um, and finally, when using LLMs as APIs, pin model versions. Uh, LLM APIs are known to, you know, exhibit different behavior that is very hard to quantify for certain tasks.

150. 30:35

     So pin, you know, GPT-4 1106. Pin GPT-4.0, whatever it is that you're using.

151. 30:43

     So Shreya mentioned that we need to look at our data, but how do we look at our data all the time? I think the way to do this is via an automated guardrail.

152. 30:50

     Here's Brandolini's law adapted. The amount of energy to catch and fix defects is an order of magnitude larger than needed to produce it. And that's true. It's really easy to call an M- LLM API and just get something.

153. 31:02

     But how do we know if it's actually bad? I think it's really important that we do have some basic form of guardrails, and some of them are just table stakes.

154. 31:10

     Toxicity, personally identified in- information, copyright, and expected language. Now, you may imagine that this is pretty straightforward, but sometimes you don't actually have control over the context. For example, if someone's posting an ad on your English website that's in a different language, and you're asking your LLM to extract the attributes or to summarize it, you may be

155. 31:29

     surprised that for some non-zero proportion of the time, it actually in a different language. Similarly, hallucinations happen more often than we would like. Um, so imagine you're trying to summarize a movie based on the description.

156. 31:42

     You just have a description for the trailer. It may actually include spoilers because it's trying so hard to be helpful, but that's actually a bad user experience. So sometimes it will include information that's not in there.

157. 31:53

     Here's a tip. If we spend a little bit more time building reference-free evals, we can use them as guardrails. So reference-based evals are when we generate some kind of output, and we compare it to some ideal sample.

158. 32:06

     This is pretty expensive, and you actually have to collect all these goal samples. On the other hand, if we have these labels, we can train and evaluate the model and just compare it to the source document.

159. 32:15

     So for example, if you're comparing summarizations, we can just check if the summary entails or contradicts the source document, and now we have a summarization-- I mean, a hallucination eval.

160. 32:26

     So therefore, if we spend some time building reference-free evals once, we can use it to guardrail all new output.

161. 32:32

     Cool. Thanks, Eugene. So we're gonna wrap up the next minute or so on some high level, bird's-eye view, 2,000-foot view, whatever you wanna call it, takeaways. First off, how many of you remember this figure from this pretty seminal paper in MLOps that came out maybe 10 years ago?

162. 32:52

     2015, so nine years ago. Yeah. So I, I think this paper really communicated the idea that the model is a small part in when you're productionizing ML systems, right?

163. 33:04

     There's so much more around the model that you have to maintain over time. Data verification, uh, feature engineering, monitoring your infrastructure, et cetera. So you might be wondering, you know, we have LLMs.

164. 33:17

     Does any of this matter? Yeah? That was-- Yeah, I'm seeing s- few nods here. Absolutely. Um, when we have LLMs, it-- all of these, you know, tech debt principles still apply.

165. 33:33

     And you can even think of the exact mapping for every single component in here to the LLM equivalent. For example, maybe we don't have feature engineering pipelines, but, you know, cast in a new light, they-- it's RAG, right?

166. 33:46

     We're looking at context. We're trying to retrieve what's relevant, engineer that to, you know, not distract the LLM too much. We have a ton of experimentation around that. All of this is something that needs to be maintained over time, especially as models change under the hood.

167. 33:59

     Similarly, for data validation and verification, right, we have evals. We have guardrails that need to be deployed, right? It's not just simply wrap your, uh, model or GPT, um, in some software and ship it.

168. 34:12

     No. There's, like, a lot of investment that needs to happen around the model.

169. 34:17

     All right. So I'd like to end with this quote from Karpathy senpai. There's a large class of problems that are really easy to imagine and build demos for, but it's extremely hard to build products out of.

170. 34:28

     For example, Charles dug up this paper of the first car driven by a neural network. That was 1988.

171. 34:37

     Twenty-five years later, Andre Karpathy took his first demo drive of Waymo, 2013.

172. 34:44

     Ten years later, I hope all of you had a chance to try the Waymo. We got the first driverless-- We got a driverless permit for Waymo in San Francisco.

173. 34:53

     Maybe in a couple more years, we'll have it for the whole of California. The point is, going from demo to production takes time. That's all we had. Thank you.

174. 35:03

     Let's build. [upbeat music]