Lessons from building GenAI based applications — Juan Peredo

Talk transcript

1. 0:00

   Hello, and welcome. Today we're gonna talk about the lessons I've learned from building GenAI application over the last year and a half.

2. 0:10

   My name is Juan Peredo, and I'll be your guide in this journey. I'm a founder, architect, consultant, developer, and everything in between. Uh, and I have over fifteen years experience, uh, in the IT industry.

3. 0:25

   And one of the first things, uh, everyone hears when they're talking about AI is about how much faster, uh, developers can code thanks to AI. You can hear this, and you've heard this before, I can guarantee, that nowadays you can just take a prompt, put it into an LLM, and you will-- an LLM will code your website

4. 0:49

   for you in about a minute or so. And we all know that that's not true, right? Um, it'll give you some code, and it'll get you going, but it won't build everything for you.

5. 1:01

   Um, but there's a lot of tools that have been created to help you with this, like some of the ones that I have listed here, like Codeium and Cursor, that interact with your IDE or an IDE by-- all by themselves, um, that help you, uh,

6. 1:17

   build your applications. You don't even have to use th-those tools, right? You can just take your code, paste it into an LLM, uh, and ask a question. All right?

7. 1:27

   And, and there's re-- some really good ones out there. Um, but it all depends on what type of application you're trying to build, right? If all you're doing is creating boilerplate, um, for yet another, uh, website, then any coding LLM will work just fine.

8. 1:44

   Um, if you're doing some more-- something more complex, or if it's something that's newer or just came out like, uh, I don't know, Svelte 5, for example, um, then you want a beefier LLM.

9. 1:56

   Uh, something like, uh, DeepSeek, uh, V3 or Claude. Right?

10. 2:04

    And, and these tools will definitely help you, um, build faster. Uh, however, they also, uh, add complexity to your application while you're trying to integrate AI to your application to actually interact with your user.

11. 2:20

    Like here in the left-hand side, we still see that in your typical, uh, application that we've been building till a year ago or two years ago, we were building frontends, backends, um, infrastructure as code, all the stuff that we normally build.

12. 2:37

    But the moment you add, uh, AI to your application, you al-- you have to start worrying about all the stuff that's in gray in the diagram on the right.

13. 2:47

    So you have to start off wondering about what model you're gonna use. Are you gonna fine-tune it? Are you gonna, um,

14. 2:55

    um, do some prompt engineering or are you try-- gonna try to use RAG to, uh, get the correct answer to your quest-- to, to your questions and to your user questions?

15. 3:05

    How are you gonna prevent it from hallucinating? Because we all know that, uh, LLMs are so bent on answering your questions that sometimes they'll just make up answers. Uh, and you need to...

16. 3:18

    a way to figure out, um, whether the answer that you're getting is right or not. And so how do you prevent that? That's something you didn't have to worry about a year a, and a half ago.

17. 3:28

    Um, also, you have to figure out, um, how often you're gonna have to, uh,

18. 3:36

    replace that model, because we all know that there's models that are coming out every other week, and they're all more powerful, beefier, but they all, they will not all work with your use case.

19. 3:47

    It's one thing that they're really great at certain benchmarks, but that doesn't mean they're gonna work, uh, with the rest of the application that you've built. So you have to constantly be ev-evaluating not only the model that you have already running but also the models that are coming out that may, uh, help you build a better application.

20. 4:07

    And you also have to start figuring out how you're gonna host these things now. Yeah, because that's beyond hosting just your application, right? All of a sudden, you don't have to worry just about CPU and memory.

21. 4:18

    All of a sudden, you have to start worry about GPU as well, right? Where are you gonna host this? Is it gonna be in the cloud? Is it gonna be, uh, on-prem?

22. 4:27

    All this information and all the stuff in gray is new to, uh, and is specific to AI-based applications.

23. 4:39

    Speaking of hosting, where to host this model is important, right? How are you gonna host it? Are you gonna host it in on-prem, or are you gonna host it on your machine while you're, um, doing some exploration?

24. 4:52

    You have to make these decisions, right? So if you're gonna host it locally, you can host it in something like Ollama. Ollama will allow you to, um, download a bunch of, uh, models into your machine where you can play with those models for free, uh, interact with them and see if they're gonna work for you.

25. 5:09

    And there's al- a host of other solutions as well. Ollama is the one I've been using. Uh, it's pretty good.

26. 5:15

    Just you just have to make sure you have a machine that's beefy enough, uh, for the model you're trying to run.

27. 5:22

    Or you can even use this Ollama and all these other tools, uh, to host things,

28. 5:27

    um, on-prem, right? But if you're gonna host it in the cloud, now you have to start thinking not only about what cloud, but also about the cost of your GPUs and is there enough GPUs to go around.

29. 5:44

    Uh, and in general, like e-everything else, you still al- have-- also have to worry about cost. And you have the, the typical players, right? You have your Google Cloud, your AWS, your Azure, et cetera, et cetera.

30. 5:56

    But there's also a, a new set of players that have come up, uh, that are

31. 6:02

    Meant to simplify the deployment of your application in the cloud. And that's where tools like Modal and SkyPilot, um, have come up.

32. 6:16

    Um, and if you're using Python, which a lot of us are using Python for, uh, interacting with your-- with our models, uh, Modal is a great choice. Modal allows you to take your Python code, add some, uh,

33. 6:31

    some, uh, uh, decorators to your code, and use their CLI to deploy, uh, to Modal. Uh, and Modal will take care of building your mo- your containers and pushing them to the cloud.

34. 6:46

    Um, it's really simple. I've used it to, to host a lot of my models.

35. 6:52

    Um, and there's also SkyPilot. SkyPilot is great to minimize your costs as they will spread the-- your clusters and your containers into multiple, uh, clouds in order to minimize your costs.

36. 7:10

    Right? And you can configure it in which, in which class you wa- cloud you want SkyPilot to, to use for hosting your application.

37. 7:20

    Um, and at the end of the day, right, if you just want to play with, with software in your machine, you can also use a, a simple library like the Hugging Face Transformers that you just can run, um, locally on your machine without any, any, any other tools.

38. 7:34

    Um, but at the end of the day, once again, how you host these models is very important.

39. 7:44

    Um, now le-let's think about, um, the fact that we're not in two thou- 2023 anymore, right? Building a chatbot powered by AI is very simple. You can build it in an hour or two, if that much.

40. 8:00

    All right? Um, there's a lot of frameworks there that will help you build this for you.

41. 8:06

    Um, however, even though building the chatbot is easy, making sure that that chatbot gives you the correct answers is really hard. Like, for example, this that I have on the screen, um, on the right is actually something that happened to me.

42. 8:23

    You-- I put in one LLM, uh, s- a few months ago now, what is three times three? And the answer that it gave me was thirty-three. Right? That's obviously not right.

43. 8:34

    Um, so while making the chatbot is easy, validating and moderating the chatbot content and making sure that it's giving you the correct answer to your user is really, really hard.

44. 8:48

    So what can we do to ensure that we are providing the right answer to your-- to our users, right? Well, there's a number of techniques. Right? You can do prompt engineering, which is basically you take the question of, of the user, you put additional, uh, details into the prompt that you pro-- pass into the, um, LLM.

45. 9:10

    However, you need to be very precise in those additional, uh, in those additional instructions

46. 9:16

    so that the LLM can follow them properly. And even then, the LLM may choose to ignore them. And at that point, what you do? Right? You get, you get the wrong answer.

47. 9:27

    You can also put guardrails. And guardrail is basically you take a secondary LLM and put it to run in front or before your main LLM. And this secondary LLM is basically a classifier that will classify, uh, the question that the ans- that the person is answering, um, as a

48. 9:49

    safe question or as a bad question. And when I say bad question, it could be, um, something that is trying to go a-a-around, uh,

49. 9:59

    uh, or asking for criminal activities, or it may be asking about, um, things that are, are not, not what you'd expect somebody to, to answer to, um, in general, right?

50. 10:12

    And at that point, the, the LLM is-- the, the secondary LLM is classifying the, your questions, and it also could be the, the answers o-of your own LLM as either good or bad.

51. 10:23

    However, you know, adding this secondary LLM adds la-latency because you're making a secondary call, right? And it also adds additional costs, and like anything else with LLMs, uh, it may or may not be right.

52. 10:40

    The next thing you could do is you could have Retrieval-augmented generation or RAG, which is basically you take a lot of information that you have in your company,

53. 10:49

    you break it into small chunks of information. So you can think of it of, uh, as, let's say, you take a document, and you break it into paragraphs, although it could be anything.

54. 10:58

    Just think of a chunk as a paragraph for now. You break it into chunks, and you store it in a vector database. And then when somebody asks a question, you take, um, you do a similari-similarity search on your, uh, vector database, pull the information, and provide that information to your, um, LLM, and tell the LLM, "Uh, here's

55. 11:21

    the information you can use to answer that question." And while that sounds really nice and good, it's really dependent on the type of information you provided in your vector database.

56. 11:32

    So if you put old information that's no longer relevant, well, guess what? What the LLM is gonna reply to you is with information that's no longer relevant. So data is extremely important in this case.

57. 11:48

    Um, the other thing you can do is you can do fine-tuning.

58. 11:52

    Right? And fine-tuning is basically you do additional training, uh, to an LLM.

59. 11:58

    This is more expensive than the other, the techniques that we're talking about, and it takes lo- a longer period of time.

60. 12:05

    And it doesn't really, uh, guarantee that you're gonna get r-good results. In fact, depending on the type of information you're putting, uh, i-into your fine-tuning, you could actually degrade the LLM quality responses,

61. 12:20

    um, because it may actually forget some of the information that it already had, or it may get confused between the new information you're passing and the information that it, that it already had.

62. 12:31

    So at the end of the day, there's no, no, no technique that will guarantee that the answer of your chatbot is gonna be good.

63. 12:39

    But, uh, using many of these techniques will help you get there, uh, or get you a long way.

64. 12:51

    Another thing that you should do is you should evaluate, uh, your models and your application

65. 12:58

    at each step of the development life cycle. For example, when you are starting to explore which LLMs to use in your, in your application or your idea of an application, you could use something like Ollama.

66. 13:14

    And we talked a little bit about Ollama before. It will let you run hundreds of LLMs locally on your machine.

67. 13:22

    Um, once you are interested in one or another, you can

68. 13:28

    start using it in your application. Or if you wanna play with thousands of, uh, open source models, you can go to Hugging Face,

69. 13:38

    and there you can build some Hugging Face space, spaces. Um, you can see the spaces that other people have built, a-and this is basically a lot of code that interacts with LLMs, um, and that you can freely use and review, um, to see whether you're interested to build something similar or, or use the, the LLMs that they're

70. 13:57

    using. Now, if you want to interact with other models, including, um, some of the closed models, you can go to OpenRouter. Now, OpenRouter will basically take your prompt, and it will run it against a number of models or a model of your choosing.

71. 14:16

    And one of the things that makes OpenRouter really cool is that it will, um, take your prompt, and let's say you want to run that prompt against Llama Three, it will look at the, uh, providers that are actually offering Llama Three as a, as an API, and it will run it until the one that's, uh, against the

72. 14:38

    one that's cheaper, which is really cool, and it's a great way to save money.

73. 14:44

    Um, I use it every day, and I put like fifty dollars about six months ago, and I still have about forty bucks left. And I use it, as I said, I use it every day, so it's a great way, um, to interact with, with models and, uh, see which one I want to use.

74. 15:02

    Uh, obviously, if you're gonna do so OpenRouter in production, uh, there are, there are a number of, um, safeguards that you, you may want to look into. You know, you don't want just, uh, OpenRouter just running into a, a random L- LLM provider without you knowing which one it is.

75. 15:20

    Um, but for testing, it's a great thing.

76. 15:24

    And finally, uh, the last tool I have here listed is LangSmith, which is created by the creators of, uh, LangChain, which is a very popular open source, um,

77. 15:36

    framework to interact with LLMs. But LangSmith, one of the great things about it is that it lets you constantly evaluate your models. So it will, uh, com--

78. 15:49

    It can help you record, uh, a number of runs that you have with your LLMs. You can compare over time, um, how your LLMs have, have been performing. Um, it's a really, really cool platform, um, just to keep track of what your application and your models are doing.

79. 16:08

    Um, so yeah, evaluation, you, you really should be doing this at each step of the way.

80. 16:17

    Another thing that it's very important is externalizing your prompts.

81. 16:22

    And I know that for all of us, it's really easy to just take a prompt, put it in the code, hard code it in there, and move on.

82. 16:31

    Um, but if you do that, you're shortchanging yourself. Externalizing your prompt, uh, in tools like, uh, repositories like Lang- LangChain Hub, which I have here, uh, a screenshot on the left, uh, will allow you to,

83. 16:48

    uh, share those prompts with experts. So if you are building an application that has to do with education, you could, you could reach out to, to the education expert that has never coded in their lives, and you can have them this-- log into, to the hub, play with, with the prompts, run the LLMs, see the output that

84. 17:09

    they're getting from the LLMs, um, and then when they're ready and they're happy with that they're getting the correct, uh, prompts, all they have to do is just, uh, commit those prompts, and that will flow directly into your application.

85. 17:24

    It's really cool. Another thing that will, that, uh, this will help you with is future-proofing. We all know that models are coming out every other week. There's a new model, and that's better than the previous one.

86. 17:38

    Um, and unfortunately, right, even let's say if you're using a Llama model and the next Llama model comes out, um, your prompt may not work prop-properly with the new version.

87. 17:49

    You may not get the, the same results. So by allowing you to easily access the prompts and modify them and test them against something new, then you c-- you are b-basically future-proofing and making your life easier in the future.

88. 18:04

    And that, in turn, leads to faster development.

89. 18:08

    So, bottom line, prompts should never be encoded in your code base.

90. 18:17

    I'll talk a little bit about agents. Agent, agents are,

91. 18:22

    and I expect them to be in two, uh-- this year, one of the best, uh, additions to the world of LLMs.

92. 18:30

    Because agents are basically allowing, um, the LLMs to break out of the chatbot and actually interact with the real world. So you can have an agent, for example, um, that, uh, allows you to...

93. 18:47

    For example, you have a book, you want to, uh, create the au-the audio for it, and you want to translate it into multiple languages. That's something that you can do, and you can have an output out of that, those things.

94. 19:02

    All right? So agents will allow us to interact with the real world. All these LLMs that have been confined to just chatting, all of a sudden will have a real impact in the world.

95. 19:13

    Um, and we're still not there. The-- We're still working on it, but it's a, a really ex-exciting, uh,

96. 19:22

    exciting, uh, set of applications that will be coming out soon.

97. 19:28

    And here I have a couple of, um, drawings, uh,

98. 19:33

    of what an agent looks like. And the one on the left,

99. 19:38

    that's, that has four nodes, so it has start nodes, assistant tools, and it's really, really powerful.

100. 19:45

     Right? Because basically what you're doing is you're--

101. 19:49

     When somebody gives you a question, you pass it to the assistant, and the assistant w- basically has an LLM embedded. And that LLM will analyze the question

102. 20:00

     and then will start calling tools. And tools are-- it's a fancy name for functions. It would start calling functions, um, until it gets to the answer that it, that it needs, uh, to answer the question of the end user.

103. 20:15

     So, for example, it could be that the end user sends, uh, the text of a book to the assistant, and it tells it to translate it and, uh, create the audio, and then the assistant will call the tool to create the audio.

104. 20:29

     Then it will call the tool, uh, to the translation, and then it will end.

105. 20:34

     Right? And the assistant is making those decisions all by itself, which is a, a really, really powerful, um, concept.

106. 20:45

     Um, and while it's really cool that those agents can make those decisions without you actually having to hard code all those paths,

107. 20:54

     um, they do have its disadvantages, right? And one of the big ones is that LLMs have a lot of latency. You know, calling one of th-those LLMs takes several seconds,

108. 21:07

     right? So all of a sudden, if you have an agent that is integrating with four or five LLMs, and each one of them takes, let's say,

109. 21:20

     three seconds to run, then all of a sudden you have,

110. 21:24

     uh, a process that takes twelve, fifteen seconds, right? And we know that our users nowadays are used to running things on the web and taking those things taking only milliseconds, right?

111. 21:39

     And if they have to wait for a second or two, people are leaving our sites and going to our competition, which is not the best, uh, result.

112. 21:48

     Um, so we should take advantage of, o-of, uh, these platforms that have been built like LangChain, LlamaIndex, LangFlow. There, there's many of them, right?

113. 22:00

     That allow us to do things like concurrent calls and branching, uh,

114. 22:05

     in these agents, right? So, like, for example, I have here the, on, on the left a diagram of an agent that's actually calling, uh, a couple of things in, in parallel, right?

115. 22:17

     So it could be, once again, it could-- this could be, um, the user passing, uh, a book to node A. Node A calling, uh, the audio translate-- the audio creation and the translation in parallel, and then sending the, uh, answer back to the user.

116. 22:37

     Now, if each one of these things took, uh, two seconds, instead of these things taking eight seconds, all of a sudden it takes six,

117. 22:45

     right? So that's much better, right? So there's a lot of things and, and, and a lot of things that, that need to come to mind when you're building these agents, so that you can make them as efficient as possible, so that we can still get to the end to fulfill the, the expectations of our users

118. 23:07

     while taking advantage of the LLM's powers. Um, and while I do believe that agents are the future for LLMs, we really have to be aware of the costs of running these agents,

119. 23:25

     um, because they can-- the cost can add up really quickly. Let's take, for example, the, the one I have here where once again, it's a simple, um,

120. 23:36

     agent. Um, and we're, we're gonna assume that this is a call center, right? This call center takes three thousand calls a day, and each one of those calls, uh, requires fifteen, uh, function calls or tool calls.

121. 23:53

     And we're gonna assume that we're using OpenAI o1. It's a great model, right? So we're gonna take advantage of it because it's the-- one of the big-biggest and greatest.

122. 24:04

     Um, and the prices for, for OpenAI o1 is fifty million-- fifteen dollars per million tokens or sixty dollars per million tokens out.

123. 24:15

     And if we assume that we're inputting fifteen hundred input tokens and three thousand output tokens, meaning, uh, we're sending a bunch of information into the LLM, the LLM is impor-- is, uh, is, uh, outputting a bunch of information out.

124. 24:32

     Um, let's figure out how much that will cost us, right? Uh, 'cause it sounds like very little, fifteen dollars per one million token, right? It sounds like it shouldn't be much.

125. 24:45

     But once you do the math, and the math is at the bottom if you wanted to take a look at it. We won't discuss it here. Uh, but it comes out to three dollars and twenty-four cents, uh, per call, which turns into nine thousand two hundred and seventy per day, and which, uh, tu-- i-in turn is almost

126. 25:01

     three hundred thousand dollars per month. Um, so if you were expecting to get a very small price tag, um, well, you're gonna be-- have a little bit of a surprise. [laughs]

127. 25:16

     Uh, at the end of the day, right, especially if you're using, um,

128. 25:23

     provider endpoints for your APIs to interact with the LLMs, you have to keep in mind that the cost of running this thing

129. 25:32

     is linear, right? So each additional user will incur you an additional cost. Now, obviously, you have to make sure that y-y-you're, you're, you're pricing your, um, products accordingly. But if you were expecting to get very low prices, uh, and very low cost, uh, you may want to, uh, do the math.

130. 25:57

     Um, but even if you are renting s-- uh, the GPUs, right, so you don't have to make the API calls. GPUs are at a premium nowadays. So, um,

131. 26:11

     if we compare the price of a GPU versus the price of a CPU on the cloud, we're talking cents versus dollars. All right? So having an A100, uh, GPU for an hour will cost you like potentially four or five dollars an hour versus, you know, a, a traditional EC2 instance that could cost you,

132. 26:36

     uh, a fraction of a cent or a couple of cents an hour. So keep that in mind.

133. 26:45

     A-and the choice of the model a-and the provider that you're gonna use in your im-- in your application is really, really important. And we can see in the price chart on the left that there's a significant va-value difference between the, uh, left, so the first LLM in the list and the last one, right?

134. 27:06

     It's, it's over twenty times difference in terms of price, right? Um,

135. 27:14

     so you have to evaluate, do you really need the power of a very pricey LLM, or can you do the same thing with a cheaper

136. 27:26

     LLM or two that may not be quite as powerful, but they can do the task, right? It might be, from a price perspective, it might be much, much, much more ef-efficient, right?

137. 27:40

     The other thing that's important, like we said, is, uh, output speed, right? How fast these, these models, um, can, uh, respond to questions.

138. 27:52

     And that's what we see in the chart on the right, the output speed. And we can see that Cerebras a-and Groq can run Llama three point one seventy B at over a thousand, uh, tokens per second.

139. 28:09

     In fact, Cerebras is over two thousand tokens per second. While all the way on the right, the, the,

140. 28:17

     the providers are running thirty-one and twenty-nine tokens per second,

141. 28:22

     right? All that translates to latency for your users. And usually, the size of the, of the model that you're picking also has a big, a big impact, right? So bigger models will take run-- longer to run.

142. 28:35

     Smaller models will go faster. Which one you're using and which one you need to use will have a big impact on, on your application.

143. 28:46

     So at the end of the day, right,

144. 28:48

     small is beautiful and better for the environment and better for your wallet.

145. 28:57

     So let's go back to our example, right?

146. 29:01

     Now, what happens if we replace OpenAI o1 with Llama three point three seventy B just from a price perspective, right? You may need to use OpenAI for your use case, but what if you didn't, right?

147. 29:12

     You can just use, uh, s-- Llama three point three. Um, well, we go from three... over three dollars per call to fifty-two cents per call. And we go

148. 29:25

     all-- from almost three hundred thousand dollars per month to a l-- almost fifty thousand dollars per month.

149. 29:35

     So once again, pick the right model for the use case.

150. 29:42

     At the end of the day, your wallet and the environment will thank you.

151. 29:51

     And the last thing I wanna touch on is observability for your agents. I can't tell you how hard it is, uh, to try to figure out what an agent is doing once you get multiple nodes that are interacting with each other.

152. 30:07

     Uh, this is a lot harder in my experience than trying to figure out what a regular application is doing because a regular application is deterministic. It will do A, B, C, D.

153. 30:19

     Well, um, a probabilistic model adds lots of unknowns to your application. So all of a sudden, things that were running a hundred times with no issue would start failing on a hundred, uh, a hundred and first item, right?

154. 30:41

     And I'll give you an example, right? I was running an LLM, and all it had to do was get the information about certain, uh, a user from the database, right?

155. 30:51

     So let's say, for example, get information for user John, and this would run

156. 30:59

     for the longest time. And all of a sudden it started failing.

157. 31:03

     And we realized that eventually, because, because we had the traces, um, that the user was entering lower cap Jo- John instead of just capital J and the rest of the word.

158. 31:16

     Okay? Uh, and all it took was going back to the prompt, which we, we had externalized lu- luckily, and changing the prompt for, uh... to add the sentence, uh, ignore case, and everything started working again.

159. 31:30

     But because this one, uh, was an agentic application and it was calling multiple, uh, models,

160. 31:38

     it would have taken us a very long time to figure out what was going on. I mean, we could put, uh, we could, we could have put the debugger and print statements, all that, and it would still have been very, very hard.

161. 31:51

     So ha- have observability for agent. There's a lot of tools out there like LangSmith that will make this very simple. Uh,

162. 32:02

     and if you don't wanna use one of the tools that's available there, um, you can build one or you can build all this. But build it because you are going to need it.

163. 32:10

     Um, and in here, um, we have our agent traces. On the left,

164. 32:16

     um, of the screen, we have an agent that's first calling Llama Guard, then it's calling Llama 3.1, then it's calling, calling Mistral. And that ha--

165. 32:26

     that, that gives us an order of everything that, that, that had run in that, in that particular call to the agent. It has metadata, it has the inputs and the outputs.

166. 32:37

     All that is crucial, right? And when it, when it throws an error, it keeps track of the errors, right? All the information that was in and out of that LLM so you can see exactly what happened, right?

167. 32:49

     And you can see the error, uh, the trace like we see, uh, on the trace on the picture on the right.

168. 32:56

     Yeah. So, uh, like I said, building observability for your agents is really important.

169. 33:06

     And with that, any questions?