Build enterprise generative AI apps using Llama-3 at 1,000 tokens/s on the SambaNova AI platform

Talk transcript

1. 0:00

   [upbeat music] All right. Well, hi everyone. Thanks so much for joining us today.

2. 0:17

   Um, so today I get the great opportunity of introducing SambaNova and some of our capabilities around reaching over [REDACTED:password] thousand tokens per second using Llama 3. Today, I'm gonna spend [REDACTED:password] little bit of time getting you oriented around SambaNova, some of the capabilities that we provide as an AI platform, and also some of the underlying technologies that

3. 0:40

   are providing the means of achieving some of the accomplishments, like [REDACTED:password] thousand tokens per second. Before I start jumping into the content, I wanna take the opportunity to introduce some of my colleagues really quick.

4. 0:52

   So joining me today, I have Petro Milan, who is [REDACTED:password] principal AI engineer. He's gonna also be leading our workshop component and be here as we're getting hands-on with the technology.

5. 1:03

   I also have Varun Krishna, who is [REDACTED:password] senior pr- uh, senior principal AI solutions engineer, uh, joining me as well. And I'm Rachelle Mattern. I'm our director of solutions engineering.

6. 1:15

   I serve our global customer base at SambaNova.

7. 1:19

   So before jumping into our content, I just wanna cover what are we gonna be talking about today. So we're gonna start off with [REDACTED:password] little bit of housekeeping, uh, talk [REDACTED:password] little bit about the prerequisites.

8. 1:28

   We are gonna be getting hands-on today, and also introduce our Discord channel, which is where we're gonna be communicating with one another and also sharing some content and information that you're gonna need, such as files and, and other, uh, like API and s- uh, uh, keys and things like that.

9. 1:43

   Um, I'm gonna talk about SambaNova, just get you oriented around who are we and how are we achieving [REDACTED:password] thousand tokens per second. Um, then I'm gonna pass it off to Petro.

10. 1:53

    He's going to, uh, talk you through our workshop today, how we're gonna get hands-on, get you oriented around, uh, uh, how to get started, uh, and we're actually gonna go through [REDACTED:password] live build, and we'll support you along the way.

11. 2:06

    Uh, we'll also spend some time around questions, uh, just in case that, um, you have any questions about SambaNova, our technology, or anything that we're doing in the hands-on component.

12. 2:15

    So before we get started, I wanna talk [REDACTED:password] little bit about prerequisites. So first of all, we are gonna be using our laptops, so, uh, hopefully you have them today.

13. 2:23

    Uh, and we're also going to need internet access to get through our workshop. Uh, we're gonna be working in [REDACTED:password] Python environment, um, so hopefully we have Python set up and ready.

14. 2:32

    Um, and also we're gonna be needing to install some packages, uh, through pip. Um, we are gonna be working in Discord, so if you don't mind, uh, I'm gonna give everybody just [REDACTED:password] second to hopefully get on Discord and, um, and join our channel.

15. 2:47

    So I'll just give everyone [REDACTED:password] quick moment to, to get set up.

16. 2:54

    Once you're set up, just maybe give me [REDACTED:password] thumbs up so I'll know.

17. 3:05

    Question?

18. 3:06

    What is the Wi-Fi password?

19. 3:09

    The Wi-Fi password.

20. 3:09

    Uh, what, what is it?

21. 3:10

    TimeToBuild, [REDACTED:password] T-

22. 3:14

    What was the last-

23. 3:15

    ... uh, [REDACTED:password] T, [REDACTED:password] [REDACTED:password]. The general one, not the one that says speaker.

24. 3:18

    Okay. Yeah.

25. 3:19

    AI engineer.

26. 3:20

    Okay. So just to repeat, uh, it's AI engineer. The password is [REDACTED:password] and [REDACTED:password] [REDACTED:password] [REDACTED:password].

27. 3:28

    No, no, [REDACTED:password] T, [REDACTED:password] T, [REDACTED:password] I.

28. 3:30

    Okay. All it-it's, it's all capitalized, each word, but concatenated.

29. 3:34

    The first letter.

30. 3:35

    Okay. Good? Um- Were you able to join? Yes. Okay. Awesome. Cool. Just wanna make sure there's no problems. Awesome. We'll give everyone [REDACTED:password] couple minutes who just got Wi-Fi access.

31. 4:15

    All right. Um, in case you haven't got [REDACTED:password] chance to set up, maybe just take [REDACTED:password] picture of this really quick. We'll also go back to it, um, before we kick off the hands-on component.

32. 4:24

    Um, but want to spend [REDACTED:password] little bit of time just getting you oriented around us at SambaNova. So SambaNova, we are [REDACTED:password] full stack AI platform, and we've existed since twenty seventeen.

33. 4:37

    Uh, we were founded out of Stanford University. So two of our, uh, co-founders are actually Stanford professors, uh, Kunle and also Chris. Um, both of them, and including Rodrigo, each bring [REDACTED:password] unique perspective to our founding team, uh, including previous, uh, startups that were, uh, acquired and had, uh, uh, various exits.

34. 5:00

    Uh, also building other AI startups that are pretty well known in the industry, such as Snorkel, Together AI. Um, and also [REDACTED:password] really, uh, depth of experience around building out hardware and chips.

35. 5:11

    Um, we are, uh, building the full stack from the ground up, so that means we build our own chip, um, I'll go into that [REDACTED:password] little bit layer- later, but also all the way through the system level and the software layer.

36. 5:23

    Um, we're on our fourth generation st- uh, chip. Um, and we have built [REDACTED:password] entire stack that allows you to both fine-tune models, pre-train models, and also deploy those models with really high performant inference.

37. 5:38

    Um, we have achieved over [REDACTED:password] billion dollars in funding from various well-known names like BlackRock or Google Ventures, Intel, GIC. Um, and so we are really well established to solve the challenge around [REDACTED:password]- building and deploying AI hardware.

38. 5:57

    So what exactly are we targeting and what exactly are we trying to solve for? Our customer base comes from [REDACTED:password] wide variety of enterprises and also to government organizations.

39. 6:08

    So we're really aiming to deliver capabilities that can help service the enterprise-grade AI capabilities that companies and governments require to deliver, um, unique and differentiated capabilities and also things like sovereign AI.

40. 6:23

    Um, our ca-- our underlying platform is delivering the means to actually achieve the scale of [REDACTED:password] trillion parameters, uh, plus. Um, and we're doing that through delivering full stack capabilities.

41. 6:37

    When I say full stack, um, m-many folks have probably utilized many of these technologies on, uh, this slide. We're not necessarily trying to compete with every single layer, uh, involved here.

42. 6:47

    But what we are trying to do is ease the process of getting started and ease the journey along the way. And so instead of having to make [REDACTED:password] decision at every single one of these layers, we are actually integrating things into [REDACTED:password] very seamless experience from deciding on what chip is going to work with, uh, what compute,

43. 7:07

    what compute and chip is going to work with what operation systems, what operation system works with what models. You don't have to actually make each of these decisions and know that they have to integrate with one another.

44. 7:19

    We actually create this very seamless experience along the way, um, where everything kind of orchestrates and works very nicely.

45. 7:28

    And what we're doing at the end of this is actually delivering the capability to not only fine-tune, but deliver really, really fast inference capabilities. So we're gonna demo [REDACTED:password] little bit of this later, but, uh, recently we released [REDACTED:password], [REDACTED:password], [REDACTED:password] demo, um, that you can actually go try live, and we'll do so later, called Samba-1 Turbo.

46. 7:47

    This is, uh, exceeding world records around, uh, speed of inference, especially when it comes to Llama Three. Um, and you can see that through some of the metrics that were re-recently published, um, through, uh, Artificial Analysis.

47. 8:01

    Artificial Analysis did [REDACTED:password], um, [REDACTED:password] benchmarking exercise to understand the different capabilities, uh, the speed at which they are able to deliver inference throughput, um, for [REDACTED:password] thousand tokens per second across various hardware providers.

48. 8:17

    And what you can see is that we are far exceeding as [REDACTED:password] platform, um, the th-the throughput capabilities compared to some of the oth-other providers out there.

49. 8:28

    And so I wanna kinda talk [REDACTED:password] little bit about how are we enabling such speed. Um, so when it comes to the underlying technology, many of us have experienced some of these trends in the industry.

50. 8:39

    Many of us got started with what you see on the right-hand side of the slide, which is the large monolithic model. This is the likes of like an OpenAI, for example, or [REDACTED:password] Gemini or [REDACTED:password] Claude.

51. 8:51

    And many of us started our LLM journey or our generative AI journey using some of these technologies. But along the way, many, uh, many other capabilities in the open source community started to pop up, specifically these smaller models.

52. 9:04

    And these smaller models allowed us to do things like fine-tuning and actually adapting some of these models to our enterprise data and our enterprise requirements. And so that really started to take off in the industry, and each of these started to see different pros and cons associated with them.

53. 9:21

    When it came to large monolithic models, when we actually started to put these into practice when it came to enterprise applications, one of the reasons many of us leaned into this is because of the broad capabilities that the likes of OpenAI brought, right?

54. 9:34

    And also the ease of integration in terms of OpenAI into the actual platform itself. It's super easy to manage, and it also was trained on the internet's data, and so it can handle [REDACTED:password] lot of different things.

55. 9:46

    But when it came to actual enterprise applications, enterprises have unique capabilities required to deliver on some of the use cases and challenges they're trying to solve for. For example, enterprises have unique data that they, they, you know, als-oftentimes segregate from the internet, or most of the time segregate from the internet, um, that's proprietary to them.

56. 10:09

    And oftentimes they have spent the last ten years trying to actually aggregate that data into the likes of data lakes and other, um, uh, kind of centralized capabilities. And so now how do you actually transform that into AI capabilities that you can leverage?

57. 10:24

    That was very difficult when it came to large monolithic models. The other challenges that we saw is that many started to become very concerned about security when it came to OpenAI.

58. 10:33

    They wanted to preserve, uh, data privacy. They wanted to also own and use the model as [REDACTED:password] differentiation for themselves. Um, and then also as you start to see more and more adoption, that cost just started to skyrocket.

59. 10:47

    Um, OpenAI and, and [REDACTED:password] lot of these closed source models charge on [REDACTED:password] per token rate. And so as you start to utilize more and more LLMs, uh, and utilize LLMs, uh, more heavily, the cost just starts to go up and up and up and up, and it's really, really hard to control.

60. 11:03

    On the other hand, when it came to adopting the smaller expert models or the smaller, uh, open source models like the likes of Llama 3-8B that we'll talk about later, um, we were able to address some of the enterprise accuracy concerns by actually pre-training, fine-tuning these models to adapt them to the enterprise requirements.

61. 11:23

    Um, while we were doing this at [REDACTED:password] smaller scale to address like different capabilities and tasks that we needed to solve for in the enterprise, they weren't trying to solve like [REDACTED:password] broad set of tasks and also adopt [REDACTED:password] broad set of general knowledge.

62. 11:38

    Um, thus manageability became [REDACTED:password] little bit challenging because now we have all of these kinda like micro models that we have to orchestrate and have them work together. Um, but we were able to solve for some things like security and model ownership.

63. 11:51

    Uh, data privacy and data ownership. Um, but again, because we had so many of these and we had to fine-tune each one of these, the cost also became very challenging.

64. 12:00

    So what were-- what are we trying to solve for, uh, through our capability of Samba-1? We're trying to bring the best of both of these, uh, paradigms together, um, to deliver the, the capabilities of each, um, in [REDACTED:password] very simplistic way.

65. 12:18

    And the way we actually deliver this is through four core capabilities. First of all, we take all of those expert models behind the scenes. So let's just say we fine-tuned [REDACTED:password] model for our legal purposes.

66. 12:31

    We fine-tuned [REDACTED:password] model for our HR purposes. We fine-tuned [REDACTED:password] model for coding capabilities. Each of those have different groups and tasks and use cases that are going to consume those.

67. 12:44

    But we want to really ease the experience of having to integrate those into the application. So we put all of those behind [REDACTED:password] secure single endpoint. And so you only have to interface with one endpoint to gain access to all these various models.

68. 12:59

    Now we need to determine how are we gonna actually use those or consume those various models behind the scenes. So now we need, uh, capabilities around orchestration. So one of the other capabilities we're delivering as [REDACTED:password] part of this is around, um, routing.

69. 13:13

    So we're delivering the ability to determine based off of an incoming prompt, what is the best-suited expert behind the scenes to solve that prompt? And we're doing so through [REDACTED:password] router.

70. 13:25

    We're also bringing the means of dynamically fine-tuning. Every expert is going to have [REDACTED:password] different cadence in which fine-tuning is going to make sense. Um, so maybe your finance model gets adjusted at annually when your policies are updated.

71. 13:40

    But maybe your coding model, because you're pushing code so regularly, needs to get updated on [REDACTED:password] quarterly basis. And so you wanna actually be able to schedule your fine-tuning jobs and adjust and swap these models at the rate at it-- which it makes sense to actually retrain these models.

72. 13:57

    And lastly, you have [REDACTED:password] bunch of models under the scenes. Not every application or group is going to or should be able to access each of those models. So now you need to figure out [REDACTED:password] way to, uh, manage the access controls for these.

73. 14:10

    So what we're also deliver-- delivering as [REDACTED:password] part of this capability is model-level RBAC. So you can actually determine this application or this person or this group of people should be able to access this set of models.

74. 14:23

    And it allows you [REDACTED:password] ton of efficiency from [REDACTED:password] computation and management operations standpoint with the security and fine-grained control that you need to actually manage acc-access to these different models and data under-underlying these models.

75. 14:38

    So within Samba-1, we have two ways of delivering this. We have something called [REDACTED:password] flexible CoE that allows you to kind of determine exactly what models lie under the hood, and we also have [REDACTED:password] pre-composed version of Samba-1 composition of experts.

76. 14:53

    So our pre-trained, um, or our pre-configured, I should say, or pre-composed version of this model has ninety-two underlying experts. And when I say experts, I'm really referring to [REDACTED:password], [REDACTED:password], [REDACTED:password] specific model, um, that can bring different capabilities, uh, associated with it.

77. 15:10

    And within those ninety-two experts, we have [REDACTED:password] broad range of languages that are covered within those models, [REDACTED:password] broad range of domains, and [REDACTED:password] diverse set of tasks that are very, very relevant to the enterprise.

78. 15:24

    All of these models are supported by seven different foundation model architectures, including Llama 2, Llama 3, Mistral Falcon, Bloom, and even some multimodal capabilities such as, like, Lava, Clip, Deep Lot, um, that is kind of starting to support some of the multimodal trends that are coming.

79. 15:42

    And of all these ninety-two models, we are actually delivering and partnering with organizations to, um, create and contribute back to the open source community. So out of the ninety-two experts, twelve of them are actually ones that we've helped, uh, e-either develop ourselves or co-develop with organizations out there, including some of the language capabilities that we've delivered, such

80. 16:05

    as models that can support things like Thai or Japanese, Hungarian. Um, and through that, we've developed [REDACTED:password] lot of experience on how to actually adapt models to different languages.

81. 16:16

    Um, we've also, uh, created [REDACTED:password] model for text-to-SQL capabilities and delivered, uh, really, really good results through that model for text-to-SQL. And lastly, we have contributed back to Bloom Chat.

82. 16:30

    Uh, it's the second-largest open source model, um, and, uh, it's also bringing [REDACTED:password] lot of the multilingual capabilities.

83. 16:38

    So organizations essentially, as they're constructing these different composition of experts, can add as many expert models as they need. As I mentioned before, while we have that pre-configured composition, this is really intended for you to be able to construct exactly what you need in terms of models under the hood.

84. 16:56

    So you can add as many as you need.

85. 17:00

    So our end goal with this is to really be able to bring the capabilities that enterprises need to, um, handle the diverse set of use cases and capabilities they need to solve their problems.

86. 17:11

    And so one of the things that we've created along the way to measure ourselves against this is an enterprise-grade AI benchmarking set. And this is really tailored to understand our capabilities against the best-in-industry models across various, uh, enterprise-specific tasks and domains that are needed.

87. 17:30

    Things like information extraction, that's where many, many enterprises are starting, um, but also [REDACTED:password] broad set of capabilities like text-to-SQL, coding, function calling. Um, and we're measuring ourselves against GPT-3.5 Turbo and GPT-4.

88. 17:45

    Um, and what we can see is along the way, um, we're meeting or exceeding, um, the capabilities that, that OpenAI is bringing.

89. 17:55

    Um, but alongside the capabilities, you also need to orchestrate these models. I talked [REDACTED:password] little bit about this before, but one of the deliverables as far as Samba-1, uh, from [REDACTED:password] product standpoint is we're bringing routing capabilities so that you can actually, uh, take [REDACTED:password] prompt, determine what is the bex-- best-suited expert, and then route to that.

90. 18:13

    Um, there's also scenarios where you may need to do something outside of routing. You may actually just wanna directly call [REDACTED:password] specific model or, in the case that is popping up really, really regularly now with agentic AI, you may need to do model chaining.

91. 18:28

    So that's another capability that we're bringing as [REDACTED:password] part of the product suite for Samba-1.

92. 18:37

    And so how are, how are we actually uniquely set up to deliver the composition of experts capability, um, and also the really fast inference speed that we're gonna see, uh, briefly?

93. 18:48

    So I mentioned earlier, but we are [REDACTED:password] full stack, uh, AI platform, but we also build our own chip. Our chip, we call that an RDU, or [REDACTED:password] reconfigurable data flow unit.

94. 18:59

    So instead of [REDACTED:password] GPU, we'll refer to our chip as an RDU. And our current version of that chip is called SN40L. And what is unique about SN40L is actually the memory structure of this chip.

95. 19:12

    So our chip supports [REDACTED:password], [REDACTED:password] three-tiered, uh, memory architecture. So we have our on-chip memory, and then we have our high-bandwidth memory, and then we have [REDACTED:password] huge, uh, memory capacity in DDR.

96. 19:26

    And this really allows us to store [REDACTED:password] ton of models, um, and have those models be swapped in and out of various tiers within the memory, um, to achieve really strong performance and also really, really efficient compute utilization.

97. 19:45

    So what does this look like? Um, as I mentioned before, we can store up to five trillion parameters on DDR. So that's like if we were to store like three to four OpenAI's on [REDACTED:password] single chip.

98. 19:59

    And then as we need to actually execute those models, they can move up the memory stack. Um, and this allows us to, one, take into consideration what models need to be used at what time, and then two, do so in [REDACTED:password] really performant way because the, the network connectivity between these three layers is really tight.

99. 20:23

    So just to kinda go into some of the specs, um, our on, on-chip SRAM, uh, has four gigs, uh, our high bandwidth memory has five hundred and twelve gigs, and our DDR has up to six terabytes.

100. 20:35

     So, um, lots of memory to work with. And when we think about how does this compare to what you experience in the GPU world, when you think about the number-- if you wanna host [REDACTED:password] really large amount of models, when you have to do so on [REDACTED:password] GPU, you're basically needing to work with the memory that, that

101. 20:55

     is, uh, the, the GPU has on chip. But with us, because we have the DDR component, we're able to store [REDACTED:password] huge amount of models in [REDACTED:password] single system and keep it coupled with the other memory, uh, tiers.

102. 21:08

     And so with GPUs, you're often-- if you wanted to host five hundred different models, you're gonna have to actually, uh, align those models to various systems, and you're gonna have to basically call the various system to actually access each of those models.

103. 21:21

     With us, it's gonna be one underlying system, um, because we're able to store, again, up to five trillion parameters.

104. 21:32

     So, um, I'm gonna hand it over to Petro. Uh, what we're gonna do next is actually, uh, get the chance to get hands-on.

105. 21:38

     Um, and yeah, thanks, uh, Rachelle, for the presentation. Um, so what we will be doing next is [REDACTED:password], uh, demo of our Llama 3 and Samba-1, um, Turbo. And after this, um, we move to the, uh, hands-on portion of the workshop.

106. 21:54

     Um, so if you wanna try out our Llama 3 endpoint, um, so you can go to our website, uh, sambanova.ai,

107. 22:05

     and then, uh, click on, uh, Samba-1 Turbo. So this is where you can access, um, our, um, chat infra. And here you have, um, options to select, um, various models, you know, Llama 3, um, 8B, 70B, our CoE, Mistral, and even some of our, um, in-house, um, models which we've trained ourselves like, you

108. 22:30

     know, [REDACTED:username] or others. Um, so yeah, we'll do [REDACTED:password] demo of Llama 3, um, 8B, and I'll ask it, uh, the following question. Um, so, you know, create [REDACTED:password] three-day-[REDACTED:password]-week workout schedule for intermediate fitness level.

109. 22:46

     Um, let, let me actually, uh, redo it again. You can see, you know, it gets, um, instant, uh, response. And then for the, uh, performance metrics, so you can see the, um, insights here.

110. 22:59

     Um, so few things, uh, to note. Um, so of course, we have [REDACTED:password] pretty high, uh, throughput, which is, uh, [REDACTED:password] thousand tokens, um, per second. But that's not only, um, the end of the story.

111. 23:11

     You know, we also have [REDACTED:password] pretty small, uh, time to first token, which is basically the, um, input inference time of point zero nine seconds. And we also have [REDACTED:password] pretty small, um, end-to-end, uh, total inference time of point six five seconds, right?

112. 23:27

     So with our full stack, um, platform, we can achieve high throughput and very small, um, inference time. And if just want to do [REDACTED:password] comparison, let's say with, um, ChatGPT, uh, for instance, um, if you ask, uh, the same question, just copy it.

113. 23:47

     Um, yeah, you can clearly see the, uh, difference in speed.

114. 23:57

     And the, uh, other, uh, cool thing which, uh, we built, so we have this, uh, real-time, uh, option where as you write your prompt, you can instantly see the model's response, and as you change the prompt, you can also see the change in the response.

115. 24:16

     Yeah. So let's say, I don't know, "Hi, I

116. 24:22

     want to write an email about blah, blah, blah," right? So the point here is that, you know, with this, um, real-time option, you know, you can do, um, real-time chatting, and this can be helpful, for instance, if you're drafting emails or even if you want to do some, like, real-time, um, prompt engineering, let's say.

117. 24:45

     So yeah, that's it for the, uh, Samba-1, uh, Turbo. I can give maybe [REDACTED:password] few minutes for folks to try it out before we move on to the, um, hands-on.

118. 24:54

     Um, so again, you go to, uh, our website, sambanova.ai, and then you click on Samba-1 Turbo. Um, so you can do it from your laptop or from your cell phone.

119. 25:08

     Yes. Yes, question. Can we tweak the generation parameters like top P- Yes, yes, yes, yeah. Uh,

120. 25:20

     let me see how to do it on this, uh, UI.

121. 25:27

     Okay. I think, uh, from this UI it seems it is fixed but, um, in the hands-on once, uh, you will be calling our endpoint, um, from the API key, uh, we will be changing some of the, um, configs as well, so.

122. 25:42

     Yeah. Any other, uh, question?

123. 26:10

     Yeah. Were you able to, uh, try it out? Uh, okay. Great.

124. 26:14

     Okay. So I think, um, yeah, we can move on to the, um, hands-on, uh, portion. Um, so what I'll do first is just, um, introduce what, uh, um, I'll be talking about in this part, and then, uh, we'll dive in, um, to the, uh, hands-on.

125. 26:31

     So yeah, we prepared two, uh, exercises, um, for you today. So the first one is [REDACTED:password] basic, um, example to get you started. Um, so here, um, I'll be showing how you can load, um, environment variables, set up the, uh, SambaNova API key, um, initialize the LLM, and do [REDACTED:password] simple, um, uh, inference call in Python.

126. 26:55

     And the, uh, second, um, example is [REDACTED:password] more practical one, so we will be, uh, building and deploying [REDACTED:password], uh, Q&[REDACTED:password] system, um, with RAG for, um, enterprise search, uh, using our platform.

127. 27:09

     And we will also be using, um, other libraries and packages like, you know, LangChain, um, various data loaders, uh, E5 large V2 embedding, um, ChromaDB vector store, and of course, the Llama 3, um, endpoint, which runs at the speed of one thousand tokens per second.

128. 27:30

     So yeah, let's start with the basic, um, example. And, um, if you wanna f-follow along, so you can, uh, yeah, go to Google, write AI Starter Kit, uh, SambaNova, and then, uh, click on this link, or you can just, um, write this, uh, URL.

129. 27:52

     So this is our, uh, starter kit, uh, repo. Um, we have [REDACTED:password] collection of open source, um, examples on, um, GenAI apps. And, um, yeah, once you get there, yeah, you can go to, uh, Workshops,

130. 28:10

     AI Engineer 2024, um, Basic Examples. So I'll go over the README and then do [REDACTED:password] live demo, um, of the work of this, um, exercise, and then I'll give you, uh, some time to, uh, try it out.

131. 28:25

     So, um, yeah, first, um, you'll need to clone, uh, this, uh, repo here, and then, uh, you'll need to, uh, create [REDACTED:password], um, .env file, um, in the repo root, uh, directory.

132. 28:40

     So this is where, um, we will be, uh, specifying the, uh, SambaStudio, um, API key. Um, yeah, so let me show you how this is done. Uh, it's gonna be...

133. 28:54

     Yeah, I guess with the two mic, uh, I have to find [REDACTED:password] way. [clears throat]

134. 29:08

     Yeah. Yeah, so I've already cloned, um, the repo. Yeah, the repo. And then, uh,

135. 29:20

     at this level, this is where you will need to create the, uh, .env file. So vi what... In your case, yeah, you'll have to do touch .env, and then,

136. 29:35

     yeah, you can add, uh, the- these information here. So for the first, um, hands-on, that's the, uh, um, only thing that you'll need. Um, and you can access or copy those from our, um, Discord, uh, channel.

137. 29:52

     So, yeah, if you go to Discord- Events.

138. 30:09

     Um, yeah, so you can copy either, um, of these keys. So we have two dedicated, you know, endpoints, um, for this workshop.

139. 30:18

     All right. And, uh, once we finish setting up the .env, uh, the third step is basically, you know, installing the, um, packages. So for this one, you can either do it with, you know, Conda or [REDACTED:password] Python environment.

140. 30:33

     Um, so first, yeah, we will go to the basic examples, um, repo. So just CD and then the repo path. Then you can create, um, [REDACTED:password] Conda, uh, environment.

141. 30:46

     So I would recommend using, uh, Python, um, three point ten. And then you activate your, uh, Conda environment, um, and here we name it basic_x. And then, yeah, you just, um, install the requirements with the, uh, pip install minus r, um, requirements.

142. 31:04

     And then you can use this, um, line to just, um, link the kernel to your, um, notebook. Okay?

143. 31:13

     So if I go to my terminal. Yeah, so we'll go to, uh, workshop,

144. 31:27

     AI engineer, basic examples. Okay, so this is where you can create the Conda. So I've already done it, um, beforehand, so I'm just going to activate the, uh, environment.

145. 31:40

     So... Yeah, and this is the, uh, requirements file.

146. 31:59

     So we only have like [REDACTED:password] few packages that is needed.

147. 32:06

     Yeah, and that's it, uh, for the installation. Um, so once this is done, yeah, you should be able to open, uh, the notebook. And again, you can do it, you know, from the terminal, which will route you to [REDACTED:password] browser or, um, uh, through VS Code.

148. 32:22

     So if you want to do it through the terminal, you just write Jupyter Notebook and then

149. 32:30

     the name of the notebook. So it's going to be, uh, example, uh, with, uh, SambaStudio.ipy, uh, oneB. I'm actually going to do the demo via VS Code just 'cause I can show you the, uh, timestamp.

150. 32:46

     So I already have this, uh, set up.

151. 32:52

     Okay, so you know, we are in the basic examples repo and then example with Samba Studio. And again, if you're doing it, uh, with VS Code, just make sure that you have the, uh, kernel set up.

152. 33:02

     Um, so this is [REDACTED:password] pretty basic, um, script. So we will first, uh, let me just restart it. Yeah. So yeah, we will be loading,

153. 33:16

     uh, the libraries. Um, so we [REDACTED:password]-actually have [REDACTED:password] wrapper with LangChain. So this is, um, where, uh, we will be, um, initializing and calling our endpoints. The second step is to load the, um, environment variables.

154. 33:32

     So these are actually the information which we added in the, uh, .env, uh, file.

155. 33:38

     And then, uh, we will initialize the, um, LLM. So yeah, we will be using the SambaStudio, um, wrapper, and, uh, we set the, um, SambaStudio API key, and then we specify the model config.

156. 33:53

     So I had [REDACTED:password] question earlier about, um, the model configs. Um, so this is where we can, uh, set this up and play with this. Um, again, I think most of you are familiar with these configs.

157. 34:04

     So, you know, do [REDACTED:password] sample. If you set it to false, this is basically [REDACTED:password], um, deterministic output. If you set it to true, then it becomes, uh, probabilistic.

158. 34:13

     You can change the temperature and also the max tokens, um, to generate. And also, this is basically our CoE endpoint, right? So we have one endpoint through which you can actually call, um, different models.

159. 34:25

     And in this case, uh, we set the expert to, uh, Meta Llama 3, um, 8B instruct. Okay, so I'll be running the, uh, the step. And yeah, we have now our model loaded, and now we're, uh, good to go.

160. 34:40

     So, um, I'll first show you how you can do an inference call using [REDACTED:password] simple, um, invoke method in LangChain. Um, so just write llm.invoke and then add your prompt.

161. 34:54

     And the prompt here is, um: What is the [REDACTED:password] of, uh, France?

162. 35:01

     And what you'll, uh, notice is it gives the right answer, but also, um, give you other stuff which you didn't ask for. And this is [REDACTED:password] common thing with open source, um, models because when you ask [REDACTED:password] prompt, you need to include the, uh, the special tags, right?

163. 35:19

     So in the case of, uh, Llama 3, you can get it from [REDACTED:password] Meta model card. So you'll have to actually, in the prompt, add these, uh, special tags or tokens.

164. 35:30

     Um, in particular, you have to let the LLM know that this is the beginning of text, and this is where you will insert the user query, right? So and then let the LLM know where it, uh, needs to, um, answer.

165. 35:44

     And once you have the special tags, um, inserted, um, now you should be able to get the right response. So this is the first way to do the, um, inference call.

166. 35:53

     The other way is to do it via [REDACTED:password], um, LCL in LangChain. So basically, um, you can use the LCL to connect, um, [REDACTED:password] prompt template with LLM and an output parser.

167. 36:05

     And LangChain has very Um, templates that you can use. So, um, we are asking the same prompt. It's just that the main difference here, we are adding the country as [REDACTED:password] placeholder, and then when you prompt the model, you can actually specify the value of this country, right?

168. 36:24

     Yeah. So that's it for the basic, um, example, and as I said, this is just to get you started. So, um, yeah, we can spend ten minutes for you guys to try it out.

169. 36:36

     Um, me, Varun, and Rachelle will be here to help and, um, yeah, then we can move on to the second, um, exercise.

170. 36:47

     I could use [REDACTED:password] hand if you're available. [laughs] Um, it might just be returning [REDACTED:password] string. Oh, that's it. That's it. It could

171. 37:02

     be... It could be, um, issues. Yeah, I'll move around if also people have questions. It could be our environment.

172. 37:08

     I think on the data loading question, it's open now. Never mind.

173. 37:13

     Do you think it's connected with

174. 37:15

     Oh.

175. 37:17

     Okay, cool. I think, uh, many of you were able to try out this simple, um, exercise. But yeah, we'll move now to the, uh, second, uh, one. Varun, do you wanna? [microphone feedback]

176. 37:31

     Um, yeah. Yeah. So, um, as I said earlier, this is going to be [REDACTED:password], uh, Q&[REDACTED:password], uh, system, um, with RAG.

177. 37:51

     And, uh, if you wanna follow along, um, again, from the same repo, uh, go to workshop, AI engineer twenty twenty-four, and then EQRAG. And like the previous exercise, I'll go through the README, do [REDACTED:password] live demo of the installation and the run setup, and then, uh, give you some time to, um, try it out.

178. 38:13

     And the, uh, app here, we have two versions of it, one with [REDACTED:password] Jupyter Notebook and the other one, um, with, uh, Streamlit, which is [REDACTED:password] UI-based. And before I, uh, jump into the hands-on, just wanted to give [REDACTED:password] brief, um, overview of what RAG is.

179. 38:33

     Although I'm sure many of you already know, uh, this concept, but just for, um, completeness. So yeah. So RAG is [REDACTED:password] technique that, um, we can use to supplement, um, LLM with, um, additional information from, um, various sources to improve the model's response.

180. 38:51

     And RAG is very helpful, um, if you want to use an off-the-shelf LLM to ask [REDACTED:password] question, um, beyond its, uh, training data, or if you want to, um, have the LLM access to up-to-date information without, um, retraining it.

181. 39:08

     Also in RAG can help reduce, um, hallucinations in some, uh, contexts. And [REDACTED:password] typical, uh, RAG workflow, um, consists of,

182. 39:21

     yeah, the, uh, following steps. So we first have, um, document loading and parsing. So this is where, um, we can use [REDACTED:password] data loader to actually load the data into [REDACTED:password], um, digital text that we can edit and format.

183. 39:39

     Um, and, you know, various data loaders are, um, available depending on the, um, extension of the file you- you're using, so in [REDACTED:password] PDF, text, um, PowerPoint. After this, we have [REDACTED:password] splitting step.

184. 39:52

     So this is where, um, we will be splitting the document into, um, smaller chunks. And, you know, the chunk size and the overlap, all of these are, um, hyperparameters.

185. 40:04

     And the, uh, next step is, um, vectorization. So this is where, um, we will be using [REDACTED:password], um, embedding model like E5 large V2 to map each, uh, chunk to [REDACTED:password], [REDACTED:password] numerical vector.

186. 40:18

     And, um, we can store, you know, the vectors along with the content and the metadata in [REDACTED:password] vector store like Faiss and ChromaDB. And today, um, we will be using, um, ChromaDB, which is, um, open source.

187. 40:33

     And again, the whole, uh, goal of this embedding is that it allows us to do, um, like semantic similarity and, uh, semantic search. And in the retrieval step, um, this is where we ask, uh, the question, which is going to also be embedded into the vector, um, space.

188. 40:53

     And then we have [REDACTED:password] retriever, which is going to, um, retrieve the closest, uh, chunk vectors to the query vector according to some, uh, similarity metric. And we can also add [REDACTED:password] re-ranker, which can re-rank the retrieved, uh, chunks, um, per, uh, relevance and also remove some of the, um, unnecessary chunks.

189. 41:17

     And the last step is basically Q&[REDACTED:password], um, generation. So this is where you provide the LLM, um, with the query and the final retrieved, uh, chunk to get the, uh, grounded response.

190. 41:31

     And, um, yeah, maybe also would like to precise that, um, in this exercise. So the... We will be using third-party tools for document loading, splitting, and, uh, storage. For the embedding model, um, you can either run it on CPU or on our hardware, and we'll be doing both to show the differences.

191. 41:51

     And for the LLM part, this is gonna be done on our, um, hardware.

192. 41:57

     So yeah, that's it for the, uh, overview, and I think, yeah, we can move on to the, uh, README. Um, yeah. So we first, uh, clone, uh, the repo.

193. 42:07

     I think if you've done the other, um, exercise, then you don't have to do, um, this step. Um, same thing, um, after this, we will set up the, um, environment, um, variable.

194. 42:19

     Um, so for this one, we will be using, uh, SambaStudio, um, API for the LLM, and also, uh, we will be using [REDACTED:password] embedding, um, API as well. So both are available on Discord.

195. 42:36

     Yeah. Let me show you, uh, in the terminal.

196. 42:42

     Yeah, and actually, I would also recommend to deactivate your previous, um, environment.

197. 42:51

     Okay, so we go to the EQRAG, uh, repo. Um, yeah, actually, for the dot and file, this one, you'll have to put it, um, at this level, the AI starter kit, okay?

198. 43:03

     So for this one, we will need the embed endpoint and API key and also the, uh, SambaStudio, um, endpoint and, and, and API key.

199. 43:15

     All right. Then we go back to the, uh,

200. 43:20

     EQRAG folder, and then we are ready to go with the installation. So here we will be needing more packages. Um, so first, um, Tesseract, uh, this is our OCR, um, data extractor.

201. 43:34

     So let's say you're using [REDACTED:password] Mac, um, just run brew install tesseract, and this one you can do it outside your local, um, environment. It should take you, you know, [REDACTED:password] few minute to install.

202. 43:46

     And you also need, um, Poppler, if you don't have it already. Um, you can just do brew install, um, poppler.

203. 43:55

     And then, yeah, we will need to set our, um, virtual environment. So since this is [REDACTED:password] more complicated, um, exercise, I would just recommend to use, um, the, you know, default option.

204. 44:07

     So the, uh, Python environment with [REDACTED:password] Python, um, three point ten. So if you don't have, uh, Python three point ten, um, let's say on [REDACTED:password] Mac, you can install it using this, uh, command here, and then you can add the path to your shell, like bashrc or, uh, zshrc, um, using this command here.

205. 44:30

     And then you can just source your, uh, shell file.

206. 44:35

     Okay? And then, yeah, we will, uh, go to the, uh, repo, if you haven't done already, um, create your Python environment. Um, so again, if you added this step here, then your laptop should recognize the Python three point ten, then minus m benv, and then the name of the environment.

207. 44:56

     You activate that environment, and then you run the install script. So this should take, I would say, you know, five minute if you have, um, good internet, and once this is done, you also need to install ipykernel and also, um, link your kernel to your, um, notebook.

208. 45:17

     So when we tested this on different laptops, um, you know, some folks were having also sometimes NLTK and SSL certificate. So you might also need to, uh, run this script here.

209. 45:36

     Yeah, it's in my bag, basically. Yeah. So let's activate the, uh, uh, Conda, the Python environment.

210. 45:51

     Yeah, and this is the, uh, requirements file. And as, and as, as you can see, right, we have, uh, more packages here.

211. 45:59

     And then, um, yeah, this is the file which you may need to also, um, run as well.

212. 46:07

     Yeah, and once this is set up, that's all you need to, uh, run. Yeah, that's fine. I can do it.

213. 46:15

     Yeah, uh, the, the notebook. And as I said earlier, right, so we, uh, have the app in [REDACTED:password] notebook and in [REDACTED:password] Streamlit. So for the, uh, notebook, um, again, right, you can open it from the terminal or, uh, from VS Code.

214. 46:31

     Yeah, let me do it, um, from VS Code.

215. 46:40

     Yeah, so you'll go to the EQRAG, uh, repo, notebooks, and then, um, rag_lcl.ipynb. And this is going to be our main, uh, script, which is using actually, you know, files and modules from other, um, files.

216. 46:56

     So in particular, uh, we will be using the, um,

217. 47:02

     the document retrieval dot py and also, uh, some files, uh, from the vector DB, which I'll explain in, in more details. All right, so, um, let's go maybe first over the structure of the notebook.

218. 47:17

     Um, so we first, um, import, uh, the libraries and set the required path. So you don't have to do, um, anything at this point, but, yeah, just know that the kit directory, this is the absolute path for your EQRAG, and then the repo directory, this is the absolute path for the, um, AI starter kit.

219. 47:38

     So let's run this, uh, repo. And then for the, uh, document, uh, loading and splitting, um... Yeah, so we added, um, you know, various, um, data loaders, you know, like PyPDF and Unstructured.

220. 47:56

     And which, uh, data loader you want, uh, you can set this up, um, in the config file, which is, um, here. So yeah, I'm just gonna do [REDACTED:password] test with PyPDF two for now, but we can switch to other, um, data loaders, um, afterwards.

221. 48:15

     And, um, yeah, for the experiment, I will be using the SN40L, um, paper. So this is an archive paper which we, um, recently, uh, submitted. So this contains, like, information about the stack, the hardware, and our, um, uh, CoE.

222. 48:31

     Um, I can show you, uh, the paper as well. Um, we also have it in, uh, GitHub, but you can also, um, upload your own, um, PDF.

223. 48:40

     And yeah, you'll have it to put it under Data, temp, and then...

224. 48:50

     Yeah, this is the, uh, paper that I'll be using, uh, uh, for the demo.

225. 49:00

     All right. And, uh, let's go back to our VS Code. So this is where, um, you know, we will be using PyPDF to actually load the content, um, into [REDACTED:password] list.

226. 49:12

     And then, um, we are using the, uh, recursive, uh, splitter from, um, LangChain. So what I'll do first is go into the notebook, and then we can go into the functions in more details if you are, um, interested.

227. 49:25

     Yeah, so let's run this step. Yeah, and, um, in the config, um, so I set the, uh, chunk size to twelve hundred, and then the, uh, chunk overlap to two forty, but again, you can change those configs.

228. 49:48

     So for this fifteen-page, um, PDF, we end up getting, um, eighty-nine chunks.

229. 49:54

     The next step is the, uh, vectorization and storage. So this is where, um, we will be using the embedding model to map each, uh, chunk to [REDACTED:password] embedding vector.

230. 50:05

     And as I said earlier, right? So we can actually run the embedding model either on CPU or on RDU. So RDU is basically our AI chip. So if you want to do it on RDU, then you'll have to go to the config.yaml file, and then, uh, set the type to, uh, SambaStudio.

231. 50:24

     And then, uh, batch size, so this one you can have it either one or, uh, thirty-two. So thirty-two means that we are actually processing thirty-two chunks at the same time.

232. 50:34

     And this is [REDACTED:password] standalone, um, endpoint, so yeah, the CoE is set to, um, false. And I'll show later, um, if you want to run the endpoint, um, for the embedding on your laptop, uh, how you can change the configs, um, for that.

233. 50:49

     All right. And then, yeah, let's run the, uh, vectorization. And after this, we're actually storing or indexing the embedding vectors, um, into the ChromaDB vector store.

234. 51:02

     Think, uh, this should take around, uh, ten seconds. Let me see what's happening.

235. 51:11

     I just restarted. Yeah, yeah. Yeah, always good to restart the notebooks. I'll just go over the steps again. [clears throat]

236. 51:31

     Okay. Yeah, so it took, you know, four seconds to embed the whole thing. And, uh, yeah, this is where, um, we will initialize our, um, QA chain.

237. 51:47

     Um, so again, we have different, uh, wrappers and classes, which I can go over it in details, um, afterwards. But for now, let's just execute, uh, the cell. Yeah, now you're ready to go, um, ask [REDACTED:password] question.

238. 52:01

     And then what happens is, right, through this QA chain, the question gets embedded to the vector, uh, space. We retrieve the, uh, top K chunks. So in this experiment, yeah, we have this set to three, since I won't be using [REDACTED:password] re-ranker, but I can also show you how to use the re-ranker.

239. 52:21

     And then, yeah, the question and the context are provided as, uh, co- basically, uh, context for the LLM to get the answer. So

240. 52:32

     yeah, what is [REDACTED:password] monolithic model? And you can see that, right, the response is, um, instantaneous.

241. 52:38

     Yeah, so that's it for the, um, uh, experiment. Uh, let's maybe try ask it [REDACTED:password] bit more complicated question. So if I open the PDF again... Uh.

242. 52:58

     I think, yeah, there was [REDACTED:password] table in the PDF. [clears throat]

243. 53:06

     Yeah, like this is [REDACTED:password] table, you know, showing, um, operation intensity versus fusion level. So yeah, let's see if PyPDF is able to, um, uh, retrieve, uh, some of the information from this table here.

244. 53:20

     So I already have the, uh, questions prepared. Let's try to access those.

245. 53:28

     Yeah, so yeah, it got the response, right? So four, ten point four, basically. Um, and again, if you end up having, like, more complicated, um, tables in PDF, um, in this case, I would recommend to switch to the Unstructured, um, data loader.

246. 53:40

     And for this, yeah, all you have to do is just go to the config file and then, um, change, uh, PyPDF to, um, Unstructured.

247. 53:49

     So yeah, that's it for, uh, this second, um, exercise. Um,

248. 53:56

     I can go into more details about each, uh, function if you're interested, um, or have you try it out first, and then we can maybe, uh, come back and then, um, go over the, the, uh, functions.

249. 54:10

     Um, yeah, so do you wanna maybe try it out first, I guess? Okay, great. [upbeat music]