Forget RAG Pipelines—Build Production-Ready AI Agents in 15 Minutes

Talk transcript

1. 0:00

   [on-hold music] Welcome, everybody.

2. 0:15

   Thanks for showing up bright and early this morning. We're gonna make your life easier for RAG, so just hang in there. If you wanna get started, we have the link right there, contextual.ai/aie25.

3. 0:29

   That's gonna get you a notebook. It's gonna get you the Getting Started link, which will get you an API key that you'll be able to use for the notebook.

4. 0:35

   So feel free to jump ahead and start on those pieces like that. All right. So let me first introduce the team that's here. So I'm Rajiv Shah. I'm the chief evangelist of Contextual AI, so I do a lot of these talks, workshops.

5. 0:52

   I endlessly talk about AI, which I'm kind of in this position. For background, I've been in the AI space for a while. Before this, I was over at Hugging Face for a number of years as well.

6. 1:02

   So besides me talking, we also have Nina. You wanna introduce yourself?

7. 1:07

   Hi, I'm Nina. I'll be presenting along with Raj. [clears throat] This is my fourth week at Contextual, so very exciting for me. Uh, thank you all for joining us this morning.

8. 1:16

   I've been working in, uh, NLP and language modeling for the past seven or eight years, um, really since, uh, BERT was considered a large language model and, uh, I'm excited to show you how you can, uh, simplify your RAG pipeline with Contextual today.

9. 1:32

   Thanks.

10. 1:34

    For you.

11. 1:34

    And besides Nina, we also have two other members of our team that are in the back that if you raise your hand and have questions, they will come to you.

12. 1:41

    So one of them is Matthew, who's a platform engineer. So if you have questions about how we scale up, what's the-- what are the things going on in the back end?

13. 1:49

    He's the person for you. We also have John on our team, who's one of the solution architects. So if you're like, "Hey, how would I integrate this into my environment?"

14. 1:57

    He's the guy who's gonna be able to handle that as well. So let's kick this off.

15. 2:03

    What I wanna do is for you to all get-- One of the big pieces out of this is we can treat RAG like a managed service. We don't go out anymore and train our own language-- large language models.

16. 2:15

    If we're working with embeddings, we don't build our own vector databases. Same thing with RAG. We can treat RAG just like any other managed service.

17. 2:25

    And so what we're gonna do today is I'm gonna do this quick introduction here, but then we're gonna get you started right away with building this RAG agent, with ingesting some files.

18. 2:33

    And Nina and I will go back and forth, where I'll go and give you a bit more of an overview of Contextual. You'll build up an agent, run a number of queries against it, get a feel for it, and then I'll do a deep dive on some of the advanced settings that developers like around extraction, re-ranking, retrieval

19. 2:50

    like that. And we'll end with a little bit on evaluation as well as how you can use, um, MCP, for example, and connect your RAG agent to Claude Desktop as well.

20. 3:02

    Now, to start with, we all know the value of AI, right? We're all here 'cause of the importance of AI. But for many of you also know the struggles of AI, how it's easy to build that [REDACTED:username] with ten documents when you're doing RAG.

21. 3:17

    But when you go to scale that out, all of a sudden it's hard to extract across thousands of documents of diversity, right? Your accuracy isn't there. Your users are complaining that they aren't happy with it 'cause they don't know how to query properly.

22. 3:30

    Now, what we at-- We at Contextual have focused on this. So our founders here, Dao and Aman, come from working on RAG for a long time. Dao was an author, was leading the team on the initial RAG paper.

23. 3:47

    Aman was with him at Meta. I knew both of them from my time at Hugging Face, where I was helping people do RAG years ago on question answer systems, and I would be able to pull their researchers in to help work with customers as I was cobbling together kind of open-source components to build these pipelines.

24. 4:05

    Well, they saw the need in that. They wanna change how the world works, and so Contextual has come around for focusing on enterprise. And we still have this dedication to AI research, and you'll see that as we go through our product as well.

25. 4:20

    Now, for, again, just the basics, just in case there's n-some of the in here not familiar with it, right? When we're talking about RAG, retrieval-augmented generation, the value here is when you have lots of that unstructured enterprise data, we want to be able to understand it.

26. 4:37

    So RAG allows us to take that information. A simplif-- A very simple RAG pipeline uses a vector database to keep all of that, use cosine similarity to find similar pieces, pass it to an LLM.

27. 4:50

    That's the simplest. We're gonna get much more complicated today like that. And what I wanna do is show off our platform to you. The platform is built for a couple of different l-levels.

28. 5:01

    So one is there's a no code. So if you're a business user, you just have a bunch of docs, you wanna be able to ask questions to 'em. We've made it very simple with opinionated defaults on that.

29. 5:12

    Now, if you're a developer, on the other hand, you know what you want in your RAG pipeline. You're spending time evaluating it. You're spending time tweaking it. Well, we built a platform where you can then orchestrate and change.

30. 5:25

    How do I wanna do querying? How do I wanna do generation along the ways? And finally, some of you might already have RAG-- You might already have RAG pipelines in production where there's just, uh, one component that isn't working that well.

31. 5:39

    Maybe your extraction isn't working that well or you want a better re-ranker. Well, we've made our system modular, so if you just wanna use one piece of it, you can.

32. 5:50

    And all we focus on is RAG. Other companies focus on other pieces. We just think about building RAG. And again, the reason we're here is as you go and try to do this in production, you'll see, like once you get to lots of documents, it gets messy.

33. 6:05

    All of a sudden, then you're orchestrating a big environment full of lots of different models. Now I need a BM twenty-five. I need a re-ranker. Putting all that stuff together Ends up sucking up your time.

34. 6:15

    It's fun to do the first time, but after that, [laughs] it ends up sucking up your time, ends up with slow, kind of costly, um, RAG pieces. So

35. 6:25

    that's enough to get started. I'm gonna turn it over to Nina g- who's gonna get you started on building that first agent.

36. 6:38

    All right. We're on the clock. [laughs] Fifteen minutes to get our agent up and running. Um, just kidding. We'll take a little longer than that to explain everything. But how you can get started is, um, you can find the notebook and the Getting Started page at contextual.ai/aie25.

37. 6:56

    And then the GUI for Contextual is at app.contextual.ai, ai. That's what that Getting Started button will point you to. And so we're going to start by loading a few documents.

38. 7:08

    Um, we're gonna load some financial statements from NVIDIA and some fun, furious correlations to see how our RAG system treats data that's not, uh, fitting with conventional wisdom. And then we'll try out a few queries, uh, cont- including quantitative reasoning over tables and some data interpretation.

39. 7:27

    And so I'm gonna have, uh, these two screens here side by side, the notebook on the left and the platform on the right. So you can get started by signing up in the platform if you haven't already.

40. 7:43

    That's at app.contextual.ai. That's where you're gonna get your API key. So everyone take maybe a minute or so. You'll need to log in to that and, uh, you'll need to set up your workspace.

41. 7:55

    I already have my workspace set up, and I'm actually using, uh, my reg- uh, not my work account just so that you'll see exactly, um, what a general access user would see on their screen on my screen.

42. 8:08

    And just a tip for setting up those workspace names, it's gonna have to be a unique name. So you can't call it AIE-[REDACTED:username]. I just throw my name into the names just to make it a little bit more unique.

43. 8:20

    So, um, so everyone just, uh, take a minute to do that. And, uh, to help me get a sense of where folks are, just briefly raise your hand and put it down once you have your workspace set up and you see this welcome screen.

44. 8:38

    Okay. Folks are making fast progress. Um, yes, Matthew?

45. 8:44

    Can you maybe zoom in on the, especially on the workshop? Yeah. Like, or sorry, on the, on the, on the student text a little bit.

46. 8:53

    Yeah.

47. 8:54

    Thank you. And maybe also on the plots on the right.

48. 9:01

    Thank you.

49. 9:01

    Thanks. Um, so yeah. So we can start by, in the notebook, doing our pip installs. Oh, start by making a copy of the notebook. So you can just do go to File, Save a copy in Drive,

50. 9:19

    and then that will create a copy for you of the notebook so that you can save whatever changes you're making to it, if you wanna make any changes.

51. 9:31

    It is a bit slowly making a copy. Um, but yeah, after the pip installs and the imports, to get your API key, you go into your Contextual app, click on API keys here on the right.

52. 9:46

    Copy.

53. 9:46

    Save a copy of the notebook, and then in our Contextual platform on the right, you can go to API keys. Um, click on Create API key. So Nina,

54. 10:01

    [REDACTED:username] is my name, and then we'll copy that. And then we can go to our notebook secrets on the left here, the key there, and then add a new secret, and then you'll just add contextual_api_key.

55. 10:20

    Load up.

56. 10:20

    Yeah, it's refreshing. [laughs] Um, but if yours is working, then, uh, you can--

57. 10:28

    I'll just do it in the notebook without making a copy. contextual_api_key, and then just paste the value there. And then you'll save that. Um,

58. 10:39

    and yeah, that's all, that's the only API key we need. For those of you that have spun up RAG on your own, it's often an API scavenger hunt just to get started, and this is literally the only API key we're gonna use for the whole rest of the workshop.

59. 10:53

    So we have just finished the, uh, hardest part of setting this up.

60. 10:59

    Mm.

61. 11:01

    So once we've saved those credentials, then we're going to set up our client here. So we're just going to, uh, run the client in our notebook. So here I'm on the left in the notebook, and then we're going to create our data store.

62. 11:16

    So you can change the name on this, uh, data store name

63. 11:21

    variable, um, and then it's going to see if there's an existing data store or load that one. [clears throat]

64. 11:31

    Um, and then after that, which seems to still not be running.

65. 11:45

    Okay. Yeah, after the pip installs and the imports,

66. 11:55

    that data store is going to be where you're going to be loading all of your documents. And then we have

67. 12:04

    some code here in step two, uh, that will download those files that I mentioned earlier to your notebook. So that's gonna be a few quarterly reports and-

68. 12:15

    Spurious correlation reports. And so here in step two, you'll see that it's fetching the data and then uploading it to the data store. And then you can go to your app.contextual.ai, and,

69. 12:31

    uh, this is the AIE [REDACTED:username] data store that I made just here. So if you click on your documents, you'll see that they're processing. And once they're done processing, um, you can click on these three dots, and then once the files are loaded, you'll be able to inspect them and see how those files have been parsed and

70. 12:53

    ingested. So I'll just take a quick look at the run I did yesterday of all these same exact files. Um,

71. 13:04

    and so if you click on Inspect, once your files are loaded, for example, if you have a table, you're going to see the raw text and then also the rendered preview.

72. 13:15

    So you can see this table is, uh, has both the structure and the contents of the table in this image below, and you're welcome to take a look at these.

73. 13:24

    Uh, the numbers match very perfectly and the structure. So that's kind of step one of getting your RAG system set up, is having really accurate parsing. And you can look at some of the, um, image files.

74. 13:36

    We chose some spurious correlations, which are a fun way to test how much a RAG system will actually, uh, answer based on your own documents rather than its conventional wisdom.

75. 13:46

    And so you can see there also there will be raw text and, uh, rendered previews, um, with like key data points from figures and, uh, summaries of the figures in addition to the text and other information extracted.

76. 14:02

    So while your files are ingesting, uh, Raj will come back and expand on the,

77. 14:11

    uh, on the platform. [laughs]

78. 14:16

    All right. Seems to me I need to update this slide with Wi-Fi issues as well. But any of you who've built RAG systems before know that there's several components of it.

79. 14:25

    There's a lot of tasks you have to do, right? You have to think about building out an extraction pipeline. Besides that, when you're thinking about accuracy, am I-- how am I gonna chunk out?

80. 14:35

    What am I gonna do for chunking? What am I gonna do for re-ranking? And then finally, you have to think about scaling it, and there's a lot of these issues that come up.

81. 14:43

    And again, it's interesting the first time you do it, but maintaining this stuff after a while gets tiresome, and that's where Contextual comes in, where it's really an end-to-end platform for managing RAG.

82. 14:54

    And our platform runs in our SaaS. It can run in your VPC. We have a UI for those business users, but we also have an API, REST API endpoints as well.

83. 15:05

    And when the-- within the platform, we manage all those pieces for you, so you don't have to worry about maintaining a vector store for your embeddings. And the way it works is, you'll come with your structured and unstructured data.

84. 15:16

    We bring that in. We have a document understanding pipeline, and that's the extraction part that we're gonna walk through here, where we need to cleanly be able to pull all that information out of your documents, your tables, your images.

85. 15:29

    Once we have that, we chunk that up, and then we go through, uh, the best practices around retrieving. So we have a mixture of retrievers, so BM twenty-five, as well as an embedding model.

86. 15:41

    We have a state-of-the-art re-ranker that we've trained ourselves that's in our pipeline, and then we pass it all to a grounded language model. Now, we don't use a model from OpenAI or Gemini.

87. 15:52

    We trained our own grounded language model because for RAG use cases, we want the model to be grounded. We want it not to give its own advice just because it can pass the law exam or know everything about medicine.

88. 16:06

    We don't want it to use its knowledge when it's answering questions. We want it to instead stay grounded, respect the context that we're giving like that. And finally, there's lots of different ways we can output this, and I'll show you this along the way as well.

89. 16:22

    Now, for each of these parts, we have that academic pedigree, as I mentioned earlier. So one thing we always do is look at the academic benchmarks. Nowadays, we have lots of customer dataset points that we use as well.

90. 16:32

    But whether we think about the end-to-end accuracy of our platform or the specific parts around document understanding, retrieval, grounded generation, each one of those we focused on to make them state-of-the-art for what they do like that.

91. 16:48

    So once we've done that, when you go to use them, there's a lot of different uses that you can have for this. I think often when we think of, initially of, of RAG, we think about, "I'm gonna build a question-and-answer chatbot," right?

92. 17:03

    Like I'm gonna ask a question, get back an answer. Well, yes, that's one use of what you can do with this, but we have customers like Qualcomm who use Contextual on their website.

93. 17:15

    So if you go live to their website right now and you ask a question there, it's powered by Contextual on the back end, and you can see similarly the question, the answers, right?

94. 17:26

    Our feedback module sources are all available there.

95. 17:30

    Besides that, we have other customers, for example, some folks in the finance-- financial spectrum. Ugh, my audio.

96. 17:39

    And they're thinking about automated workflows. How can I take all the unstructured data I have, structure it to make it useful for folks? Well,

97. 17:47

    we have APIs. You can hook up those APIs, be able to run them against here.

98. 17:55

    Is it good? Ah, okay, no worries. But we can take all that unstructured, use our RAG agents in, in the context of something like a spreadsheet or some other workflow to be able to use that.

99. 18:08

    Finally, we can connect it with lots of other tools.

100. 18:12

     I'm gonna put this in there even though I know it's not gonna work. But just for all of you, we can show you, and I'm gonna show you at the end here today, how you can, for example, integrate the RAG agents with something like Claude with MCP.

101. 18:26

     So that way you can take advantage of the enterprise knowledge that you have in there.

102. 18:32

     All right. So with that, I'm gonna hand it back over to Nina, and let's get you started on your agents.

103. 18:49

     Looks like the Wi-Fi might be down again.

104. 18:54

     Um. Um, yeah, it looks like

105. 19:20

     the speaker Wi-Fi and my hotspot are not working, so we'll just continue with the notebook, and we'll talk through it, and then, uh, I guess, is anyone else's Wi-Fi working?

106. 19:31

     Raise your hand if you have Wi-Fi. Okay, well, you guys can follow along. [laughs]

107. 19:37

     You have the videos too, right?

108. 19:38

     Hmm?

109. 19:39

     You have the vi-

110. 19:39

     I do have-- Oh.

111. 19:40

     Yeah.

112. 19:40

     Oh, it's back up. Okay, cool. Um, so yeah, where we were in the notebook on the left here is, um, hopefully all of your, uh, data stores have loaded now.

113. 19:54

     Um, yes. So as-assuming you had Wi-Fi to upload them, uh, these are the ones I just loaded in the workshop earlier. And then in the notebook on the left, you can see, um, that you can access these files, uh, through the API just the same as you can through the GUI.

114. 20:11

     And then you can have the document metadata, like when it was created, the name, uh, the status of the ingestion. And now we're going to create our agent. Uh, so over here we have our system prompt.

115. 20:23

     Uh, this is kind of just our default system prompt that the agents are loaded with. And then you can run this next block of code here on the left to set up your agent.

116. 20:33

     You can change the name if you'd like. I called it [REDACTED:username]-AIE. And then in, on the right, in our GUI, if you click on Agents, uh, there it is.

117. 20:42

     There's my new agent right here that I just created. And, um, and then you can put in your question based on these documents. So, for example, um, one question that we can ask is, "What was NVIDIA's annual revenue by fiscal year twenty twenty-two to twenty twenty-five?"

118. 21:01

     And so, uh, what you'll notice here is we have our responses. What we were provided was quarterly data, so what the agent is doing is adding that up and then listing it here on the right and then noting that this is based on quarterly data.

119. 21:17

     And then you'll see these little numbers in a circle, the two and the one. So if you click on the two here, you'll actually see the image that this data came from.

120. 21:27

     Uh, and similarly, if you click on one, you'll see the other image. So interestingly, this came from two separate files that were unrelated in any way. They actually don't even have like a standard naming convention for these.

121. 21:38

     I pulled these from the NVIDIA financial reports website, and it knew to reference those documents, um, to pull that information across documents. And then you can do the same exact thing that we just did pro-- um, in the GUI programmatically.

122. 21:53

     So here in step four, we're running our query, uh, and we're getting the exact same response in the API. So this can make it easier to integrate it into your, um, into your application.

123. 22:06

     And so we can also here visualize which files were used to reference in the API. And so now we're printing, uh, the documents here. The Base sixty-four encoding is getting returned with another API query.

124. 22:21

     Um, so just 'cause it's a more fun and visual [REDACTED:username], uh, we'll ask the next few questions via the GUI here. So, uh, NVIDIA used to be a gaming compute company.

125. 22:33

     So I'm gonna ask the chatbot, "When did NVIDIA's data center revenue overtake gaming revenue?" And so,

126. 22:41

     um, the crossover point was in Q1 fiscal year twenty-three, uh, with data center revenue at three point seven million and gaming revenue at three point six. And then here I can click on the one here.

127. 22:55

     So that's Q1 fiscal year twenty-three, and we can look at our slide here, and here it is. That's actually like the exact crossover point where, uh, data center revenue overtook gaming revenue.

128. 23:09

     Now we're gonna look at the spurious correlation files, uh, that we've also loaded. So for those of you unfamiliar, this is a website that, uh, basically data crawls and finds correlations between things that have no correlation.

129. 23:25

     So we're searching for what's the correlation between the distance, uh, between Neptune and the Sun and burglar-burglary rates in the US. That would be some extreme astrology there if that actually [laughs] determined burglary rates.

130. 23:41

     And so what you'll find is that our RAG system that's really, really focused on avoiding hallucinations, um, and really following the data that you've loaded, it will give you that correlation coefficient, which is very high, but then it will also share that context.

131. 23:58

     Um, so otherwi-elsewhere in the document, they talk about how they're doing data dredging and, um, that this is not really a valid statistical correlation. And so you can then also reference where in that document these claims were made, both the statistical correlation and the, uh-

132. 24:18

     And the, the caveats which are not loading, I think from Wi-Fi issues, but you can test this out in your own instance, uh, as well. Um, we have another fun one that was, uh, what's the correlation between global revenue generated by Unilever Group and Google searches for "lost my wallet"?

133. 24:38

     That's another spurious correlation that we loaded, and it will do the same thing. It will give you the actual answer, uh, and then note the caveats. And I'll note that if you run these same queries in ChatGPT here on the right, if you just ask the question without any files, not doing RAG, it will just start with

134. 24:59

     telling you there's no meaningful correlation. We know that's true, but think about if your own documents don't follow conventional wisdom or have some information that's not out there, it's going to argue with the information in your documents rather than presenting it, even though it may not be, um, may not actually be true in this case.

135. 25:18

     Uh, and then if you do a document upload and use the long context in ChatGPT, you will get that response of the actual correlation, but then it will just say it's a spurious correlation, and it won't really go into why.

136. 25:33

     So it's not really going to, um, you know, a simple system like this is not really going to hew to the facts that you've presented it with. Um,

137. 25:43

     and, uh, for a fun question, we just searched about, uh, global revenue from Unilever Group and Google searches for "lost my wallet." Does this imply that Unilever Group's revenue is derived from lost wallets?

138. 25:58

     And it will answer, "No, that's not true." Um, which we know. And then if we ask another query that relates two different documents that don't have any information shared between them, now we're looking up the correlation between the distance between Neptune and the Sun and global revenue generated by Unilever Group, and it will not answer that question.

139. 26:18

     It will just say it does not have that information. And so, you know, these are all, uh, pretty straightforward questions and answers, but, um, you set up with our default settings, but if you go to your agent and you click on Edit in the admin panel, there's actually a lot that you can change if, you know, for

140. 26:38

     example, that last question I asked, if there was data that supported it and it wasn't found or something, uh, you can check which data stores are linked, so you can link multiple data stores.

141. 26:48

     You can adjust the system prompt, and that is, uh, also something you can do via API that we'll go over briefly later. There are some settings on query understanding.

142. 26:59

     So we disabled multi-turn in our setup just to have everyone, uh, have the same consistent responses as they're trying it to make the workshop easier to follow. Um, you can set up query expansion or decomposition.

143. 27:13

     Uh, you can set some retrieval settings, settings for the re-ranker, settings for the filter, including the prompt, generation settings, and you can even set up a user experience where you have suggested queries for the user.

144. 27:29

     Um, and so, uh, going back to our notebook here,

145. 27:35

     um, what we have next in the notebook is going to be examples of code for some of the components. So these are the individual components that make up the full RAG system.

146. 27:46

     We won't run through this in the workshop. I'm just gonna sh- uh, share what we have here before Raj does a deeper dive into what these components can do.

147. 27:54

     So for each component, we've shared a link to, uh, a full-- a more complete notebook that lets you use more features of that component. We have our benchmarks, uh, that compare it to other solutions.

148. 28:07

     And then we have a little bit of example code that we'll run in the notebook, um, to try out these components alone. And you don't need to set up the-- You-- It's the same API key that we set up before, so everything will just run.

149. 28:17

     I will just say you want to, uh, let your file process before you display it. But in this example, uh, we're just printing the first page of the Attention Is All You Need paper.

150. 28:27

     We have our re-ranker, which is the world's first instruction following re-ranker, and that's also a standalone sub-component that you can use independently of the full platform. We have our generate model that is, uh, according to the effect grounding benchmark, the most grounded language model in the world.

151. 28:44

     That's available as a standalone component, and here's some sample code for that. And then, uh, after Raj's deep dive on the platform and components, we'll go over the LMUnit, the natural language unit testing model available as an endpoint that we will use for our evaluation.

152. 29:03

     Uh, so yeah, back to Raj for the, uh, next portion of the workshop.

153. 29:13

     All right. Yep. All right. I just need this one.

154. 29:21

     All right. If anybody knows me, I love to kinda talk and get into the technical pieces and what's going on a little bit deeper. So this is useful for learning about contextual, but if you're even building your own RAG pipelines as well, all these components come into play, um, to start with.

155. 29:36

     Now, to start with, how many people have had issues with hallucinations in their RAG pipelines? Has this been... A few. Have people not have issues, not had issues? [chuckles]

156. 29:49

     Okay. Yeah. I think everybody, yeah, finds this.

157. 29:52

     Now, when we start thinking about hallucinations, there's a couple of ways we can think about how to prevent them. The first is making sure we retrieve clean information. If we don't do a great job at extraction, if we don't do a good job at retrieval, well, then we're not gonna give good information to that generator model.

158. 30:11

     It makes it-- The generator model will then put its own-- will try to answer it itself, leading to hallucinations. So that's one big piece.

159. 30:20

     The next is that language model itself, and we'll talk about that, is how we can ground that language model so it refers back to the context that it was given rather than substituting its own judgment.

160. 30:32

     And then I wanna talk about having some checks in place because even if you do that, you still want your end users to be able to trust the system.

161. 30:39

     So I'll talk about groundness checks as well as, you've already seen it inside the platform, the bounding boxes for attribution as well.

162. 30:49

     So to start with, when you first build your first RAG project, right, you need to extract from some PDFs. You maybe have used, like, an open source PDF extraction tool.

163. 30:58

     Have folks used, like, open source extraction tools like that? PDF plumber, things like that? Not that many.

164. 31:06

     Have people been happy with them in terms of how they've worked for tables and charts?

165. 31:13

     Yeah. Uh, it's-- The, the document on the left is the easy one, right? That's what you show in the demos. But your folks are gonna show up with complex tables, multimodal, and that's when it gets much trickier because when you get to these types of documents, if you have a parsing error in that table and things get

166. 31:32

     shifted over, right, that's gonna cause problems that you can't fix later on, right? If your v-vision language model hallucinates and adds some other information in there when it's reading that multimodal chart, right, you're stuck with that, right?

167. 31:45

     Like, that's gonna lead to downstream problems. So making sure you have very good extraction is, like, top of the list when you get to complex documents.

168. 31:55

     And so this is, this is how I like to think about kind of what we're doing at Contextual right now. It's an evolving system. We update, like, every couple weeks.

169. 32:04

     The engineers probably don't exactly like how I've diagrammed it like this, but at least for me, the mental model works where, say you have a PDF document. The first thing we're gonna do is add some metadata around it, right?

170. 32:17

     Like, when was it created? What's the file name?

171. 32:21

     Once we have that, then you wanna do a layout analysis. Is this a document that's image only that we need to do an OCR for? Does it have images, maybe technical descriptions, multimodal charts, graphs that we need to add image captioning where we're gonna get a text description of those features?

172. 32:42

     If you have tables, we have a special table extraction mode that focuses on getting high quality results out of tables.

173. 32:51

     Now, beyond that, the structure of the document also carries meaningful information, right? What are the section headers? What are the different subsections in there? Using that information can make your, make your extraction go a lot better.

174. 33:07

     Now, once we have that, we create a nice markdown kind of JSON version of that. I'll show you that in the platform here. But then for RAG use cases, we're gonna take those documents, and we're gonna chunk them down into smaller pieces to work with, right?

175. 33:22

     Because LMs, as much as the long context is growing, you can't take some of these large documents and chunk them in. And especially there's the overall compute cost of trying to use those long contexts fully.

176. 33:33

     So we create chunks. We use some of that metadata that we've created, inject that in, like knowing the hierarchical structure of that. Along the way, we set bounding boxes.

177. 33:43

     This is one reason when we do, uh, image captioning, for example, we don't wanna take the entire whole page and do that. We like to know where the images are, so we can do bounding box so the user knows when we say sales went up by, you know, ten, ten dollars, they know where on the page exactly

178. 34:00

     it was. So within the platform, and I know you can't get to it right now, we have what's-- the components you can see on the left side. Let me see if I can kinda...

179. 34:14

     Right there. All the individual parts of our platform you can work with individually, and we have a playground for that. So this is the parse piece here where you can just take a PDF document, decide how you want the extraction.

180. 34:28

     I just want text or, hey, I want the full works where I want image captioning as well, or I have long tables, I want the table extraction mode. And you can do that right inside the UI, see what the results look like.

181. 34:40

     Now, of course, right, we're developers. We also have an API, Python SDKs, JavaScript SDKs as well, so you can do this programmatically as well.

182. 34:50

     But that's the first step, getting that clean information.

183. 34:55

     Next, and this is again my, my diagram. The engineers would not be happy with this. Um, the query path.

184. 35:02

     So once you have that query, when your user's asking a question, what's the process that it goes through?

185. 35:09

     Uh, first, we have a translation module for folks that-- for when we need to do that. But the first big step here is the query reformulation. And so here, depending on if you're doing multi-turn, we need to take into account the p-previous conversations.

186. 35:26

     Query expansion. Maybe the types of users are doing lots of abbreviations inside your company. So we need to take that query, make it a little richer and fuller before we pass that in to get good results.

187. 35:38

     Or on the other hand, maybe your folks are putting in long, complex queries where what we want to do is take that query, break it up into smaller sub-queries, answer each of those.

188. 35:49

     So this is where you have the flexibility. You can turn these knobs on and off as you want, as Nina was showing you in the edit, to use those.

189. 35:58

     Now once you've gone through that process, that long query might-- that you have might break into three sub-queries. Each of those reformulated queries then passes through your semantic search and lexical search, so hybrid search, kind of classic best practices there.

190. 36:15

     And then we give you the option of using a, what we call a datastore filter, because maybe your query is asking what's the revenue of Apple computer and your datastore has Apple, Microsoft, Tesla, all those documents.

191. 36:28

     Well- If you already know they, they're interested in the Apple company, why not just filter out all the other documents so you don't have to worry about that? That's where that data store filter comes into play.

192. 36:40

     Now, typically for retriever, our, our defaults are something like a hundred chunks that will come back.

193. 36:45

     Retrieval's fast, you can do that. But then to get better accuracy, we like to use a re-ranker. And again, this is best practices around RAG. We've trained our own, the instruction following re-ranker, and that helps you go from that eighty and like the default is fifteen, but again, you can change it for your application and give you,

194. 37:04

     let's say, fifteen high-quality results that you can then pass all the way through to the end. We have another filter stage if you want to, for example, drop out some of those as well.

195. 37:14

     So this is-- all of these pieces are orchestrated with lots of different m- m- machine learning models like this. So this is where it's like a pain to build this for yourself, but we've built this like this.

196. 37:26

     Now, if I had the time, I'd go into all those pieces in depth, but I wanna highlight a couple of things. I like our re-ranker, I think is interesting, where you have the ability to provide it instructions.

197. 37:37

     So a common one might be, for example, if you're asking questions about Apple and you wanna get the most recent documents. So you can give it a prompt, and then when it re-ranks those, it's gonna look at the contents of the chunks and use that prompt to help re-rank it.

198. 37:51

     So lot of possibilities kind of with this, and we're kind of excited to have this piece there.

199. 37:59

     Now, the last step is generation. So we've retrieved back all of the query. Next, we're taking it to our grounded large language model. Again, the notion here is we want a large language model that's been specifically fine-tuned on to, to respect the knowledge that it's given, not give its own knowledge like that.

200. 38:21

     Besides that, we have the two pieces that I've talked about earlier, the groundness and the attributions. You've already seen the attributions, and I'll talk a little bit more kind of about groundness.

201. 38:30

     Now, the grounded language model, since we trained it ourselves, we got to kinda do some different things with it. So one thing we've done is the m-- language model will tell you the difference between facts and commentary.

202. 38:43

     So when you have some text, it will look at that answer and see, hey, this is the important facts in here. The rest is just other commentary. So this is where when you're working with this, if you want, I need a super grounded model, like I just want the facts in my RAG system.

203. 38:59

     I don't want any of the superfluous stuff. Like you can turn this setting on to eliminate the commentary and just focus on the facts, for example. So this is where kind of training our own gives us flexibility to do this kind of thing.

204. 39:15

     The second thing is, is the groundness check we have. So the groundness check works directly in our UI. You can also use this through the API, where for every response that comes back, what we do is we look s- sentence by sentence essentially and see, is the claim that's made it found in the documents that were given?

205. 39:37

     So here, if you look, you'll see the bottom sentence is in yellow.

206. 39:41

     And it's in yellow because the claim it's making is a totally BS claim that it just made up, right? It's not actually in the source documents. I literally prompted it to make up something.

207. 39:53

     And so here, we'll automatically in the UI highlight that's yellow. So if you have users that you're worried about, ugh, like they, they, they keep falling for these hallucinations, this is kind of a helper feature inside there.

208. 40:05

     And the, and the way it works is when we retrieve all the context, and again, you have access to all the chunks, all the things that are retrieved,

209. 40:14

     that answer's gonna be generated, but then we're gonna decompose that answer into specific claims. So we have a, a model that just does this decomposition and looks and sees, is this claim grounded back in that context or not?

210. 40:29

     So we can run this live at time, be able to get those groundness scores, show you that with the response as well.

211. 40:37

     All right. Nina has got more on evaluation like that.

212. 41:01

     So for our last step of setting up our RAG agent, we will get to the evaluation step. So we're gonna use our LM unit model for natural-language unit tests, which may be a little bit different from how you've been doing evaluation of your RAG system before.

213. 41:18

     So we have a fine-tuned model as a judge, and that has state-of-the-art performance on Flask and BigGen Bench. And the way that it works, as we'll get into in the notebook shortly, is that you'll put in, um, the prompt and get the model response as your RAG system would generate.

214. 41:37

     And then we're going to create unit tests for the prompt that will ask specific questions that you want tested about those responses. Then we're going to use the LM unit model to evaluate these unit tests-

215. 41:50

     Every time, victory

216. 41:50

     ... and then we can look at those scores. So we'll be fully in the notebook now, and I have just given up on the Wi-Fi now. I'm just gonna go through what I ran earlier, so we can just look at the results.

217. 42:02

     Uh, if you had internet, I could run this all on your own. So, [clears throat]

218. 42:08

     um, in this section, we have, uh, we have commented out the code that we used to generate the dataset. So that has our set of questions about the dataset, uh, six questions.

219. 42:23

     And then this is the section that, uh-

220. 42:26

     That's some pictures of me when I'm up there. I hate that camera

221. 42:26

     ... will generate those results from those queries. So to save some trees, we just ran this ahead of time and saved it in our GitHub, so you could load it directly, um-

222. 42:38

     And, uh, that's, uh, at this file here, we have this eval input CSV. Um, and then that just has prompts and responses for six queries from that same agent that we set up earlier.

223. 42:53

     And the unit tests we're going to ask, uh, I thought these would be interesting ones for this, uh, for this document set and the queries. Does the response accurately extract specific numerical data from the documents?

224. 43:09

     Um, does the agent properly distinguish between correlation and causation? That could be useful if you have some sort of statistical analysis in your documents. Are multi-document comparisons performed correctly with accurate calculations?

225. 43:24

     Are potential limitations or uncertainties in the data clearly acknowledged? Are quantitative claims properly supported with specific evidence from the source documents? And does the response avoid unnecessary information?

226. 43:40

     We know that, uh, LLMs like to blab on and on in their response, so this is a good unit test, I think, overall. Uh, some of the other ones you can set based on your own documents if you wanna try this out later.

227. 43:53

     And so we just set up our unit tests here. Um, and then

228. 44:00

     the LM unit model will score on a one to five scale, and it's been fine-tuned for this specific task. And so then the next block of code that you would run would be this, uh, response generation.

229. 44:14

     So we put in our query, what was NVIDIA's data center revenue in Q4 fiscal year twenty-five? And then you get your response, and that does have the, uh, the correct amount, the citation, but then it has a lot of other information.

230. 44:32

     And so the unit test we used was, does the response avoid unnecessary information? And the score was two point two out of five. So for example, if we wanted to-- if this was an important criteria for us, then we could later update the system prompt or change some of those other settings to specifically target this, um, the

231. 44:51

     results from this unit test. Of course, we don't wanna operate from an N of one, so, um, we're now testing this batch of six, which of course, again, is a small batch.

232. 45:00

     This was meant to run in, uh, real time if we had internet. [chuckles] Um, and this would just run through all of those six queries and all of those six unit tests to get the results.

233. 45:14

     Then this is the line here, um, run unit tests with progress. This, uh, will give you the results from those unit tests. And so here we can look at those results.

234. 45:25

     We can see the prompt response and, um, and then we can see the scores here for each of our six unit tests, and those are gonna be from one to five.

235. 45:41

     Um, and then we can also then map those long sentences to just a one-word category. So just accuracy, causation, synthesis, limitations, evidence, and relevance. You can set these as you'd like.

236. 45:55

     And we're gonna create polar plots that show us, uh, for example, for our second question, what is the coefficient between Neptune's distance from the Sun and US burglary rates?

237. 46:06

     We can see in the polar plot that, um, the causation was very well addressed by our model. The accuracy and limitations also scored high on the unit test, but the synthesis was not as highly scored, um, and these other factors were not as highly scored.

238. 46:23

     So we can look at all six, and, uh, we can kind of see some of these queries scored really well across all our unit tests. Others may point to areas to improve.

239. 46:34

     We linked earlier to the full LM unit notebook where you can look at these categories and, uh, cluster them. So for a larger scale evaluation data set, you could actually, like, look for, um, for meaningful insight, insights from this unit test.

240. 46:50

     Um, and so, uh, your first homework, if you've made it this far, is to improve the agent based on the unit test results. And so one way that you could do that in the API would be, uh, here to update the system prompt.

241. 47:05

     So I have our original one, and you could, you know, change something here like, "Hey, keep the responses really brief. Only answer, um, uh, the direct question." And then you could run this line of code here to update the agent and, um, and yeah, that's our, that's our end-to-end pipeline.

242. 47:26

     And now for a bonus, Raj is going to show you how you can connect your agent to Claude Desktop via MCP. And for that, we have links to our GitHub here and the YouTube how-to if you wanna try this out yourself.

243. 47:40

     Um...

244. 47:43

     Okay. Yeah.

245. 47:44

     Yeah.

246. 47:51

     I'm dealing with the challenges of the Wi-Fi and working around that, so let's talk about this. Let's see if we can get this running.

247. 48:04

     So if we're lucky for all of you, I spend way too much time making videos, so I have a copy of the video integration that I wanna show. So we built a RAG agent.

248. 48:12

     You can ask questions, right? You wanna be able to use it. One of the most common ways of being able to use it, right, that was introduced, what, like six months ago here, um, at AI Engineer was MCP server.

249. 48:24

     So one of the things I like to do is kind of show people how I can connect the RAG agents that we built at Contextual, and you can use them inside of other MCP clients.

250. 48:34

     And so here's an example of using it inside of Claude Desktop, where now- When it has this particular topic, it's going out to the contextual RAG agent to be able to get the answer to do that.

251. 48:47

     So you can do this working inside of clients like Claude. You can also be able to do this, for example, if you're working with code, you wanna be able to do this in something like Cursor, that works as well.

252. 49:00

     So to do this, if you're like, "Hey, I wanna do this, I wanna be able to use this RAG agent everywhere else," to do that, we have a repo that's out there.

253. 49:09

     So the contextual MCP server, it's the, the link is in the notebook as well. So you can grab that. It's a neat-- It's a very basic MCP server. You could probably build your own, let us know, um, like that.

254. 49:21

     But the workflow is you clone that repo, and then inside that repo, there's a server.py file, and we're just using our contextual, um, our contextual APIs and just pointing to that.

255. 49:34

     So the important thing, of course, is that docstring that you kind of describe what your RAG agent is about because that's what your client, whether it's Claude or Cursor, is gonna be able to use that.

256. 49:43

     So to make this a little bit more--

257. 49:48

     Here's a copy, for example, inside of mine, where

258. 49:54

     you can see in my local environment I have that MCP server locally hosted. This is what my server.py file looks like, where you can see I've got two different

259. 50:06

     ones set up, one for technical queries, one for financial queries, depending on who I [REDACTED:username] to, and it can have the client then pull into that piece there.

260. 50:17

     So that's the, the s- the server that's sitting in the middle. The next piece is configuring it with your client. And every client's a little bit different like that, but you just wanna be able to have that client point to where that server file is.

261. 50:31

     So if we look at, for example, in the case of Claude Desktop, which is not running now, but luckily, we can still do this.

262. 50:40

     And Claude Desktop has a developer piece where you can edit the config for where it should look for MCP servers. And if I click here, Edit Config, I can go see the file that it's there.

263. 50:52

     We can open that up. We're not gonna open it up in Cursor. But you get a sense, that's the config file that's gonna point towards where my server location is.

264. 51:02

     So again, if you're not sure about this, I have all the directions on the GitHub. There's an accompanying video as well. But it just allows you to use that RAG agent now in lots of other ways as well.

265. 51:13

     For example, if you're building your own deep research tool, you want to be able to do that, you can do that.

266. 51:20

     All right. We're gonna open it up for questions here in a couple of minutes. And again, if you're having any issues, raise your hand. We got a couple of people in the back that can help out.

267. 51:30

     So one question often is, is like, "What is this gonna cost me? Like, you give me the sales pitch, what is it gonna cost me?" Well, if you use our individual components, the parse, the re-rank, the generate in LMUnit, we've made that all consumption-based.

268. 51:44

     You pay by the token for those. So the pricing is up on the website. It's pretty straightforward like that. Um, when you signed up today, you started off with a twenty-five dollar credit, so you should be able to go start playing around with it as well.

269. 51:59

     So those are the individual components. We're also making our RAG platform consumption-based pricing as well. So the pricing will be based on the number of documents you ingest and how many queries you do.

270. 52:13

     So based on that, you'll be able to kind of calculate out, know what your workload is like that. Now, some-- We-- I work with some enterprise customers. For some sensitive places, they're like, "Hey, we really need a guarantee on latency or qu-- amount of queries per second like that."

271. 52:28

     This is where we can work with that team, do a provision throughput, so that way you have dedicated kind of hardware for your particular use case like that. But I think one of the great things that's gonna really open it up to developers is the consumption-based, because just like you're doing today, you're gonna be able to sign

272. 52:44

     up, pass documents through that, and just pay for what you use like that.

273. 52:52

     All right. So some final takeaways. I'm hoping that you got out of this that how you can treat RAG just as any other managed service. You don't have to go through, build out all those pipelines yourself like that.

274. 53:03

     You've got the code, you can get started now building this pipeline individually. Or if you just need parts of it, just the components, try them out as well. You, your re-ranker you're not happy with, give it a shot.

275. 53:15

     So go try out the app to do that. All of this stuff is documented over in docs. We have kind of full API docs over there. We also have a bunch of example notebooks showing, for example, integrations with things like Ragas as well, if you wanna use that for evaluation.

276. 53:32

     So those example notebooks will walk you through, "Hey, how do I improve an agent? W- How would I use all of these settings?" If you need other example notebooks, let us know.

277. 53:40

     Nina and I will work on that as well. So finally, fill out our survey, kinda share your feedback as well. We've got some nice kind of merch we can hand out to people who have good questions as well.

278. 53:53

     I think we got a little bit of time for that. How does that sound?

279. 54:02

     I've thrown this RAG platform at you. Are you all ready to sign up? Okay. Yeah.

280. 54:08

     Go a-- Go ahead.

281. 54:10

     When you found people using this platform, there's a whole-- there's an interesting feedback loop where people can set their email with a prompt and then like-

282. 54:19

     Can you use the Q&A mic?

283. 54:21

     Is there a, is there a mic?

284. 54:22

     There's a Q&A mic, or I'll try to repeat back the question to the wider audience. So if you want my interpretation of the question or [laughs]

285. 54:31

     I can't massage it then.

286. 54:33

     Is that on?

287. 54:33

     Yes.

288. 54:34

     Yeah. Um, so there's a really interesting feedback you're defining here around you can- Define evals via prompt. You can see how they're doing. You can let people change the system prompt.

289. 54:46

     Do you find that your users of, like, the platform or the enterprise customers, like, who is it that's doing that? Is it, like, technical people, or is it the people that's using it day to day?

290. 54:55

     So one of the things, by putting it in the UI, what you have is you have some of the business users will try to mess around and play with it, and some pieces like a system prompt, like non-technical people have a sense of using ChatGPT and doing that.

291. 55:08

     But when you get to what's your-- my top K for my retrieval or using query reformulation, those business users are just gonna mess up, and their agents are gonna just kinda not stop-- work well like that.

292. 55:19

     So a lot of the settings there, as you get to the a-advanced hill climbing, because a lot of that stuff comes into, like I've built a good RAG agent, it's hitting eighty percent, but the business wants us to get to ninety percent.

293. 55:32

     Now I have to build the evaluation. Now I have to see where are my errors. Are they on the retrieval side? Are they on the generator side? And this is where having that developer perspective of understanding and being able to do that error analysis is important to figure out which of those settings to do it.

294. 55:45

     Because we give you a lot of settings, [chuckles] but you still have to have some sense of, like, which settings are appropriate to change for what outcomes like that.

295. 55:56

     Um, company I work for is relatively new to AI. We're loving what we're discovering, but, uh, it's also pretty overwhelming.

296. 56:03

     Mm-hmm.

297. 56:03

     Uh, the information you presented, uh, hints at some of the answers to, to our major questions, which are-- We have thousands of PDF documents. We have thousands of e-- uh, you know, Excel documents.

298. 56:17

     We have an MRP system with an SQL back-end. Do we need to be writing custom queries for this environment to consume? Um, I guess, does your environment have a checklist of the approach for different data types and how to package this information for your environment?

299. 56:35

     Yeah. I, I, I can repeat back the question like that. So, uh, so the question is, is, uh, g-- and I'll, and I'll paraphrase here. Getting into it, we have lots of documents.

300. 56:43

     We have Excel documents, PDF documents, structured data. Like, how am I gonna make sense of that and figure out what the best way to kind of use your platform is?

301. 56:51

     Is that fair?

302. 56:52

     Yeah.

303. 56:53

     Yes. And so this is where, as a startup, we have a couple of different levels. We have, we have the developer, go figure it out yourself. We've given you the buttons, the API commands to do that.

304. 57:03

     But this is where we've grown, and we have a dedicated team that we call customer machine learning engineers, that all they do is work with customers, and they do that process of, "Hey, let's walk you through building your evaluation datasets.

305. 57:16

     Let's help you kind of hill climb and improve that." So depending on what you need, we have a team that's just focused on helping customers through that process as well.

306. 57:25

     And in the meantime, Nina and I are trying to document it and make it more available for the rest of the users. But there is that gap of, like, there's a lot of knowledge to effectively use these systems to be able to do that.

307. 57:38

     Is that fair? Okay.

308. 57:39

     And it's a consulting fee or a startup package?

309. 57:42

     Yeah, talk to us. We, we can-- we got it. We'll find a way to take your money. [laughing] [laughing]

310. 57:47

     Maybe a little bit.

311. 57:48

     Uh, go ahead.

312. 57:51

     Hey, Rajiv. Um, um, we've built a few agents, and I'm very interested in exploring the RAG-only part of, like, what you described. Is there a way to integrate these into our agents that we're building in TypeScript, JavaScript?

313. 58:09

     A-absolutely. So each of the-- Okay, yes. I don't have to repeat. So each of the pieces here, we have kind of APIs. We have a j- we have a JavaScript SDK like that.

314. 58:18

     So if you just wanna use the components of this, this is where-- When the company first started about a year ago, if you'd asked to kinda do a [REDACTED:username], we would've sold you.

315. 58:26

     Like, "You have to buy the entire platform. You have to get the end-to-end." But we've realized, like, if we want to appeal to developers, that lots of times you've built stuff out that you just want some components of it.

316. 58:36

     So yes, w-we, we've kind of modularized that out. So if you just wanna use parts of it, you can, and integrate with others like that, so. And in fact, you know, most of our customers don't use, like, the UI that we've showed.

317. 58:47

     Most of them have other UIs that they're integrating their applications with like that, so.

318. 58:54

     Uh, in response, uh, in addition to the costing or pricing, um, how far can the twenty-five dollars go? Like, because we weren't able to test it right now. Like, how many documents can we, you know, experiment on it, you know, so-so-

319. 59:07

     So, uh, uh, the, the question is, like, how far will your twenty-five dollars go? Uh, I, I, I don't know exactly. [chuckles] I get unlimited use of it. Um, but try it out.

320. 59:18

     If, if you run into issues, let's let us know. We can, we can talk to sales. We can figure out something like that if, depending on what you're doing like that.

321. 59:26

     So don't let the twenty-five dollars be an issue. We want the-- We want you to use it. We wanna hear your feedback. If you need help, let-- we can talk to you and figure out something to do like that.

322. 59:34

     So, yes. Do people like the idea of a managed RAG service? Does this feel like something? Okay. Okay. Okay. Yes.

323. 59:44

     Well, to that point, uh, I guess I'm wondering how managed is it? So at my company, we build RAG applications for government and other clients who have really strict data sovereignty, data residency questions, uh, or requirements.

324. 59:55

     Mm-hmm.

325. 59:55

     So are you in either of the big clouds where I can decide to be in, like, a gov cloud or some other sovereign data service? Like, how much control do I have over where stuff lives?

326. 1:00:10

     Yeah. So, so again, we're early-stage startup. We're starting out. Right now, we have our own cloud, which doesn't help you like that. We have partnered with Snowflake, so we are on Snowflake if that can work for you.

327. 1:00:21

     We can also install in VPC, but right now we're limiting it to VPC. We're not doing kind of custom on-premises deployment just because that takes a lot of upfront work to be able to do like that.

328. 1:00:34

     Does that help?

329. 1:00:35

     Uh, yeah. Can you do VPC in, say, like AWS GovCloud or Azure's government solution?

330. 1:00:41

     We have not yet done, like, AWS in kind of GovCloud like that. Like, if you have a strong demand for that, let us know. We can, we can work and try to figure out something like that, but we haven't taken on that yet.

331. 1:00:51

     Okay. I'll come find you. Thanks.

332. 1:00:52

     Yeah. Thanks.

333. 1:00:55

     Hello.

334. 1:00:56

     Hi. [laughs]

335. 1:01:00

     Sorry.

336. 1:01:00

     I think-- Oh, yeah, go ahead. [laughs] I don't know.

337. 1:01:02

     Hi.

338. 1:01:02

     It's all right.

339. 1:01:03

     I was curious, you know, like, how does the performance of your RAG platform vary as the number of documents vary? For example, like, do you have s- uh, like, recommendations of best practices when you're working with millions of documents as compared to hundreds of documents?

340. 1:01:15

     What kind of, you know, configurations and knobs work better? Would be curious to learn.

341. 1:01:19

     Yeah. So I'm gonna give the hand-wavy answer of, we have customers like Qualcomm, for example, that have tens of thousands of docum-documents. Their documents have hundreds of pages in them, and we handle that fine.

342. 1:01:31

     All I know is that I have engineers, like Matthew in the back, that works the platform that makes sure it kind of scales up like that. So I would say go grab him if you have questions about scalability.

343. 1:01:41

     Right now, we've been able to kind of scale with our customers, um, along those lines like that, so...

344. 1:01:49

     Let me get-- take this, and then I'll come back over here.

345. 1:01:51

     Hey there. This is specifically about the LMUnit, um, tool.

346. 1:01:54

     Mm-hmm.

347. 1:01:55

     Um, how deterministic and repeatable are the re-- are the-- is the scoring from that?

348. 1:02:03

     I'm looking to see if they have... I f-- I mean, it's still a large language model, so I think there might be a little bit, but I think it, it's fairly good.

349. 1:02:14

     Is it-

350. 1:02:15

     It's, uh, it's, it's pretty repeatable. There was actually a paper released when this model was released, so there's a lot more details in there too. But, like, there was a lot of analysis done about the correlation with other trusted metrics and, you know, how, you know, running-- Sorry.

351. 1:02:28

     Run it-- running the LMUnit tests repeatedly, making, like... Um, we use kind of a random seed to make sure it's repeatable. So, like, all of our testing has suggested that, uh, it is repeatable.

352. 1:02:42

     Of course, like, you're-- if you're, like, altering the prompt, you're altering kind of the natural-language unit test, that can have, you know, unforeseen impacts on the results. But I think keeping the prompt consistent, it should be pretty consistent with different types of queries.

353. 1:02:58

     Awesome. Thank you.

354. 1:02:59

     Yeah. Check out the paper as, as Matthew said, that it goes into a lot of those pieces like that, so thanks.

355. 1:03:07

     Question here.

356. 1:03:07

     Yeah.

357. 1:03:09

     So is this already integrated with, uh, Microsoft Copilot and Azure? And the second question is, uh, are there any APIs that you guys are exposing it? Meaning if we are developing our own custom code, but we want to, you know, use your RAG, and what's approach?

358. 1:03:28

     So we have the API. So if you're developing your own custom solutions, you can easily integrate kind of what we have to do that. In terms of installing inside of kind of, um, Azure like that, that's one of those custom VPCs that we can support like that.

359. 1:03:43

     In terms of integrations with Copilot, I don't know if we've really done anything. Usually, like, people are like, "I'm sick of Copilot," and that's why they come to our RAG solution like that.

360. 1:03:52

     So again, we have the APIs though, so I don't know what the integrations would require for something like Copilot like that. Mm-hmm.

361. 1:04:00

     Thank you.

362. 1:04:02

     Hi. First of all, great presentation because I have used your product. And the other thing is, like, regarding RAG, I have a question, like, uh, you as a company, what are the challenges that you're facing in the RAG and where do you see RAG would go in future?

363. 1:04:19

     So what are the challenges we're facing in RAG?

364. 1:04:22

     Apart from hallucination, internet search, or things like that. Apart from that, what are the different challenges that you're facing?

365. 1:04:29

     Uh, Matthew, I might have you g- chime in on this too. I think

366. 1:04:35

     one of the big challenges right now is around the extraction, where we've spent a lot of time and energy, and there's still, I think, room for improvement when we're talking about working with complex tables along charts.

367. 1:04:46

     I think there's one there. I think scalability is always a piece like that because everybody wants to ingest more documents in a, in a faster amount of speed like that.

368. 1:04:59

     I think if you talk about doing generated, um-- working with structured data, I think that's a challenge for everybody in the industry when you're trying to do those queries of, like, text-to-SQL type queries like that.

369. 1:05:10

     So tho-those are some of the top ones on my list like that. D- do you have...?

370. 1:05:15

     Yeah, I, I agree. I think document understanding, like making sure that you are correctly understanding long, complex documents with hierarchy and interrelations between sections and really large tables and gnarly figures.

371. 1:05:27

     Like, I think that has been a consistent thing that we've been working on and seeing improvements on, but there's definitely still room to grow there. Um, I think another kind of broader change in the RAG space that our research team is very focused on, and it hasn't directly translated into our product yet, but it's kind of coming

372. 1:05:44

     soon, is, um, moving away from the more, like, static RAG pipelines that Raj, like, beautifully described in his presentation, toward kind of a fully dynamic flow with, you know, model routers, uh, at certain kind of inflection points, deciding which tools to use, how many times to retrieve, you know, w- how to alter the retrieval config in order

373. 1:06:04

     to correctly answer the query. So I, I think those sorts of, like, dynamic workflows will greatly increase what you can answer with any RAG platform. Um, almost like deep research for RAG, uh, is one way to think about it, and I think that's something that our research team is working very hard on and will be, like, coming

374. 1:06:23

     into our platform in the near-ish future.

375. 1:06:27

     Great. Thanks.

376. 1:06:29

     Come back in six months. [laughs]

377. 1:06:32

     Last one.

378. 1:06:33

     Uh, do you have plans for a public-facing, like a web search feature, particularly one that could be configured on a company's domain?

379. 1:06:44

     So are what you-- Are you looking for kind of a general, "I need to be able to search the internet," because, like, there's Tavily, Firecrawl, places like that, or...?

380. 1:06:52

     It's more that, uh, companies that have public-facing websites where it makes sense to, instead of running RAG on

381. 1:06:59

     Your, like, CMS database, you're actually just-

382. 1:07:01

     Yeah

383. 1:07:02

     ... yeah.

384. 1:07:02

     You, you wanna run it on that live. And so, John, I might pull you in here because I think one of the things John's worked with quite a bit.

385. 1:07:08

     Because a lot of what that requires is the integrations to be able to pull from that webs, from those sources as well. And so I know, John, you've done some of those things for customers with Firecrawl and pieces.

386. 1:07:18

     Yeah. Sorry, what was the question for me again?

387. 1:07:20

     About ingesting customer websites-

388. 1:07:22

     Oh

389. 1:07:22

     ... and being able to use that in RAG.

390. 1:07:24

     Oh, yeah. So, um, there's a lot of, like, uh, ways that you can scrape for websites or, like, directly you're like, um, kind of set up, like, an ETL pipeline with, like, pulling the records from the APIs or just pulling the unstructured data from, like, a data cloud, um, or blob storage, and then kind of just setting

391. 1:07:39

     up, like, an ingestion queue, which is what we do on our side for larger customers. Um, or basically any customer that needs, like, a kickstart for lots of data coming in.

392. 1:07:48

     Um, some customers are, like, super happy to write their own scripts or, like, their own kind of functions or daily cron jobs to update, uh, documents. But we're also working on, like, a managed solution on our side so that you can just give us your credentials and then on our front end, and then we'll just start pulling

393. 1:08:02

     and syncing all the data on our back end. But that's, like, something coming on the roadmap in, like, two to three months.

394. 1:08:09

     And so please, if there's pieces like that you see that you want us, let us know because we like to kind of push the product team to kind of make that stuff happen.

395. 1:08:15

     So. Yeah. All right.

396. 1:08:19

     Yeah. Hi. Uh, hi. Uh, how do you deal with, uh, frequently updated content and, uh, uh, document-level permissions if not everybody can see specific documents?

397. 1:08:30

     Y- yes. So those are both tricky things. One of the things we'll do for frequently, um, frequently updated documents is we have a continuous ingestion pipeline. John, if you can talk to in the back, can talk to you more about that piece as well.

398. 1:08:44

     So that's one piece for the frequently updated. For-- The other piece is the harder one, entitlements. How do you deal with all the permissions, right? Like you're indexing HR RAG stuff next to customer support stuff.

399. 1:08:57

     How do you make sure that the person doesn't look up the salaries of anybody else like that? So this is where within our platform, one of the things we're adding is an entitlements layer like that.

400. 1:09:07

     Because as we've talked to lots of customers, they're like, "This is nice when you do it over our customer support. But inside of a real enterprise, we have governance, we have permissions.

401. 1:09:16

     We have to be able to respect that when we do these types of searches." And so we're adding an entitlements layer on top of ours to be able to handle that.

402. 1:09:24

     So yeah, that's an important piece.

403. 1:09:26

     Hi. Uh, so my question is regarding, uh, in last six months, uh, what are some of the major breakthroughs in, uh, RAG area? If you can-

404. 1:09:35

     Ooh, what are the major breakthroughs in the last six months in the RAG area? Um,

405. 1:09:40

     uh, so I think, uh, the re-rankers continue to get better. I think that's, that's an easy place, um, where we've seen that. I mean, I think part of it is just every piece of that has steadily been getting better.

406. 1:09:52

     I think one of the biggest changes you've seen in the last six months, year, is the rise of the vision language models and how strong they are at being able to handle kind of images like that.

407. 1:10:01

     That's, that's off the top of my head. I see Matthew's busy talking to somebody else, otherwise I'd have him weigh in as well. Does that help? Yes.

408. 1:10:10

     Um, does the contextual parser module can replace like DocLink or GSP invoice parsers and things like this? And if, if it does, does it have the ability to parse QR codes and barcodes and extract their metadata to kind of read that data as well?

409. 1:10:29

     Yes. So I think one of the things is there's a lot of different types of documents out there in terms of document types, and there's a lot of different solutions.

410. 1:10:36

     So I don't-- We're not gonna be able to handle every type of thing perfectly like that, that I think there's gonna be lots of pieces. But this is where we've made those APIs, so you can try that out, um, o- on those piece.

411. 1:10:46

     Like we haven't specifically focused on, for example, that type of document. We do have the image captioning, so maybe it would pick them up, maybe it wouldn't like that.

412. 1:10:54

     So but the idea here is to go in that same space where there's a lot of these other companies doing this parsing thing and making sure that we have a module that you can just stick in and replace and kind of use with ours like that.

413. 1:11:06

     So. Yes.

414. 1:11:09

     How do you deal with, uh, domain-specific language?

415. 1:11:13

     That's a good question. How do you deal with domain-specific language? And, and this can get a little tricky. And we've seen it, for example, when we're working with technical companies that have very specific words that they use inside there.

416. 1:11:25

     Where on one hand you can do things like changing system prompts a little bit, um, for the retriever part. But sometimes what we've done with some of those customers is fine-tune that grounded language model that's closer to their vocabulary and how they speak, and that's been one technique we've used.

417. 1:11:43

     Now, and this is where at one point in the platform we had that available to end developers, but fine-tuning models can get a little complex for using for RAG.

418. 1:11:51

     So we've kind of put that away now, and that's more for kind of our service-oriented customers that work with our customer machine learning engineers. But that's one thing we've done to, if that knowledge is out there and you need to get that into your grounded language model, is to do a kind of a fine-tuning process step.

419. 1:12:07

     Go ahead. Oh, I don't know where the mic is. We're making them run. [laughs]

420. 1:12:13

     Getting their steps in today.

421. 1:12:16

     Hey. Um, so how about HIPAA regulations like, uh, PHI data-

422. 1:12:22

     Yeah

423. 1:12:23

     ... like this kind of stuff? Like how you deal with that? Should we hash it before go to the platform or the LLM can, can understand and, and deal with that?

424. 1:12:34

     So there's two aspects to that. One is we just got HIPAA certified, and people are gonna think I'm paying the audience members out there, [laughing] um, like that. The second is, is well, what do you do in extraction?

425. 1:12:44

     Like maybe I have PHI data, I want you to automatically mask or filter that. So this is one thing that we've been talking with the product team about whether to include that capability for during that parsing piece, um, to automatically kind of flag and identify that, that behavior.

426. 1:12:59

     Right now, we don't kind of have that in the product. But again, like if that's something you need or want, let us know. It's not that hard to kind of build on.

427. 1:13:06

     But you can also do that as a supplement to the parsing and have a second job that runs and looks for that PII information. Does that help? Good

428. 1:13:15

     Uh, I know garbage in, garbage out, but if we have a document with, say, two different numbers on a particular, like, statistic or fact, how does your RAG deal with that?

429. 1:13:26

     Or how would it answer a question?

430. 1:13:28

     Yeah. So what, what happens in that case is often the retrieval's gonna bring both of those pieces of information over to the grounded language model, and it's gonna be up to the language model to use its reasoning ability to try to r-reason out the two differences.

431. 1:13:44

     So sometimes if one is, for example, if it's obvious that one is n-not, like, uh, let's, let's, let's say it's the size of a mosquito and one of the answers is 100 feet and the other is, right, like, three millimeters, the ra- the language model's gonna know, like, one of these is junk and, and to ignore it.

432. 1:14:00

     But if they're both close and reasonable, yes, that, and that can get you in trouble, um, with, with those pieces like that. So I assume this is a pain that you're having now. [laughs]

433. 1:14:11

     Yeah. We have lots of, like, not updated Excel sheets, and so we're just trying to, we're just trying to see how we can organize that. Um, because a lot of my job right now is just telling people information that should be in a document, but it's not updated. [laughs]

434. 1:14:27

     So, so one thing that can help with that is taking advantage of metadata and using as much rich information about those documents, so when at retrieval time, it can help it kind of figure out, like,, "Hey, these are the two differences, and this is why I should prioritize this answer over another answer."

435. 1:14:43

     Like, maybe this is the most recent or this was written by the authoritative person, like that. Like, that's, like, one, one thing that we kind of recommend for those use cases like that.

436. 1:14:52

     Thank you.

437. 1:14:53

     So. All right. Okay. Well, thank you all for staying. I've loved the kind of the questions like this. Please use the application. Give us feedback on how you're finding that as well.

438. 1:15:04

     Good, bad, negative, ha-happy to take that all like that. So thank you again for kinda spending your morning with us like that. Anything else we have, Nina? Or are we good?

439. 1:15:13

     Oh, you've got the mic off. [laughs] Oh.

440. 1:15:15

     I think I still have my mic.

441. 1:15:16

     Oh.

442. 1:15:17

     Uh, yeah, I've been going around distributing swag to folks that have asked questions. Sorry if I missed some of you that were farther away. I have one more pair of socks.

443. 1:15:25

     Anyone wants them? [laughs]

444. 1:15:27

     The guy in the back wants the socks. [laughs] [laughs]

445. 1:15:30

     All right.

446. 1:15:31

     Yeah.

447. 1:15:33

     All right. Thank you all. We'll be hanging around for a little while if you have that, so. [outro music]