VoiceVision RAG - Integrating Visual Document Intelligence with Voice Response

Talk transcript

1. 0:00

   [upbeat music] All right.

2. 0:16

   So, we're almost on time. Uh, firstly, thank you so much for your time, uh, for joining us. And, uh, what we are gonna do is for next an hour or so is, uh, we'll try to explore something around, which is-- which I found, uh, pretty interesting when I started working on this.

3. 0:37

   Uh, uh, and I'll tell you some background about that, how I end up into this, uh, on vision-based retrieval. Uh, but the idea of, uh, that I had was just to share a few of my learning on this particular approach of retrieval.

4. 0:53

   And there are a bunch of things that we have here. Uh, I'm going to share one of the latest research paper around retrieval, which is a, a vision-based retrieval.

5. 1:04

   And also, uh, I just thought to wrap this around with an agent. Uh, without agent, we cannot talk about anything these days, so, uh, [chuckles] right? So, uh, it's funny, uh, uh, without agent I had this, but then, uh, the organizer said that, you know, "We need to have some agent."

6. 1:21

   I'm, "Okay, that's not a big deal." [laughing] Right. So yeah. So...

7. 1:27

   All right. So, um, we'll focus mostly on the, uh,

8. 1:35

   science side of this, like how that, uh, uh, vision-based retrieval works, and then we will switch gears and wrap it around with an agent. I mean, that's a very simple task.

9. 1:45

   And, uh, I'm gonna use one of the open source, uh, uh, framework that, uh, uh, we launched recently, I think it was two weeks back, called Strands Agent, uh, which is kind of a, a framework, lightweight framework, uh, to build agentic application.

10. 2:00

    Uh, I'll talk about that little later. And I have a session on that tomo-- uh, day after tomorrow.

11. 2:05

    Um, but that's the premise. And, uh, before we get started, uh, how many of you are, are from, uh,

12. 2:16

    science side of things? Like who-- how many of you have worked on transformers? Uh, okay, perfect. How many of you have worked on RAG in general?

13. 2:26

    Fantastic. Okay. And, uh, how many of you have worked on AWS?

14. 2:33

    Okay, great. So there's nothing about AWS here. Okay. So, [laughing] uh, so the last question was sponsored by my manager. Okay. So- [laughing]

15. 2:44

    ... what, so what we are gonna do is, um,

16. 2:47

    uh, we are gonna sh- uh, share one, uh,

17. 2:51

    notebook. You can just, uh, clone that repository and, uh, there is lot more, uh, there inside that. Uh, but we are just going to use one part of that, uh, repository.

18. 3:04

    Okay? And I'm going to share a few of the, I think, I think some $25 credit code, uh, which I was given. So I think, uh, you may like to use that.

19. 3:14

    So, uh, let's get this logistics, uh, sorted, uh, first.

20. 3:21

    Okay. So first thing first, uh, can we switch the screen, please?

21. 3:31

    Yeah. Uh, can you just, uh, uh, take a moment and see that if the URL is working?

22. 3:39

    Uh, you may... If you are on laptop, you may like to, uh,

23. 3:46

    open the URL or you can just take an image, uh, on your cell. You can have a look later on.

24. 3:54

    Is it working?

25. 3:55

    Yeah.

26. 3:55

    Okay, perfect. Okay. So now this is something. Uh, you can take an image now or you can, uh, do that survey later on. I mean,

27. 4:10

    I don't like this, but again, this was given by my manager, so... [chuckles] So, so this is just-- It might ask you a few questions. I have no idea what questions they will ask, but, uh, you will get some $25 credit.

28. 4:22

    And, uh, if you don't wanna do it, don't do it. I'll give you $25 credit, so I have it. So it's... [laughing]

29. 4:30

    Okay. So-

30. 4:33

    I have it.

31. 4:33

    Yeah. And, uh, I don't know why this slide is next, but... So this is my li-- Oh, I, I, I actually forgot to introduce myself. So I work with AWS, uh, as a principal, uh, machine learning advocate.

32. 4:47

    I'm with this company for last six months. I focus mostly on, um, natural language and, uh, RAG and fine-tuning. And if you have any questions around the talk that we are going to discuss, uh, or anything around, uh, machine learning or generative AI, feel free to, uh, ping maybe.

33. 5:05

    It's not just about this session, but, uh, my takeaway at, uh, whenever I go and speak at any conference at this scale is, uh, just to make a, a few, few connections with whom I can work with, uh, you know, after this conference.

34. 5:21

    Uh, because as long as learning is concerned, we can learn everything at home, right? And so you don't have to come to a conference. Uh, so feel free to, uh, connect.

35. 5:32

    So with that, I will just switch to, uh, the GitHub repository. Okay? So, and I'll just, uh, uh, walk you through the notebook. So the-- my idea is not to have any presentation because, uh, first, uh, I'm lazy, and second, it's little complicated.

36. 5:52

    I thought that taking images and embed it in the notebook is, uh, much easier. So, uh, you will find this, uh, GitHub repository. Uh, and in that there are, uh, uh, um, many things there, but what we are gonna focus on is if you come to this, uh, section eight and come to this first one, Agentic Voice-based

37. 6:15

    RAG. Uh, so I just added that agent thing yesterday, so that's why. So I had no idea of that. So, uh, so what we are gonna do is, uh, these two notebooks are exactly same.

38. 6:29

    One is without output, one is with output. I find it, uh, useful to have both copies because if you are doing it for the first time, um, you may like to start with this.

39. 6:39

    Don't see the output and run through, and at the same time, if you want to see what it is, you know, what is the expected output and all that, you can go into this.

40. 6:47

    All right. So for the purpose of today's, uh, uh, workshop, I will start with an introduction, and then I will, um, come here. Okay? And if you feel that, uh, this is not something that you are interested in, or this is not something that, uh, you are looking for, you know, feel free to, uh, you know, uh,

41. 7:10

    go to some other place because I don't want to waste your time. Uh, uh, but I want to make sure that if you are here for next one hour, you learn something new, uh, with respect to what you already know at this point in time.

42. 7:23

    Okay? And if you have any questions, uh, feel free to ask. And so that's, that's the other thing. Okay, let me just expand this. It's little too big, uh, here.

43. 7:33

    So, um, okay. So I noted that most of you are aware of RAG, uh, but we are gonna talk about, uh, multimodal RAG for a moment, uh, just to set the premise, and then we will, uh, get into the vision-based retrieval.

44. 7:52

    Okay. So if you think about, uh, multimodal RAG, uh,

45. 8:02

    what we essentially do is-- And this is by no means is the only architecture. This is just one of the architecture. There are many different ways that you can do multimodal RAG.

46. 8:12

    But in general, this is what, uh, we have been doing and still today we do. You take a data, and that data will contain images, text, and, uh, tables.

47. 8:23

    The first thing that we do is we use some framework or... Is that so bad or?

48. 8:31

    Yeah. [laughing]

49. 8:32

    Okay. Thank you. So, so we can use any framework of our choice, or you can write your own custom script, or you can use any managed service like extract OCR based technique.

50. 8:43

    The idea is you extract the images, tables, and, uh, uh, text out, you know, separately. You can have some metadata, uh, to make an, uh, hash, uh, which tells you that this image is coming from which page and all that.

51. 8:58

    But essentially you divide all these three separately. Then you can use one multimodal embedding model, and this multimodal embedding model can take any of these three entities because it's multimodal.

52. 9:10

    It can take any of these three. And when I say multimodal, you can think of it like input can be multimodal. Okay? And then it will generate some vectors.

53. 9:19

    For images it will generate some vectors, tables, text and all that. Then you go to the database, any vector database, and store all these embeddings. So what you are essentially storing here are the actual embeddings of text, tables and images.

54. 9:33

    And then comes the retrieval part. When you ask a question, any raw question like raw text, it goes through the same embedding model. Then it is first searched here.

55. 9:44

    It will get some relevant chunk, which could be again, image, text or table. And then you take those chunks along with your text and use a multimodal LLM. Why multimodal?

56. 9:55

    Because your relevant chunk can be images, text or table, right? And then you get an answer. So this is one approach.

57. 10:03

    The second approach is you do the same thing. Like this part is common. After that, you use a model which will just generate a summary of all this separately.

58. 10:16

    So it will use, uh, you can think of it like a summary of an image is nothing but, uh, image captioning, right? It will generate an summary of this image, summary of the table, summary of the text.

59. 10:26

    Now all you have is the summary. That means it's all text. So now you can use any text-based embedding model to generate the embedding of this summary. And then you store these embeddings here.

60. 10:39

    So what you are storing here are only the embeddings of the summary, not the actual data. Right? And then when the question comes, now we are talking about the option number two.

61. 10:52

    When the question comes, we do a semantic search with the database, and what we get as a chunk are some summary. Now that summary could be a summary of an image, table or text.

62. 11:04

    We don't know. Whatever it is. But whatever we get, both of them are of text format. So that's why we can use a general text-based LLM to generate the output.

63. 11:15

    Okay? So that's, uh, option number two. The option number three is exactly same as option number two,

64. 11:22

    uh, with a slight change here. When you store the summary, you also-- Think of it like this. You have a hash here, let's say a dictionary, which says that, um,

65. 11:34

    this image number one, uh, summary is this. Image number two, summary is this. Table number one, summary is this. You create a hash file or hash, uh, any data structure of your choice so that you can, you know, come back later on, uh, from a certain summary, and you can figure out this is summary of what entity.

66. 11:54

    Okay? But you store only the summary, just like before. But the difference here with respect to option number two is when you ask a question- You get some relevant chunk, which is a summary.

67. 12:05

    Then you go back to that hash and find out the actual data, not the summary, the actual data which is mapped against those summary. And then you take those actual data and then pass it on here.

68. 12:18

    So what you're doing is, summary you are using here just to reduce the search space, just for semantic search. Once you get that, uh, relevant chunks, you don't care about that summary.

69. 12:28

    You just take the original data from that hash, and then you take those relevant chunks and your question. And since relevant chunk can be, again, text, image or, um, a table, you need a multimodal LLM, okay?

70. 12:43

    And you generate the answer. Are you with-- Oh, yeah.

71. 12:47

    Um, let's say you have a table, but the table is an image.

72. 12:50

    Mm-hmm.

73. 12:51

    Which one would you prefer to use in that case?

74. 12:54

    Yeah. So that's a good question. So the question is, if you have a table, which is an image, right? So now it's all-- Whatever you are saying is all at this level, so you don't-- You-- It's all up to you how you segregate these three entities.

75. 13:10

    So let's say you use an OCR-based, um, uh, uh, technique, and let's say it, it's identified a table as an image. So it will be treated as an image because the model, till this point, the model has no idea from where these three are coming.

76. 13:26

    Because these are like prerequisite for this particular pipeline. Okay? So are you with me with all these three approach? Yes? Yeah.

77. 13:35

    Uh, what is the ranks mean on, like, the left side?

78. 13:38

    This one?

79. 13:39

    Uh, no, like the rank symbols on the left side.

80. 13:42

    Oh, okay. So these are nothing but uh, uh, models, basically.

81. 13:46

    Oh.

82. 13:46

    It, it doesn't resonate, it's what I understand now, but-

83. 13:49

    So it's not a language model?

84. 13:50

    Yeah, yeah, yeah. Right. That's correct. Yes. So it could be-- This is any-- So here, we have a multimodal embedding model, right? Here, it's actually, first, you need a model which will generate the summary.

85. 14:02

    Then you can think of it like another model, which will generate the embeddings. So, uh, it's just, uh, one icon, but think of it like there are two things happening i-in sequence.

86. 14:12

    Okay? Are you with me so far? Yes? Okay. So do you see any problem in this, uh, when you have a multimodal data? There's no problem as such, but, uh, there are few scenarios when this may not work.

87. 14:25

    Okay? So scenarios like, like, well, uh, what you mentioned. Uh, there are few documents or data which are-- we have seen where the PDF was created using images.

88. 14:40

    So basically you can think of like this. Um, let's say, uh, uh,

89. 14:47

    toll that we, uh, cross in a highway. All are images, right? They just take the images of our, um, number plate and all that. Uh, similarly, you can think of, uh, any government organization where forms are just, uh, they, they just keep on, uh, taking images.

90. 15:04

    And later on, all those images are converted into a PDF. So in that case, not always, uh, these techniques of extracting images, tables, and text works nicely. It's not like, uh, it never works.

91. 15:20

    It all about how your data behaves with your technique that you are implementing. Okay? So now the next technique that we are going to, uh, discuss today is using a vision-based, uh, retrieval model, and we will see that why we are using this.

92. 15:34

    But the premise is this: if you use-- if, if, if your data-- with your data, any of these, uh, three options works, you just go with this. What we are going to discuss in next one hour, you-- i-it's not relevant for you.

93. 15:47

    But this, uh, you know, what we are going to discuss is an option number four, which is, uh, a smarter technique, which is based on a vision-based model to, uh, perform the retrieval.

94. 15:59

    You don't have to extract all these three entity in the first place. Because think of it like this. The moment you have your data and you first-- in the first place, you segregate these three things, it's just like you have a family.

95. 16:13

    You just, uh, you know, let your kid go somewhere, you go somewhere, and, you know, your partner goes somewhere else. It's a good thing, but, uh, you know, uh, i-if all of them goes, uh, you know, separate and you expect somebody else to identify that they all are part of one family, it's a, it's a task, right,

96. 16:31

    so for that external person. Uh, so that is what we are going to solve. That can we, can we come up with a technique where we don't do all this?

97. 16:43

    Okay? So before we go, uh, I think, uh, you have a question. Yeah.

98. 16:46

    Yeah, I think you kind of answered my question-

99. 16:48

    Yeah

100. 16:48

     ... because you were explaining the case about, uh, scanning all the PDF-

101. 16:51

     Mm-hmm

102. 16:53

     ... and it wouldn't quite work, and I was a little bit confused as to why these approaches weren't working.

103. 16:59

     Yeah.

104. 16:59

     But then I think you are going towards the notion that we need to establish relationships between-

105. 17:03

     Exactly

106. 17:04

     ... things on the, uh, PDF.

107. 17:05

     Absolutely. Yeah. So I think, uh, I'll give you one more, uh, one more, uh, example. Uh, if you go to IKEA, you buy something from IKEA. If you have seen the IKEA, uh, inst-- you know, instructions, uh, you know, we don't-- I personally never look into the instruction.

108. 17:22

     But while I was reading the research paper, they said that, refer to that. Uh, uh, because we generally go to YouTube and search what are the instruction steps and all that, right?

109. 17:31

     But if you look at the IKEA instruction set, they just have emoji kind of a human, uh, and they are just assembling something. There is no text there. There's nothing there.

110. 17:41

     So unless and until you have a visual understanding of, uh, what it is, you will not have any idea what they're talking about. Okay? So there are some datasets, and I will show you a few of the datasets, uh, where the, uh, there are some text embedded within the image, and there are just an image, they don't

111. 17:58

     have any text. So you need some model or some, uh, technique which can help us to understand, uh, what is the semantics of the data. Okay? So let's see how, uh, we are going to solve this.

112. 18:13

     So this is, uh, again, the text might be small. Uh, you can just leave that, okay? You can open it in your laptop, uh, or I'll try to explain it as much as I can.

113. 18:25

     So this is the traditional technique that we discussed, right? You first place, you, uh, divide all these three entities, uh, uh, separately. Uh, but this is not very helpful because if you look, uh-- you know, think about it.

114. 18:37

     Let's say you were given a book and, uh, you were asked to

115. 18:43

     answer a particular question. Let's say I, I give you this book. By the way, this is a fantastic book from Simon. Have you heard of this book? Yeah. If, if you are getting started with machine learn-- uh, deep learning, uh, you know, you can, uh, read this book.

116. 18:56

     This is a really fantastic book. It's a recently, uh, published book, and Professor Simon is very reachable. So it's, it's, uh, it's a fantastic book. So let's say if I give you this book, and if I ask you some question, and let's say you are not aware of this particular topic, you will not, uh, go and scan

117. 19:15

     the entire book. What you will do is, you will try to first find the structure of the book. Maybe you will find the index. Where is the index? Where is the appendix?

118. 19:23

     And all that. And then you'll try to, uh, figure out which chapter this book might-- uh, this question, uh, can be answered from. And then you will go to that specific chapter and then read through those chapters, right?

119. 19:36

     So that is what a human will do. And that is exactly the philosophy of, uh, ColPali. Okay? So

120. 19:47

     when you get a question, what you do is first, you will first scan through the appendix and all that, and then you will figure out where exactly, uh, uh, the portion where your question can be answered.

121. 20:01

     And then you will accumulate all those relevant chunks or relevant information, and then, uh, finally, you will come up with a response, right? So this is where, uh-- Or rather, this was the motivation of this vision-based retrieval model called ColPali.

122. 20:19

     Have you heard of this model, ColPali? Yeah. Okay. Yeah, few of you. So ColPali was introduced, I think, in, uh, July 2024, just, uh, less than a year back.

123. 20:31

     Uh, and the motivation is, we will treat every page as an image. So assume that you have a PDF document of, let's say, hundred pages. Your data set is not one PDF, but hundred images.

124. 20:47

     Okay? There is no concept of retrieving, uh, images, text, and tables from there. So how it works.

125. 20:54

     So it first creates patches of every page. So now let's consider one page. One page is nothing but one image. And

126. 21:08

     the same will apply on all the pages that you have in your document. The first thing that it will do is, it will create some patches. Uh, in the paper, I think it was, um, the model was trained with, uh, thirty-two patches, like thirty-two cross thirty-two.

127. 21:25

     Here, how many patches we have? We have one, two, three, four, five, uh, fifteen patches. Right?

128. 21:33

     Now, what we do is, after that, once you have those patches, you will use this ColPali model, embedding model, and it will generate

129. 21:43

     one vector per patch. Okay? So in this document, how many vectors we will have?

130. 21:52

     Fifteen. Fifteen. So now if I-- if my document is having ten pages, how many vectors I will have in total?

131. 21:59

     One fifty. One fifty. Okay? So now what we are gonna do is, we are going to see this middle part, how it generates the embedding, and then at later-- uh, and in last section, we will see that how it does the retrieval, and then we will go to the code.

132. 22:12

     Okay? Okay. So before we get into that, um, uh, this embedding process, let's take a, a detour of, uh, vision-based language model. Uh, have you worked on a vision-based language model, any of you?

133. 22:26

     Okay. Okay. Few of you. So ultimately, if you think about it, um, we had, uh,

134. 22:35

     language models, uh, I'm not talking about, uh, uh, large language model, but text-based, uh, models, uh, since we had transformer-based architecture, right? And then at that time, we also had, uh, models which can work pretty well with images, which are based on, uh, CNNs.

135. 22:57

     Now, what researchers thought was, uh, now that we have a language model, why can't we just, uh, make use of that to work with, uh, vision? It could be images or videos.

136. 23:08

     Videos are nothing but, uh, images with, but with a timestamp, right? Another dimension you can think of it. Uh, and then

137. 23:18

     what people did was they took some vision-based model, and then they took some, uh, text-based model. They both are separate, right? At, at this point, what we're talking about is, uh,

138. 23:31

     before the training, right? Like, the- these two models are completely separate. They all have, uh, you know, they, they, they are in different space, basically, right? And the idea is,

139. 23:42

     at the end of the day, come up with a model where if you send an image of a dog, the vector that you will get, and if you send a text about dog, the vector that you will get at the end, those two vectors will be very close to each other.

140. 23:59

     Initially, it will not be close because when, when I say a dog is sitting on a field, and if I use any text-based model, it will generate a vector.

141. 24:08

     And for the sake of simpucity-- simplicity, let's assume the vector dimension is ten. So it will generate a array of ten numbers.

142. 24:17

     Similarly, when you pass an image of a dog, it will generate an im-- a vec-- final vector with ten numbers. Let's say the embedding vector size is ten. Now, those ten numbers and these ten numbers for the text, they will be anywhere in the space because they don't have any correlation be-at the...

143. 24:34

     before, uh, the training. Now, what happens is,

144. 24:39

     at the time of training, we take a lot of samples, positive samples, where the text is there, uh, text is there, which replicates the image, and there are a lot of, uh, negative samples where image is there, but text is something random.

145. 24:53

     And the idea is the loss function that we use is if they are similar,

146. 24:59

     we want to make sure that the loss is, uh, uh, less. But if they are orthogonal or very separate, we will say that, okay, the loss is high. And during this loss, uh, you know, this training process, we kind of optimize, and at the end, we, we see that when you send an image or a text, the

147. 25:19

     embedding that we get at the end are very close to each other. Okay? So we are not going to deep dive into, uh, a vision-based model, but there is something called, uh, contrastive learning, where if you send an image and a relevant, uh, positive tag, and if the vectors are very, uh, uh, very sparse, like, very, uh,

148. 25:43

     uh, uh, v-very much apart from each other, then the loss will be very high because we want these two vectors to be close, right? So that way, uh, during the ba-- uh, back, uh, uh, backpropagation, we update the weights accordingly.

149. 25:56

     So this is one of the reason, if you think about it, uh, you might have seen in language models or when you use any, uh, let's say, any, um, foundational model, they say that always your prompt should be, uh, about what you want, not about what you don't want.

150. 26:13

     Have you seen this if you are into prompt engineering? Why they say this? Just think about it. Let's say, if I say, uh, uh... Okay. Le-let, let, let, let me give you an, uh, analogy, right?

151. 26:28

     If you are going for a dinner with your wife, and if you ask your wife, "What would you like to have?" She will say that, "I, I, I..." Let's say, uh, "I don't like this.

152. 26:38

     I don't like that." But that was not my question. My question was what you want. That is always difficult to, uh, answer, right? People will say that, um, uh, "Okay, would you like to have this?"

153. 26:50

     "No, I don't like this." But when you ask them, "Okay, you tell me what you like", it's very hard. So that's why when you give a prompt that, um, I want a dog sitting on this chair, it's a very nice prompt.

154. 27:02

     But if you say that dog should not sit on the floor, it can generate any image because you are not saying that it should sit on the chair. It might be sitting on a desk or somewhere else, right?

155. 27:15

     So that is the reason we always say that the prompt should be, uh, very much positive, what you want, not the negative, because, uh, you know, dataset doesn't have that much, uh, of negative samples.

156. 27:27

     Now let's come back to ColPali, how it works. So you give an image. So now this image is, um, you can think of it like, uh, one of the patch.

157. 27:38

     Okay? It goes through, uh, the, uh, vision-based encoder, and it will generate an, uh, embedding. And then we have a linear projection. And the reason that we have the linear projection is because at the end of the day, uh, when you ask a question, that will also be generating some vector.

158. 27:57

     We want to make sure that these vectors are comparable to each other. They are of same size, and that's why we have added a new projection layer. You can simply, uh, think of it li-- as a fully connected layer.

159. 28:08

     And ultimately, you will have a standard transformer, and then you will get the output token. Okay? So

160. 28:17

     now if you think about... Let me just scroll down. Yeah. So if you think about ColPali, when you give an image,

161. 28:33

     it will have, let's say, in this case, there are fifteen, uh, batches. Just think of this patch, okay? This patch will go through this. It will generate an, uh, uh, vector, and this will be the final representation of the first batch.

162. 28:49

     Similarly, when you give the whole image, you will not give the, uh, you know, uh, single, uh, patch, uh, in the batch. You will give, give one full image or let's say page number one of the document.

163. 29:01

     This model will do all that patching, and it will finally generate one embedding vector. Now, at the time of...

164. 29:11

     A-and if you, if you see here, in this case, this is grayed out because now we are talking about after training. Once that model is trained, after training, you will create the embeddings of your document.

165. 29:24

     So while you are creating the embeddings, there is no question here, right? So it will just use this path. Now, once those embeddings are done, like, once you get all the final vectors for your entire document,

166. 29:39

     in the query time, you will just use your text-based query. So ColPali doesn't say that, uh, you can query with your-- as an image. Like in ChatGPT or any g- uh, GPT-based model, you just upload an image.

167. 29:53

     We are so lazy, we don't even ask the question these days, right? We just upload the image and, you know, model just generates something. So here, the question should be always in text.

168. 30:01

     That's the prerequisite for this model. And then this goes through the same, uh, model, and then it finally gives you a response. Now, this response, this vector, you will do a semantic search with the vectors that you have stored in your vector database using those, uh, image batches.

169. 30:22

     With me so far? Yes? Okay. So, if you think about it, uh,

170. 30:29

     both for query as well as, uh, your embedding, there is a certain amount of, uh, uh, uh, pre-processing that is needed because, uh, your images can be of different size, right?

171. 30:41

     So you, let's say you have an, uh, a PDF document, um, and, uh, uh,

172. 30:47

     the tool that you use to convert that into an image, uh, it created an image of eight hundred by eight hundred. But let's say somebody else have used another technique and the image was of fifty cross fifty.

173. 30:58

     So we need to make sure that the images are of standard size, right? So that's why when we look into the code next, you will see that always before it actually generates the embedding, there is a pre-processing, uh, that we do.

174. 31:11

     Okay? So let's, uh, go to the code and see that. But before that, let's, uh, let's share-- I mean, let's talk about how it generates, uh, the similar chunks.

175. 31:25

     So this is the most important part of ColPali. Okay? Now, imagine that your page... Now just consider page number one of your document. And the page,

176. 31:38

     and the patch size that you use is, let's say two cross two. That is total four patches. Okay? Now, let's say this is page number one, and this is the embedding of your fi- fi-- uh, first patch.

177. 31:53

     This is the embedding of the second patch, this is the embedding of the third patch, this is the embedding of the fourth patch. Okay?

178. 32:00

     And you ask some question. Let's say, uh, "What is AI?" Just for the sake of simplicity. What is AI? And you have used to, uh, it went through the tokenizer, and it generated three embedding vectors, right?

179. 32:15

     Three tokens, basically. So now what we do is we do a dot product between each vector and each vector of all the patches. Okay?

180. 32:28

     And then for every row, we try to find which is the maximum number. What this number signifies, this eight nine signifies, point eight nine signifies that the first part of your question has the maximum similarity with the second patch of the image,

181. 32:49

     right? Similarly, if this is ninety-seven, that means the second part of your question has the maximum similarity with the third patch of your image.

182. 33:01

     Right? And at the end, what we do is we just take the addition-- Uh, we, we just take a sum, the maximum numbers of each rows. And if, let's say, that is two point five eight, that means

183. 33:14

     this query has a score of two point five eight for page number one.

184. 33:20

     Similarly, we will do it for all the pages. And then in RAG, what we do at the end when we do a semantic search, we say top five chunks or top ten chunks.

185. 33:29

     So in this case, chunk is nothing but pages. So if I say top five, then in that case it will show us the top five pages based on this score.

186. 33:39

     You getting it? So this is the most important thing. So this is called late interaction. Have you heard of late interaction embeddings, all that? And the reason that we say late interaction is because these token embeddings are already stored.

187. 33:54

     We have already done that. It-it's there in your vector database. All you have to do is, we need to just do the dot product and then use this matrix to generate the top five or top three, uh, pages.

188. 34:06

     Okay? Uh, with me so far? Yes? Okay. Now, this functionality is not supported in all the da- all the vector databases. We are gonna use one of the vector databa-database called Qdrant.

189. 34:19

     Have you heard of that? But there are a few other databases. I have not done enough research which are the databases that it supports. Uh, but, uh, this, uh, max sim calculation is not supported by all the database.

190. 34:31

     Okay? There are some open source contribution that we have for few of the vector databases. I've, I, I, I tried with OpenSearch, um, it did not have, but I think there is a, a extension, uh, which you can use to make this functionality.

191. 34:44

     Okay? So now we are going to get into, uh, the demo. So just like what I said, uh, once you have those, uh, scores, like in this case, two point five eight, uh, like this, you will have for all the pages in your document.

192. 35:00

     And then at the end, uh, you can pick the top three or top four pages of your choice. Okay?

193. 35:06

     So now, see this. So far the, we are not talking about agents, okay? Because that's a very simple task. Uh, we will just wrap this with an agent at later point in time.

194. 35:16

     All right. So let's try to do this. Okay? So this is an, um, uh, I, I'll just come to this, uh, image later on. So now let me just increase, uh,

195. 35:28

     the font. Can you see this? Yeah? Okay. You don't have to read all that, but just, uh, you should have an idea what we are doing. So first, we are just importing few of the libraries.

196. 35:37

     Uh, I have no idea what I'm importing, but, uh, there are few, right? So I think it's, uh, it's a ColPali. Yeah. So this is the ColPali model that we have.

197. 35:46

     Okay? And this is the Qdrant database. And this Qdrant database, we are going to run locally in a Docker container. Okay? So if you are planning to run this, uh, make sure that you have Docker installed in your, uh, in your laptop.

198. 36:01

     Okay? I, I, I think the README have all the information.

199. 36:05

     Okay. Uh, so first we need some data. So I have used one dataset. Uh, um, basically it's a small textbook. And if you see this textbook, uh, this is a science textbook, uh, chapter number thirteen.

200. 36:22

     So We have-- Let's say, uh, see, o-one of the thing that, uh, which is interesting here is if you see this image, there is no text here, right? So it's, i-if you ask anything about this image, uh, and use a traditional technique, it might not answer properly.

201. 36:38

     Uh, like this. This is also a-another image along with some text. And, uh, you know, you can pick any dataset of your choice, but, uh, this is the dataset that I have.

202. 36:49

     Okay? And feel free to use any dataset of your choice, but, uh, for the purpose of this, uh, demo, you may like to download one of these, uh, PDF from this URL and, uh, play around with this.

203. 37:01

     And then you need to have a Hugging Face, uh, uh,

204. 37:05

     uh, token, uh, because we are going to download this model from Hugging Face, right? So you should not do this, right? So this is, uh, uh, you know, I was just trying this because without creating a .env file.

205. 37:18

     But you should have an env file. Inside that, your token should exit-- exist. Okay? So this is, uh, a token. Like, this is not my token if you see this.

206. 37:28

     Uh, this is just a dummy one, right? This is not my token. Okay. So it's... But, uh, this is, this is just, uh, the, the, the Hugging Face token that you should have.

207. 37:40

     So here we are just loading and logging into our Hugging Face account.

208. 37:45

     And next, we are trying to check whether we have a CPU, GPU, or, uh, MPS. In this case, it's a MacBook, so I'm just using MPS here, uh, as a device.

209. 37:59

     Since it's a vision-based model, it's better to run it on a GPU. It will be faster. But you can very well run it on CPU. That's fine.

210. 38:06

     I'll tell you, uh, uh, you know, you should be a little cautious about this if you're running with a, uh, within your laptop, uh, on CPU. Uh,

211. 38:16

     if it's office laptop, no one cares. But if it's your personal laptop, make sure that the batch size is very small, otherwise it will, it will crash. In fact, I-- when I first ran this, I did not check, uh, the processing time and all that.

212. 38:29

     It just, uh, went on, um, you know, crashing, and it, uh, rebooted my laptop.

213. 38:36

     Uh, and, uh, I, I did not even read through all this, and I raised a IT ticket and I actually got a laptop, new laptop, uh, so... But it was my fault here.

214. 38:46

     So they thought that my work, my work needs a laptop with more memory. So, so if you are finding out tricks to get a new laptop from a company, so this is the sell. [audience laughing]

215. 38:58

     Okay. Uh, you can try that. Uh, I'll tell you what you have to change to get a new laptop. Okay. [audience laughing] So just increase the batch size to, batch size to twelve.

216. 39:07

     It should work fine. Yeah. Yeah. Okay. So

217. 39:13

     this is the model that we are gonna use. It's a ColPali, uh, version one point three. There might be a new version, but just have a look. I, I checked the last one, it was still one point three.

218. 39:24

     And I'm having a model and a preprocessor. Remember that we discussed that we need to have a preprocessor first. Uh, we will process our data and then we will use the model to generate the embeddings, okay?

219. 39:36

     The same model, but there is a preprocessor and the model. And these all are coming from Hugging Face. And we are using a cache directory so that we can load this model locally in our, uh, local directory, uh, so that every time you run, it doesn't download from the internet.

220. 39:53

     Okay? And once that is done, uh, you have to have a vector database. So if you have a Docker installed, you can just copy and paste it. It, it is nothing, but it just created a, a container with a port forwarding.

221. 40:08

     And, uh, there is a folder which gets created locally, uh, w-- as a storage. So all your vectors will be stored locally in your laptop. That's all. And if you click on this dashboard, you should be able to see, uh,

222. 40:24

     that, uh, UI of that. And if you come to console-- Uh, sorry, um, here, collection. Initially you-- Uh, since I've executed that code, that's why we see this, but, uh, you should not see anything here.

223. 40:36

     Uh, and as you run through the notebook, you will see, uh, the, uh, collection here. So collection is... Uh, how many of you are aware of databases?

224. 40:46

     Okay, okay. Many of you. I have no idea about what it is, but, uh, I just asked. So the collection is basically, you can think of it like a database and where you will just store all the, uh, schema and all that.

225. 40:59

     So I'm creating a, a quadrant client, and this is something that I imported, uh, uh, earlier. And this is a local host and port number this. And this is just, uh, we are just creating the setup, right?

226. 41:13

     So now we have a vector database and we have the data, so n-- and we have also, uh, downloaded the model. So now the second thing that we need to do is we need to create a collection.

227. 41:25

     Right? And if you see this, uh, we have a collection called, uh, class ten science. Uh, so you can give any collection name. Here we are s-- mentioning what should be the vector size, right?

228. 41:38

     So, uh, this is the, uh, embedding length. So here it is one twenty-eight. And in this, in this code, what we are essentially doing is if there is a collection already exists, it will not create any new collection, or else it will create a new collection.

229. 41:55

     Yeah, you have a question.

230. 41:56

     Uh, yeah. Do you have any idea on how to determine, like, the best size to choose from there?

231. 42:01

     Yeah. So there is a... Le-le-le-let me ask you this. What do you feel if I increase the embedding size from one twenty-eight to two fifty-six?

232. 42:14

     What do you feel? How, how, how it would behave? Just a guess.

233. 42:18

     You've got more content in each fragment-

234. 42:22

     Mm-hmm Mm-hmm.

235. 42:27

     It might have more as well.

236. 42:30

     Okay. Okay, let, let me, let me give you an example, okay?

237. 42:35

     Let's say I- I- I have just come here, and I mentioned you two things about me. I work for Amazon,

238. 42:44

     and I'm [REDACTED:marital_status]. That's all. Okay? These are the two information that you have. Now, if he asked you some question about me that, uh, okay, uh, uh, tell me if Suman plays cricket or not, will you be able to give some answer?

239. 43:00

     No. You will be giving some answer based on these two information, but it will be random, right? But now,

240. 43:06

     let's say if I give you more information. I am Suman. I work for Amazon.

241. 43:12

     I am [REDACTED:marital_status]. I have one wife, uh, as of now. [laughing]

242. 43:18

     I, let's say I have one kid. Okay? And few other things.

243. 43:23

     So I keep on giving more features about me. You are having a richer, uh, information about me. So now if he ask you a question, it's more likely that you will be able to give a, uh, you know, you are-- you'll be able to give a more accurate answer.

244. 43:37

     Same with this. The moment you increase the embedding length,

245. 43:41

     you are... It's not about chunk and all that. It's just about how much granular information that you are having about a specific thing.

246. 43:50

     Okay? So you can always embed, uh, any entity with just one number, a vector of size one. But it will not have much of an information. As you increase the length, it will, it will be more richer.

247. 44:03

     Okay. Okay. So coming back to your question, uh, in the documentation, I think they, uh, they said that one twenty is a good number, uh, but you can always use two fifty-six, right, if the vector database also supports that or the embedding model.

248. 44:18

     Okay? So, so this is where we are just creating that collection. We have not even, uh, started creating the embeddings and all that. And see this here? Uh, this is what, um, I was referring to when I said quadrant supports that matrix multipli-- uh, that, um, uh, late interaction thing, right?

249. 44:39

     So it says, uh, I'm setting some configuration that it's- it should have multi-vector configuration and multi-vector comparator as maxim. So maxim is what, uh, helps us to get those three numbers from that matrix and then add those three numbers and give us the final value of the-- your query and each page, so that at the end of

250. 45:03

     the day, what do we want? We want the relevant pages, uh, based on our question, right?

251. 45:09

     Okay. So now once it is done, it's now pretty simple. I have to, uh, first create the embedding. But before that, I need to create, convert, uh, my data into images.

252. 45:23

     And that's what, uh, this, uh, this function does. So what it does is it go-- it takes you... it, it takes an, uh, directory, and you can have hundreds of PDF files, and it will go through all the PDF files, and it will create, uh, images of each pages.

253. 45:40

     Okay? And not only that, it will also add all of that into an, uh, list called all images. And this is just for my own housekeeping with some metadata like document ID, page number, and the actual image in the form of RGB.

254. 45:56

     And it will store it in a local directory called PDF data. And if you just see the first two entries, you will see that, okay, this is document number zero.

255. 46:05

     That is, let's say I just have one PDF, so all the entries will have document ID as zero. Page number zero, and this is the image. Page number one, and this is the image.

256. 46:16

     Okay? So this data set contains everything. With me so far? Yes? Okay, great. Now that I have these, uh, uh, images, I can use the embedding model to generate the embedding.

257. 46:31

     And, and this is where, uh, you know, I just crashed my laptop. I initially used a batch size of ten, uh, uh, twelve, so it took a lot of memory, and I had just, uh, I think sixteen gig of memory, so

258. 46:45

     it, it, it actually crashed. But, uh, if you're trying in your laptop, make sure that you start with two or three. So it basically means how many images you want to process.

259. 46:54

     And now here we are generating the embeddings. And first we are going through this ColPali preprocessor, which will just, uh, preprocess the image in a standard, uh, size. And then I'm passing it through the ColPali model, which actually generates the embedding.

260. 47:12

     So this will have my embeddings. And once I have all those, uh, embeddings, what I want to do is I want to store it in the vector database. And that is what I'm doing it here.

261. 47:25

     I'm just inserting into the collection that I've created for all the points. Each point is nothing but you can think of it like, uh, each, uh, vectors. Okay? And in this case, I have just ten pages, so it will just generate the am- amount of number of embeddings for those ten pages, and it will store it here.

262. 47:46

     Now is the final thing. You know, how we can retrieve. See this? I've just asked this question: What are the different, uh, tropical levels? Because this is there in the book.

263. 47:58

     And this question also need to be, uh, need to go through that embedding model just like images. So I will do, uh, go... I'll make that through the preprocessor and the model.

264. 48:11

     And once that is done, I will do a semantic search from the vector database. And that is what we are doing. We are just querying, uh, the vector database with our query token.

265. 48:21

     And I'm saying that the limit is five. What this limit five means? That means I need the top five pages which is relevant to this, uh, question.

266. 48:32

     And at the end, you will find some five pages. And if you want to see, uh, those five pages, how it, uh, uh, you know, how it looks like, uh, you can actually visualize.

267. 48:42

     So this is just a, a wrapper, uh, Python function, which will just take all the images, and it will just generate, um, the images in a pictorial format. Okay?

268. 48:53

     And in fact, if you see, uh, the-- this image, I think this was, uh,

269. 49:00

     this was the image, I guess. Uh, yeah. So this is where the topic and levels are mentioned. It actually identified based on the question, uh, and the, uh, ColPali embeddings.

270. 49:11

     Right? So this is a page, and also there are other pages which we got. Now comes-- So retrieval is done, right? So ColPali just talks about retrieval. Its, its job ends here.

271. 49:23

     Okay. And, uh, if you, if you think about it, um, with respect to, uh, sorry, with respect to this,

272. 49:34

     uh, sorry, here, I guess. In the traditional technique,

273. 49:40

     we came to this point, right? Uh, we came to this point. Sorry.

274. 49:50

     We came to this point when we got the retrieved images and the question is already there. Now we can use any multimodal LLM to generate the answer. Right? But we have skipped everything here.

275. 50:03

     Right? So now when we, when we use

276. 50:08

     any generative model, you can use any generative model of your choice. Uh, if you don't have any AWS account or i-if you have any other, uh, a model access, you can always use that.

277. 50:21

     Uh, but let's say, uh, you don't have Bedrock access. So we can use Ollama. Have you used Ollama? Just a local model. The response may not be that great, but you can work it out, right?

278. 50:34

     So this is, again, a wrapper function just to, uh, convert all the images, uh, into the format that the model expects because we are-- we need a multimodal LLM, right?

279. 50:46

     So we will take some multimodal LLM from Ollama, but, uh, depending on what model you're using, the model will ask you to have the input in a certain, uh, format, right?

280. 50:56

     So it needs the data to be in Base64. That's what this small, tiny function does, right? Uh, and then we just say, "Ollama generate", and this is the model that I'm using,

281. 51:08

     and I'm sending the query and the image. That's all. Right? So see this, now I'm sending the full query, not the embedding of the query because Ollama has nothing to do with that embedding of the query.

282. 51:19

     That embedding was needed just for semantic search. Right? And then, uh, we get some response. If you want to use Bedrock, then you should have Bedrock access. How many of you know about Bedrock?

283. 51:32

     Okay, perfect. So it's just a managed service on AWS through which you can access any different, I mean, different kinds of model. And the way that Bedrock expects you to give the input, uh, a multimodal input is little different, and that's why we have some wrapper functions, uh, which will,

284. 51:52

     which will make your prompt, uh, you know, according to the multimodal, uh, models requirement, right? And you can go through these two functions. It's standard, uh, you know, uh, converse API that we have used.

285. 52:06

     So nothing fancy. So I don't want to go there because that is not the purpose of this, uh, problem. But ultimately, you give the images and the query, and you mention the model ID.

286. 52:17

     So in this case, I'm using Sonnet, uh, Claude Sonnet 3.7. You can very well use, uh, Sonnet 4 if you would like to, and you generate the image. Uh, sorry, the final response.

287. 52:28

     Okay? With me so far? Okay. Now comes the agentic. How we can make this agentic. So it's very simple, right? You don't have to go through all these things because what we have done is, ultimately what we want when somebody's asking a question, we want an agent to, to retrieve the shortlisted images

288. 52:54

     and give it to me. That's all. Right? So we have seen how to shortlist those images, right? What we are gonna do is,

289. 53:02

     uh, here. Mm. I'll just go through that later. Yeah. What we are gonna do is we are going to create a function called retrieve from quadrant, which will just take your query.

290. 53:19

     And if you see the, uh, uh, return for this are the matched image paths. That is what we want. Nothing else, right? And the code here in this function are the exact same code which we have gone through in multiple cells, uh, you know, uh, previously.

291. 53:35

     It just, uh, it does the same thing.

292. 53:38

     And now to make it agentic, I have used a framework called Stance. Have you heard of Stance? Right. Okay. So Stance is a new agentic framework. Let me just show you this.

293. 53:51

     It's stanceagent.com. This is a, uh, SDK which was launched by AWS. Uh, I worked with on, on, on, uh, at the launch. There are some YouTube video as well.

294. 54:02

     Uh, you can just go over and just search for Strands Agents. You will find a launch blog as well. Okay? But basically, it is very, very simple. Just to give you an example how to get started with Strands.

295. 54:14

     Uh, you just pip install and, uh, do you want to see a quick demo of Strands before we go to that part? Will that help? Yes? Okay. So let me just show you.

296. 54:26

     I think I have that. Mm. Okay. So let me quickly spend four minutes on that. Four or five minutes. I have a good demo actually, if you want. How many of you have heard of, um, three blue, one brown?

297. 54:43

     Okay, perfect. Okay. So then let me show you that. You might, you might, uh, it might be interesting. So Strands is an, uh, framework, very simple. It's a model-first framework.

298. 54:55

     So we are just taking a pause on that, okay? We are just-- We will j- Whatever we learn here, we will just use this framework to make our workflow, whatever we have done, agentic.

299. 55:04

     And we will add a voice part of that as well.

300. 55:07

     So here, there's an open source framework, which is model first. That means, uh, now the models are so strong, we expect that the model should reason rather than we telling, uh, the agent with a lot of backstory, goals, prompting, and all that.

301. 55:26

     We don't want all of that. We throw a question, we expect the model to generate the response and do the reasoning, uh, on, on the model side. That's, that's why this is very, very lightweight, and it has the integration with, uh, different models.

302. 55:40

     And you can use models from Bedrock, you can use directly from Anthropic, you can use LightLLM. Have you heard of LightLLM? Yeah. So when you have an access to LightLLM, you can access any model that LightLLM supports, right?

303. 55:52

     Now, this is what it is. So Strands, uh, you know, by the definition of strands, uh, it's a DNA structure, and it just have two strands, and that two strands stands for model and tool.

304. 56:08

     That's all. So you make an agent with one model, uh, uh, uh, and few tools, and simply you just ask the question. That's all. It's as simple as that, right?

305. 56:18

     And let me show you, uh, one quick demo.

306. 56:23

     Let's see if it, uh, if it works. Okay.

307. 56:30

     So this is-- Is it visible from, you know, last row? No, not that much, right? Mm.

308. 56:38

     Okay. Th- Um, um, I-I just read it out. So we are just importing, um, agent and we are importing the tools. Okay? And...

309. 56:53

     Okay. This is-- I think this is the MCP one. No, no, this is not the one I wanted to show you.

310. 56:58

     Mm. Okay, let, let's, uh, see this. Okay.

311. 57:13

     Just, uh, uh, I think it's a video. It should work fine.

312. 57:22

     Yeah. Let's see. So we first install Strands agent and Strands tool. Pip install. Simple pip install.

313. 57:36

     And it's open source, okay, so you don't have to... Uh, and it supports Ollama as well, uh, so you don't have to have an AWS account or anything of that sort.

314. 57:47

     So what we are going to do is, we are going to create a file, create a summary, write the summary into the, into our file, and, uh, also add a voice part of it.

315. 57:58

     Let's see. I think it would be pretty quick.

316. 58:06

     So we are importing St-- Is it visible or I'll, uh-- Should, should I show it? It, it, it's pretty straightforward.

317. 58:13

     So we are importing agent, and we are importing the Bedrock model. By default, it uses Bedrock model. Uh, it actually uses Claude three dot seven, but you can use any other model.

318. 58:26

     And I've used some built-in tool called read file, write file, and speak. And this is the model ID, and this is the prompt. You can have a prompt, you can skip a prompt, doesn't matter.

319. 58:39

     And lastly, you have to create the agent. So

320. 58:47

     see, this agent contains the model ID, system prompt, and the tools. And all these tools, you have not written the code for this. This is by default, right? And I'm just asking, uh, a particular question.

321. 58:59

     And see this? I've, uh-- In the prompt, I'm saying that this is a textbook, uh, in my local directory. Read that, create a summary, and write it into the local directory, and also speak out the final answer.

322. 59:14

     And see this? It is using the tools for read the file. Second, it is create-- After creating-

323. 59:19

     The human eye functions like a camera-

324. 59:20

     See, this now it is speaking

325. 59:21

     ... with light entering through the cornea and focused by a lens onto the retina.

326. 59:24

     Okay. So we have not done anything, just pip install and started with this. Okay.

327. 59:26

     The iris controls pupil size to regulate incoming light.

328. 59:29

     Okay.

329. 59:29

     The eye can adjust focal length through accommodation. See-

330. 59:34

     All right. So now I, I will share one more thing. Now, I'll not show you the code. That is not the purpose of this. But have you heard of, um, uh, of course, you have heard of MCP?

331. 59:46

     Yeah. So s-see this. I'll, I'll not tell you. I think you should be able to...

332. 59:54

     I've created an MCP server called, um, uh, created an MCP server with Manim. Okay. So Manim-- Have you heard of Manim? Okay. Ju-just see that.

333. 1:00:05

     So idea is-- So let me just show you what we are doing.

334. 1:00:11

     We are creating a Manim server, and this is the MCP server. Now, this is the client, or nothing but, uh, our Strand agent, and this will call this MCP server.

335. 1:00:23

     Okay? And I can give any question. So question is, create a Manim s-screen which draws a cubic function like two x power three minus and blah, blah, blah. Okay?

336. 1:00:35

     And see what happens Now it is executing the code. It's calling this, uh, MCP server. It is working, uh, here, and then it should give you some response.

337. 1:00:54

     So it generated this video, okay? And now you will get some familiarity.

338. 1:01:07

     Looks similar, right?

339. 1:01:09

     Yeah.

340. 1:01:10

     I have not done anything. All I have used is a Manning, um, uh, SDK and created that, uh, MCP server, which can generate videos like, uh, what three blue and brown create.

341. 1:01:23

     So this is just a small demo of how you can make use of Trands with an MCP and write simple code and, you know, do wonderful things. Okay? All right.

342. 1:01:34

     So this is about Trand. The core idea of Trand is, uh, just pip install and use it with the by default tools and, uh, your model of your choice.

343. 1:01:48

     That's all. There's nothing, uh, no scaffolding, uh, beyond this. Okay? So i-it just like this. You pip install, create an instance, and just ask question. Here, we have not mentioned any model.

344. 1:01:59

     That means it will by default use Bedrock model. But in the demo, we have seen that you can define your Bedrock models here. Okay. So now let's come back to our, our, um,

345. 1:02:11

     our problem. In this case, our tool is not the default one, but the tool that we have defined. And what is that tool? The retrieval tool. And how I can create a, a custom tool is just by importing tool and just use that as a decorator on top of your function.

346. 1:02:29

     That's all. Now, this becomes a tool for me. Just like read file, write file, speak, this is just a tool for me. Okay? And we can def-- uh, we can use-- make use of Bedrock model or Ollama, up to you.

347. 1:02:44

     And now look at this. We are also importing an image reader. Why we are importing this image reader, I will, uh, uh, tell you a little later. But, uh, you, you remember that when we used this Bedrock model for final answer, when we used this Bedrock model to generate the final answer, we created some custom functions, which

348. 1:03:06

     are nothing but, uh, contains the information about how to, uh, create the prompt for your images for Bedrock models, right? So I don't have to do all these things, and, uh, I can simply make use of this image reader, which just takes an image and generates, uh, the, uh, prompt for us.

349. 1:03:28

     And now I have a system prompt. System prompt says that you are a RAG-based system and all that, and it also says, uh, these are the two functions that you have or, or the tools that you have to use and all that.

350. 1:03:39

     And that's all. And now you create an agent. Again, just like before, you define the model, system prompt, and in this case we use two tools. One is the retrieve from quadrant, which is our tool, and the image reader for the generation part.

351. 1:03:55

     Okay? And then we ask this question: What are the different, uh, tr- uh, different tropical levels? And now it just agents re- uh, generates the response. Just like before, but now everything is done by the agent.

352. 1:04:10

     And the beauty is, let's say now you want to add the voice feature. I don't only want the answer, but also the final response in the form of voice.

353. 1:04:20

     So far, I have done this. I'm just reducing the image so that everything fits in. So far, we have done this. We ask a question, it goes to a Strands agent.

354. 1:04:32

     It uses this retrieval tool, custom tool that we have created. It gets the relevant chunk, which are nothing but the shortlisted pages. And then

355. 1:04:43

     it uses any of these models, w- let's say Bedrock, Ollama, whatever, to generate the final response. And to generate this, it uses this image reader tool. Now what we have to do is to add voice functionality.

356. 1:04:57

     I will just use the speak, uh, tool. That's all. Just one, uh, import. Okay? And

357. 1:05:07

     that is what we are doing. We are just adding speak here.

358. 1:05:12

     And again, the system prompt remains the same, and I'm querying the same thing. And now when I ask this question-- So let's say-- Let's ask this question. Okay. So let me run this.

359. 1:05:24

     And let me, uh, let me just ask in the question itself. Explain the answer over a female voice in a natural way.

360. 1:05:37

     And let's see. And now let's run this.

361. 1:05:50

     I hope I'm connected with the internet, but let's see.

362. 1:05:56

     So when you run this, uh, code in your environment, you can simply remove the system prompt. You will still get the right answer. In fact, try this prompt. I have not tried, but try this.

363. 1:06:08

     Change this prompt and say that a male voice or something like that, right? Robotic, uh, uh, way of, uh-- You know, not a natural way, maybe robotic way, something like that.

364. 1:06:19

     The idea is see that whether Strands is able to,

365. 1:06:25

     you know, forward that information to the model or not. Right? So you don't need a system prompt. Uh, it may be because of my internet, uh, but it's-- it doesn't take that much of time.

366. 1:06:36

     You just give it a shot, it should work fine. Okay. So that's what I, uh, I had, uh, uh, for, for, uh, this particular, uh, workshop.

367. 1:06:47

     Um, I would-- Okay, it's now running, so it's little slow.

368. 1:06:54

     But let me... So it is now able to

369. 1:07:00

     generate the images. I mean, shortlisted the images, and now it should speak in a female voice.

370. 1:07:16

     So while that happens... Okay.

371. 1:07:18

     Trophic levels are the different feeding positions in a food chain, representing the flow of energy through an ecosystem. There are typically four-

372. 1:07:26

     Okay. So let me just stop this and let's say if I-- Let's try this, okay? Um,

373. 1:07:34

     let me delete this system prompt. And let me just have this model and the tools. There is no speak, nothing. No system prompt. There is nothing there. And here, I will change this to a male voice.

374. 1:07:50

     Okay? And, uh, I'll give it a shot. Let this... I don't want to interrupt.

375. 1:07:57

     -mers, which are small carnivores that eat herbivores. These might include frogs, small birds, or foxes. The fourth trophic level is occupied by tertiary consumers or top carnivores.

376. 1:08:09

     You can, in fact, say something like summarize in fif-- uh, you know, fifty words or a hundred words, uh, rather than waiting for this to complete. So...

377. 1:08:17

     That energy transfer re-

378. 1:08:19

     It's still going on. [laughing] Let me just give it a second.

379. 1:08:28

     And before I forget, if you want to know about that multimodal, that-- the traditional technique, in this GitHub report, there is the part three, and here you will find the details of, uh, that architecture.

380. 1:08:46

     Like, um, uh, this architecture, right? And, you know, th-this notebook is about how you can do the same thing, but, uh, preprocessing this image, text, and table. Okay? So just play around this GitHub report.

381. 1:09:00

     Okay, it's done. So now I will quickly, um...

382. 1:09:10

     I just created this agent now, but, uh, without any, uh, system prompt. Now, I just executed this, and now let's run this. So now I'm letting, uh,

383. 1:09:23

     the agent know only about the models, nothing else. There is no system prompt.

384. 1:09:44

     I just hope we get a male voice at least. [laughing]

385. 1:09:48

     Let me explain trophic levels- [laughing]

386. 1:09:50

     It worked. Perfect.

387. 1:09:55

     Think of them as the levels in nature's dining hierarchy. Starting at the base, we have the producers. These are mainl-

388. 1:10:02

     Okay, let's see if I have... Mm. Okay.

389. 1:10:21

     Male voice. Let's try this.

390. 1:10:44

     Trophic levels are essentially-

391. 1:10:50

     Now.

392. 1:10:50

     -the different feeding positions-

393. 1:10:51

     It's not happy

394. 1:10:51

     ... in a food chain, showing how energy flows through an ecosystem.

395. 1:10:55

     But it was-- It, it worked yesterday.

396. 1:10:55

     Let me walk you through the main trophic levels.

397. 1:10:57

     Yeah, you can try this out, uh, you know-

398. 1:10:57

     At the very bottom, we have the producers.

399. 1:11:00

     You can try this out and see what you can mention so that you can augment the tool. In fact, this is not the right way to do this because by default it is a, a female voice.

400. 1:11:14

     You can actually change the behavior of this, uh, speak tool. Okay? So the way that you can do it, uh, you can go to the documentation and, uh, if you see this documentation, uh, here we have tools and, uh, you can see the overview.

401. 1:11:34

     And if you see this here, the, the tool spec. Yeah. Here's a tool spec for different tools, and you can mention what persona that you want. So that is a more deterministic way, uh, to do that.

402. 1:11:51

     Or else you can put that in the, uh, system prompt. Okay? So that's all I have. Uh, uh, if you have any questions, feel free to ask or, uh, you know, uh, you know, feel free to connect and, uh, you know, would be more than happy, uh, to connect offline.

403. 1:12:08

     Yeah.

404. 1:12:09

     So, uh, I have a couple questions.

405. 1:12:11

     Yeah.

406. 1:12:11

     One.

407. 1:12:12

     Yeah.

408. 1:12:13

     So have you seen any, um, uh, companies already using this in production, and what type of scaling, uh, were you able to like-

409. 1:12:22

     Yeah, yeah. That-

410. 1:12:23

     -do with this?

411. 1:12:23

     Yeah, that's a good question. So we have used this in, uh, one of the insurance-- a leading insurance company, where they had, um, the images of driver, um, licenses, and they have the images of insurance policies and all that.

412. 1:12:39

     And we tried with different techniques. One of the technique that we used was OCR, which worked fine. Uh, but ColPali was working pretty well. And it was-- The only drawback which I have seen With this ColPali, uh, uh, model is it is very heavy.

413. 1:12:55

     But, uh, that heaviness comes only at the time of data ingestion, so when you create the embeddings. Once that is done, at the query time, it is pretty fast, okay?

414. 1:13:03

     Uh, but when you are putting the data, at that time it's a little heavy, okay? And, uh, uh, I guess

415. 1:13:11

     if you are thinking that if you have one thousand documents, each has thousand pages, you will do a search a-among all those images. That is not how it works.

416. 1:13:22

     Because imagine if I ask you the same question, forget about all this. If you use a text-based embedding model, and if you have a book of

417. 1:13:32

     one million pages, you have hundred million vectors. And when you ask a question, does the vector database search for all the vectors? No. There is a different indexing techniques that all the database uses.

418. 1:13:44

     Same indexing technique, uh, uh, are used here as well. It's just that now the vectors represent different things. Now the vector represent patches. In the previous case, the vector represent images or, uh, sorry, uh, a chunk of text.

419. 1:13:57

     But, uh, that semantic search happens, uh, very efficiently, uh, using a different indexing technique. One of the technique that we use is, uh, I think hierarchical, uh, small world navigation, um, for where it uses a tree-based hi-- uh, you know, structure.

420. 1:14:14

     Uh, it just finds a dilute node. Uh, I mean, it, it starts from the top layer. It finds one of the closest node, and whichever node is closest, then it goes down and finds its neighbor.

421. 1:14:24

     So you are just-- You can think of it like, uh, uh, you know, in, um, in computer science, we have tree pruning, right? Uh, so that's what we do.

422. 1:14:32

     So it reduces the s- uh, search space. Yeah.

423. 1:14:35

     Uh, so a quick follow-up.

424. 1:14:36

     Yeah.

425. 1:14:37

     Um, so can we see, um, like, um, more companies, uh, implement this? And then can we see this as a replacement for-

426. 1:14:45

     No

427. 1:14:46

     ... traditional RAG?

428. 1:14:46

     Yeah, that's a good question. No, I don't think this is a replacement. This is just another technique. And this is also a, you know...

429. 1:14:54

     You know, it, it's a space where things are changing very fast, right? Um, I personally feel if we get a vision-based model which is more efficient in terms of computation, this might be a good model.

430. 1:15:06

     Uh, but again, this may work for your data, may not work for your data. So it's all about your data. What I would do and what I do generally is whenever I get some problem, I try to solve with the least, uh, I mean, the most cost-effective way or most efficient way, basically.

431. 1:15:23

     More than the cost, first we have to find out which architecture works fine for my data. If that is working fine, I don't-- why to complicate things and create images and all that?

432. 1:15:34

     I will go to this only when my dataset is very much convoluted and where my-- you as a human, you feel that I can read this data only if I look at it.

433. 1:15:45

     Imagine that you have a PDF file. For that-- Simple text file. For that, you can get the answer from that PDF file, even if somebody converts that into a text file and give it to you.

434. 1:15:58

     But let's say you have a PDF file which contains mostly images and embedded text on top of that, then you will say that, "Okay, okay, don't give only text.

435. 1:16:05

     You give me the book, I will figure out, because I need to see what is the context of that." So it just like it replicates humans', uh, you know, uh, behavior to understand any data.

436. 1:16:16

     Um, so I would recommend not to start with this. Start with the traditional technique, because that is more effective, um, cost-effective, and also it, it is less heavy. Because here we are storing a lot of, uh, vectors for each page, right?

437. 1:16:30

     So, uh, but use this when you have a very convoluted data. Yeah. Okay. Perfect. Yes, sir.

438. 1:16:37

     So, um, I'm trying to get a sense for when it's good and when it's not, and I'm, I'm trying to-- I'm wondering whether when you chunk the image up into these little squares, is there an issue where when the chunks don't overlap, you know, let's say you're in the middle of a paragraph and you kind of chunk

439. 1:16:52

     that into, um, two different segments-

440. 1:16:56

     Mm-hmm

441. 1:16:56

     ... does that cause problems in practice?

442. 1:16:58

     Yeah, that's a good question. But here the model doesn't know that, that there is any chunking or anything. That we are understanding it that way. But to the model, it's just an image, and the way that it creates the embedding for that image is by, uh, by doing that, those patches.

443. 1:17:17

     And why the model knows this? Because the-- when the model was trained, it's a vision-based model. So when the model was trained, it used to chunk all the, uh, training dataset like that.

444. 1:17:28

     And that's how, uh, the, you know, it has optimized, uh, for that data. For example, during the training time of ColPali, not at the inference time, during the training time, when it was given an image of a cat and the text about a cat, the cat image was also chopped into those many pa-- uh, ch- uh, patches.

445. 1:17:48

     Similarly, when there was an image of a, of a PDF page, it was chopped with the same, uh, patches. So that was inherited during the training process itself. So we don't have to question that, "Okay.

446. 1:18:01

     Hey, model, how you are doing this?" Uh, you-- The model will say that, "I have been doing this. Don't give me advice. I have been doing this, uh, with, uh, you know, the plethora of data."

447. 1:18:10

     So if we, if we just look at it blindly from outside, I also had the same thought. How the model is going to create an embedding when it splits a table into multiple chunks?

448. 1:18:24

     What is the relationship between one chunk and the other chunk? How the model is, uh, you know, uh, doing that? Later on, we realized that this has been incorporated during the training process itself.

449. 1:18:33

     Initially, it was not able to do that, right? But when-- During the training process, the loss must have been very high, right? So... And that's how it has been optimized.

450. 1:18:43

     So one that is optimized, you don't have to worry about that. And this is basically, if you think about this, this patching and embedding, it's, it's not a new technique.

451. 1:18:53

     Uh, uh, you know, it was there in lot of, uh, vision-based models. Now we are using it for retrieval. So that's, that's how it works.

452. 1:19:02

     In fact, if you are curious, I would recommend, I, I, I'll try to do this later on, but I would recommend that, uh, try to fine-tune this model or train it, uh, from scratch if you have some resource, um, in a-- for a smaller dataset, um, and use a different patch size.

453. 1:19:20

     Uh, uh, so let's say start with a patch size of four, right? And, uh, you know, try to see that, how it works. Uh, I have a lot of assumptions, uh, on that, but this will give you a lot of clarity of how, uh, the semantic search things work and why that matrix max-- matrix multiplication that we

454. 1:19:39

     have done, right? Why that is a good technique. Uh, uh, because imagine you have uploaded-- your dataset is the attention you all you need paper. And you ask a question about what is positional embedding.

455. 1:19:56

     Now, this positional embedding, this text, is there in lot of pages, almost all the pages. It should not give me all the pages, right? So it should give me the page where there is an actual information of positional embedding is there, right?

456. 1:20:10

     And when you, when you think through that, you will find out that the ma-max multiplication that, that we have done, right, that actually takes care of that. That, uh, it will just show you the page where all the tokens of your query has the maximum similarity with a particular page, not just, uh, one chunk of your question,

457. 1:20:30

     uh, with, uh, just one patch of your page. Are you getting what I'm saying? Otherwise, you know, when you say top five, it will give you any five, uh, random pages where this positional embedding is written.

458. 1:20:41

     So just give it a shot. Yes, sir. Yeah.

459. 1:20:46

     Is there any sort of hybrid approach where you can-

460. 1:20:49

     Mm.

461. 1:20:49

     -process, you know, um, PDF and only send the image-heavy stuff to ColPali?

462. 1:20:55

     This is something that, uh, one of my teammate started to work on, uh, where we are trying to use, um, ColPali along with the traditional technique. And the way that we are trying to do this is based on the question that we are getting.

463. 1:21:10

     And while we are doing the pre-processing and, uh, storing the embeddings, we are trying to store, uh, in a different way, like not for all the data that we are using ColPali.

464. 1:21:20

     Just for a few data we are using ColPali. For the rest of the data, we are just using the traditional technique. But for a particular dataset, we just use one single model.

465. 1:21:28

     We cannot just go into that, okay, first five pages of this document, we will use ColPali. The next five pages, we will use the traditional technique. That's not how, uh, uh, you know, uh, uh, uh, we are exploring.

466. 1:21:39

     Uh, but we are kind of trying to use two different approach in the same, uh, uh, architecture. But this is-- We are using because the dataset that we got from the customer, they started off with a requirement, certain requirement, then it changed.

467. 1:21:53

     It changed means it appended. And now when the new request came, the dataset is completely different. But they want a one unified system. So that's why we are just checking the question is coming from where, and we are storing some metadata to identify this question should go from this space or that space.

468. 1:22:08

     Uh, but, uh, nothing beyond that, that I have seen. I've seen either this or that. Yeah. Yes sir.

469. 1:22:14

     Did you have to fine-tune the ColPali model to for it to work well on the-

470. 1:22:19

     No, I have not done that. So this is, uh-- These are all, uh, fine-tuned models. You can just make use of this. I forgot the dataset that they have used.

471. 1:22:27

     Uh, you can read the research paper on that. The, the link is there. But you don't have to fine-tune that. Can you do that? Yes, of course, you can do a fine-tuning.

472. 1:22:34

     That's what I was referring to him. I myself have not done that, but I will certainly try this out, uh, to fine-tune that. That's a good exercise.

473. 1:22:42

     So it worked well for your use case?

474. 1:22:43

     Yeah. It just worked fine. Yeah, yeah. Yeah, yeah. Because I use a standard textbook, uh, which are publicly available, uh, but convoluted data. Try to do that with, uh, um, IKEA dataset.

475. 1:22:55

     IKEA dataset is good because you cannot use an OCR-based techniques in that, uh, dataset, and because that's a very strange, sparse dataset. And that will give you a good intuition that, okay, this is, you know, you can understand-- Only you can answer those questions.

476. 1:23:09

     If you-- I-if somebody asks you that question from that, uh, IKEA, uh, manual, you can do that, not, um, a, a computer if you use a traditional technique. So that actually a good data point to make use of this.

477. 1:23:22

     Yeah. Okay. All right. Thank you so much everyone for, uh, uh, coming. I really appreciate it. Thank you. [audience clapping]

478. 1:23:34

     And, uh, and one last thing is if you need, uh, uh, any AWS credit for any of your project, uh, just ping me on LinkedIn. I'll share a few credits.

479. 1:23:43

     Even if you need more, I can give you more. [upbeat music]