Bypassing the Multimodal Tax: Framework-Free Hybrid RAG, Raw SQL RRF, and Live UI Telemetry

Talk transcript

1. 0:01

   Hello, everyone. I'm Abed Matini. I'm Senior Backend Developer at Ogilvy.

2. 0:10

   Thanks to AI Engineer World Fair 2026 for giving me a time slot for Online Track. And, uh, I'm going to walk you through bypassing the multimodal text today, how we can have a, have a framework for your hybrid RAG, uh, withdraw SQL RRF, and live telemetry.

3. 0:32

   As you can see, I have got my live demo in the right side of the screen. So, uh, any slide that I pass and if there is anything related, um, uh, related to my, uh, presentation, I can show you.

4. 0:50

   I have a few tabs also ready, uh, here for you to show and walk you through, uh, the slides and the demos.

5. 1:04

   Okay, let's begin. So we have two problem for every document... Uh, we have two problems that, uh, we try to solve today. One is when we start to, uh, chat, uh, with any LLM, and if we have to upload a document, uh, we usually gonna just drag and drop a PDF or a Word doc or an image

6. 1:29

   and ask the chatbot to get ready for the questions that's gonna come up after we upload those documents. The issue with that is, uh, as soon as we upload these documents, we're going to, uh, basically lose some of our tokens that we supposed to have only for,

7. 1:49

   uh, processing these documents without even asking a question. So we already, uh, spent some tokens there without even asking any questions. That would be the first issue. And then if we're building a chatbot and it's, uh, we wanna have vector database there, we wanna have keyword search and semantic search there, and we wanna just add more

8. 2:14

   tools, this is gonna be too many tools, uh, uh, combined all together. So in the production, uh, in the production basically chatbot, we're going to have, uh,

9. 2:27

   too many tools and it's gonna be complicated for us to manage, uh,

10. 2:32

    our production chatbot properly. So today I'm going to, uh, walk you through in, uh, three main, uh, sections. One is how we can, uh, upload our documents and get our documents and make it ready bef- uh, for our chatbot before, uh, user's gonna chat.

11. 2:56

    So on the, on the right side I ha- I do have a FAQ assistant chatbot that's running locally now for me. And, um,

12. 3:09

    I'm gonna show you the dashboard, the dashboard that I've created for it and how we can chat and, uh, how we get the results. So this is gonna be a, uh, supposedly FAQ, um, assistant for a sample employee handbook.

13. 3:26

    So imagine that you have a HR company and this HR has uploaded the, the, the, the handbook for, uh,

14. 3:34

    for, uh, for the employees to ask their question, any question they have about leave, about, uh, uh, like, uh,

15. 3:46

    different kinds of leaves or sickness or, uh, parent, uh, like, uh, parental leave or any other questions related to or, uh, possibly an HR would have as a common question that they get every day.

16. 4:00

    So, um, that would be one sample that we're gonna use to upload. So our documents can get, uh, can get in the different formats, or we can upload in different formats of PDF, uh, like PowerPoint documents and Word documents or even image.

17. 4:18

    So we're going to talk about that. We're going to talk about how we have to chunk these because we're talking about RAG here. We need to know how we can chunk this information, uh, properly based on our, the doc- uh, documents.

18. 4:32

    So we're going to talk about, uh, different type of chunking here as well. We can, uh, we show you in the admin how we can choose different,

19. 4:43

    uh, different strategies to, uh, to chunk the documents. Then we're going to see how the search is gonna work, how the embedding is gonna work, how the... our database is gonna be, how the search is gonna be based on the keyword search or the semantic search.

20. 5:00

    And we're gonna see about, uh, how RRF in the Python is gonna work. And then we're gonna talk about the observability, how we're gonna check the safety, how we're gonna observe, uh, all the process with Langfuse, and how, uh, we can prevent, uh,

21. 5:19

    prompt injection pattern or, um, any risky question that might come to LLM. I'm planning this talk for about under an hour and this, uh, the whole thing, the plan is the entire code base can be easily, uh, be running on a GitHub code space, which you can just pull from the repo and one command run, and it's

22. 5:43

    gonna be ready to test on GitHub code spaces.

23. 5:52

    Um, so the stack we're using here are Python FastAPI for the back-end, React for the front-end, our PostgreSQL for our database.

24. 6:04

    Uh, Docker for just easily being able to conte-- uh, have the containers and we can just re-re, uh, produce it anywhere and rerun it anywhere, for example, in code spaces or code spaces or any other, uh, environment we want.

25. 6:19

    Then we have Ollama, local language models, some local language models and embedding models that are running only locally that can be run on any, uh, server as well. It doesn't need the GPU.

26. 6:32

    The CPU is gonna be good enough for it. So, uh, any staging server, any code space, um, uh, going, going to be fine for it. And Langfuse for, uh, tracking what's going on and how we can see the chat and the latency, uh,

27. 6:52

    so we can improve our chats better. Um,

28. 6:59

    this is the end-to-end blueprint of, uh, what we're going to talk about. So we're going to have a document-- few documents as a source of truth for our chatbot.

29. 7:12

    So we're going to upload some documents. So what we can do with Python is we're using Doclink to convert those, uh, raw documents, uh, to Markdown files, and then we're going to chunk those Markdown files to, uh, in our, into our, uh, Postgres.

30. 7:31

    And then we can also do some, um, scan as well here. So, uh, that would happen as well. Then we would have some queries from the user. So the employee is asking questions, HR-related questions, and it's gonna be retrieved from the database.

31. 7:46

    Is the, the, the result, relevant result when the top rank would be passed through the LLM, and LLM would, uh, return the, in the, in our chatbot that we have here, uh, in a format of chat.

32. 8:00

    So we can see, we can get our questions, uh, answered here, and we can observe in the Langfuse what's going on in our system.

33. 8:11

    So, um, optimizing data ingest. So we have two different ways of, uh, optimizing. So if we first, uh, gonna just throw it into the... We have documents and we, we're just uploading it on our, um,

34. 8:30

    uh, cloud chatbot anywhere. The issue we just, uh, having-- The one issue we have is we're spending tokens before we doing anything. Another issue is you don't know how that document looks li-looks like there on that chunking, uh, how it's being chunked and how, uh, basically that LLM is seeing it.

35. 8:50

    So that would be a risk quality if, if there is a table that we don't know how it's being read there, so we kinda don't see. We're trying to, uh, have more clarity about how the, how the document's being stored in our database and how it's being retrieved.

36. 9:06

    So another way would be a structure first. So we use the local CPU. If we're running it here, we, we can, uh, upload our documents. Our Doclink would convert it to a Markdown file, and we would know the better chunk, uh, how the chunking is gonna work.

37. 9:24

    I'm gonna show you a few examples. And, uh, of course, it's gonna be a cheaper prompt if after that you wanna talk to a lo- um, an online cloud.

38. 9:34

    Any- but if you wanna, uh, if you wanna have it locally, then it would be actually no cost for you. Uh, so the Doclink pipeline would be,

39. 9:44

    uh, we, we run it ba-- uh, it's like a local, uh, locally running it on our server. The Py- the, the Python is gonna get that document. It's gonna export it to the Markdown and save that Markdown file.

40. 10:00

    Then we can-- then it's gonna chunk it based on different strategy and, uh, we're using embedding models to embed it to our database and insert that on, to our, uh, vector database.

41. 10:16

    Uh, so if we say why local? Because, uh, imagine if, if one or two pages the cost is next to nothing, but if you have two hundred pages, um, then the cost is gonna be too much.

42. 10:29

    And if you have a, a big, uh, company with lots of employees and they wanna just ask, keep asking the same questions or, uh, if they wanna upload documents, then it's gonna be, uh, gonna be a big cost.

43. 10:46

    So, um, there are four ways of chunking a document. Let me first... Before I show you the four ways, let me, um, show you the admin page. That what I'm talking about.

44. 10:59

    So imagine here you have, uh, we are, we're having our FAQ admin page. So, uh, imagine you have, uh, let me delete few of the documents that I have chunked before here.

45. 11:16

    So, uh, you wanna upload a new FAQ, uh, file acro-- uh, for your HR, as for, uh, for your HR policy. I wanna show, uh, give your chatbot, uh, data updated with our RAG system.

46. 11:33

    So imagine you have, um, you have some question answers here. Uh, before I do that, let me show you here what I mean. So, um, this is the heading FAQ here.

47. 11:50

    So, um, one way that we can feed our chatbot with information is if we actually create a question answer, uh, file. So that would be easier to digest for our, uh-

48. 12:06

    system to understand what's going on. So part of that is, uh, imagine we're having this entire FAQ file. So the entire, uh,

49. 12:17

    basically handbook. If we want to chunk this, uh, handbook and just upload it, imagine, uh, we're uploading the entire PDF. This is gonna be the way our PDF is going to look like after chunking.

50. 12:30

    So it started by acknowledgement, then you see a few chunk that doesn't have any meaning, and then something starts here. There are some chunks that doesn't have any meaning.

51. 12:44

    Uh, for example, here signature date that's basically kind of useless. So this is gonna slow down and reduce the accuracy of our chat and it's gonna increase hallucination. So, um,

52. 12:59

    instead of uploading, uh, straight, uh, twenty-eight pages, we can... If we can divide our documents to different chunks and just up- to different files, relevant files, and clean up that data a bit as much as we can, uh, and then upload it, then we will have a better result.

53. 13:19

    So, uh, one way of, uh, uploading in the FAQ case is to actually have your-- you're turning your data to question and answers. So, um, so talking about working hours, talking about different situations here, so you can have it based on questions and answers.

54. 13:41

    Then this way you know when you reference your documents, you're having the, uh, correct reference and it won't be any confusing.

55. 13:52

    And there are, uh, different way to, uh, upload your documents. One way is you have the question and answer, and then you would like to upload it. This is my FAQ PDF file, uh, same PDF file, and I would like to upload it based on, uh,

56. 14:11

    heading-based chunking strategy. That means, well, while I'm explaining the, the parsing where DocLink is happening in the background. So that means each, uh, chunking would be based on the heading that DocLink would find and the answer that comes based on that.

57. 14:31

    So each heading and answer is going to... Heading as a question and answer is going to become a chunk. So we know our chunking is clean, and it's easy to, uh, reference.

58. 14:46

    So when this chunking finishes in a second, we would see, okay, this is the first FAQ. It could chunk successfully to, uh, uh, based on the headings. So it recognized the heading question, uh,

59. 15:04

    and then the relevant answer. So each chunk has its own, uh, question and answer here. So if we say, uh...

60. 15:17

    If we say, for example, um, as our questions here,

61. 15:29

    uh, do we, do we have... Or something like this. So we should expect a kinda answer with the reference that's coming to that chunk. So we're talking about the heading chunk.

62. 15:44

    It's gonna understand the documents based on, uh, headings that you have. That's how it's gonna chunk each document, heading by heading. So as you can see here, um, our question was about how much...

63. 16:01

    This was the question we prepared, and this was the answer we prepared. Uh, sample co-occur ninety percent of individual employee premium and seventy-five percent dependent premium. What we see here as a reference, we're getting that answer.

64. 16:18

    And not only that, you're getting the correct answer. You're also getting the, uh, you see that it's been... This ge- answer has been generated based on this document, based on this chunk.

65. 16:30

    So it's easy to reference. It's easy, you know, where the issue is. If there is anything's not working, you know why. Then you can go track it why this right chunk hasn't been retrieved.

66. 16:41

    So in this way, you have more, uh, more, uh, clear path towards how you can debug it. Because I was, uh, in our system, I was asking for, um, return top two chunks.

67. 16:58

    It would, uh, suggesting me... I'll talk about all these models here. Um, it would bring top two response. So then it was also checking, uh, that sample employee handbook, the big, uh, file here.

68. 17:13

    Sample employee handbook. As you can see here, that's a big chunk. And if actually it's a good example because if you compare these two, uh, because we just drag, uh, uploaded the entire chunk here.

69. 17:27

    You can see although it's returned some information, but it's not that accurate and you don't know... Of course it went through that file and found something, but, uh, you don't know how accurate that information could be and you can't follow because the chunk is, is just, uh, uh, untrackable that easily.

70. 17:50

    So one way of chunking is chunking based on the headings. Second way of chunking is based on the paragraph. So technically when you upload your documents, you would say, um

71. 18:06

    Just chunk each paragraph you catch, have it as a separate chunk. Doesn't matter there is a heading or not heading or anything. So, um, if I wanna go back to the, to our s- uh, to our documents and say, let me first choose, I wanna do a paragraph chunking.

72. 18:25

    I wanna say, okay, uh, for paragraph, um,

73. 18:31

    let me find the paragraph. General policy. For paragraph, I wanna upload that. Um,

74. 18:41

    so for general policy, it's uploading here. As you would see, the whole file is just being paragraphed by paragraphs being chunked. So there is no heading, there is no, no separate information.

75. 18:55

    And if you look at the, uh, if you look at that information as well, if I can find it here quickly, um,

76. 19:07

    general, general paragraph, uh, general policy paragraph, as you can see, the information is just paragraph by paragraph. So you kinda cleaned up your data again, but di- dividing it based on a different, uh, system.

77. 19:25

    So, uh, instead of heading, you're talking about paragraphs. That's another way of, uh,

78. 19:33

    basically feeding your RAG for your information. Another way would be a fixed 512 character. You can say, okay, one of the best practices for RAG is, uh, you just divide ba- based on 512 character, and you have 64% overlap between the chunks, so it won't lose the track.

79. 19:54

    So if your data is just too, uh, is not organized, if there's just random data that you have and, uh, you can't clean that up, that's another way, uh, you can upload your document.

80. 20:08

    So that would be instead of this, you can choose another file and, um, you can say basically... There's another paragraph to, uh, for global travel, um,

81. 20:25

    file here. So that's gonna-- When it's gonna overlap here or when it's gonna see here the-- about this chunking strategy, you would see, okay, it started for chunking, and then it ended in the 512.

82. 20:38

    Uh, and then it's again, uh, using that, uh, I think it was 64, uh,

83. 20:47

    it was 64 overlap. Yeah. So it's using the 64 overlap here again. So, um,

84. 20:55

    you would see it's basically, uh, chopped it from here

85. 21:02

    and, uh, so from here till here is 512, and from here to here is 32, then another 32 from here to, um, here. So this is the 32, 32, 64 in total.

86. 21:19

    So that's how it's relating each chunks together. It's happening same for here. Uh, as you can see, it's happening same for here. It's the way that, um... But you see the issue with that is it's coming here to assist and assistance, so it's breaking.

87. 21:36

    It's not the best, but in some cases it works for you. Another way that, uh, we can do is, uh, sentence base. So it can be based on five.

88. 21:48

    It's counting based on this number of sentences, and it chop it there. There are different use case for each ones. So another interesting one is imagine that you, uh, there's some maintenance coming, you're receiving some email, and you just, uh, wanna quickly upload, um, the

89. 22:11

    document in your chat without, um, without... 'cause you, you don't have the... It's just a screenshot and you can't do anything. Imagine you just receive an email and says, "Okay, there is a, a maintenance plan for this day, uh, from this hour," and you just want to upload this in your chat, uh, for the, for the user.

90. 22:36

    So, uh, if me as employee ask, "Okay, what is... Is there any maintenance plan for this weekend?" Then I would know, uh, immediately. So another way that we can easily upload that screenshot as well, um...

91. 22:53

    Um, let me find the... Easily upload that.

92. 23:02

    Oh, sorry, that was in the, my screenshots. So the screenshot, uh, you can upload, then we're using another, um,

93. 23:13

    uh, model that just turns that, uh, image to text and text to .md, and then we're using this, uh,

94. 23:26

    uh, basically, uh, what's the name? Uh, the sentence model for the emails that can convert,

95. 23:34

    uh, that image to a text and then it can feed. So we can see we're using the chunking strategy, a sentence group. It's turned that image to text, and it's already, uh, good for the,

96. 23:50

    uh, for our chatbot, so we can, uh, uh, we can index it into our knowledge base now. So then it's gonna be a next one that's been uploaded here, screenshot.

97. 24:01

    We can see how it's been chunked. Because this is just an email, it doesn't have heading, it doesn't necessarily have all the paragraph chunked properly. So, and if it's a shortage of time, it's a small message, temporary change.

98. 24:13

    You don't need to sit and clean up the data for here. You just wanna drag and drop, update it for a weekend, and everybody knows, okay- Uh, this weekend we having a maintenance plan.

99. 24:25

    So, uh, when then we come here and ask, "Is there any maintenance plan for this weekend?" Then technically we should get the result of, okay, this is the maintenance plan that's been happening and the reference documents.

100. 24:45

     So, uh, the maintenance p- uh, this is the exact maintenance, uh, window that we got. And ac-actually the reference you can see that's a screenshot that, uh, we uploaded just now.

101. 24:59

     So if we do-- If we... [chuckles] Had I done the screen, uh, uh, this, the file name better, that would have been even a more, uh, relevant. And somebody says, "Okay, according to this information that I got from here, we have a maintenance plan, uh, tomorrow from 6:00 p.m.

102. 25:17

     onwards till, uh, midnight." So... And I can easily archive, uh, delete any of these, and then it won't be visible as well. So, uh, these are basically different ways that you can chunk.

103. 25:30

     Uh, that's one of the-- That's basically the one of the most important part of the whole system because most of the LLMs would just work. The, but the way you wanna feed that data and you wanna clean that data and, uh, not for, not having, uh, just drag and drop the entire handbook here and have a messy

104. 25:51

     data, that would help you a lot to have a more successful, uh, in this case, FAQ chatbot. Uh, yeah. So these different strategies would help you as well. And, um, yeah.

105. 26:07

     Let's move on to the next slide. Um,

106. 26:17

     so we using, uh, we using Postgres and, um,

107. 26:24

     we are also having a few agents that they are just, uh, Python codes. They are not really any-- We're not calling another LLM as a agent to do something for us.

108. 26:36

     The re- one reason for that is the speed, 'cause I'm running this locally. I'm having running this on Ollama. I don't have-- I don't need to three, four agents to loop three times, four times because then I'll have to wa- I have to wait 20, 30 seconds for it, and user would lose the interest.

109. 26:54

     You want, just want one natural language to run. The rest, if something can be done by, uh, a Python function, lets the Python function then run it for you.

110. 27:06

     Then you have also the full control over the function. It won't be any hallucination because you need to, you're gonna to write your test suite to, to just cover most of the situations.

111. 27:14

     So, uh, that would help you. The agents, um, I just wanna show you something here in the settings. Uh, what can it be? So I can, uh, I can, uh,

112. 27:28

     off and have it, uh, agents off and on as well. So I can say, uh, if somebody wants a current date, then this function easily can be called and retrieved.

113. 27:39

     You don't need the LLM to bring you back the current date. You don't need the LLM to calculate for you something. 'Cause also another plan for expanding this, uh, uh, expanding this chatbot is having it for, uh, customer service for that's gonna talk about the products and the prices and calculations.

114. 27:58

     So that's another thing, product info or even having another deeper search on the agent mode. But the point with that is then, uh, 'cause it's gonna be another search, then you would lose that referencing on that.

115. 28:11

     That would be important. So, um, while I'm here, I'll show you what do I have, um, uh, as a LLM actually I'm using here. I'm using the smallest Qwen 2.5.

116. 28:25

     So it's 0.5 billion is, uh, parameters instruct model, and it's, I think it's 400, uh, megabytes only. So it's that small, and I'm using, uh, this model also for, um, for the agent mode if I wanna run.

117. 28:44

     So I would run, and two mo- uh, uh, I would run another one. I have to, uh, run the agent mode. It's gonna make it slightly slower. Um, and then, um, yeah.

118. 28:56

     Let's go back to the slides here. Um,

119. 29:02

     so yeah. This, uh, way of working, you have a full visibility on your Python function. That's what's going on. So you only pass the in needed, needed information, clear information to LLM and receive a clear information.

120. 29:17

     So it would reduce the chance of halluc- hallucination dramatically. Um, also showing you here about,

121. 29:28

     uh, vectors and text, uh, on the, in the database. So, um, so when we're gonna chunk, we're gonna chunk the information. Uh, this is based on the, uh, ID and the section, um, and what kind of vector embedding are we doing.

122. 29:44

     And then, uh, we have our chunk. We have, we can see, uh, our, uh, basically how it's going to find the nearest neighbor here and, um, also the keyword match at the same time.

123. 30:00

     Because if, if you, uh, having a customer service product chatbot, you don't ha- don't need the nearest information. You need the exact information. If it comes to the number as well, if it's come to the product as well, you need the exact...

124. 30:17

     Uh, if it's a medical chatbot, you need the exact medication. You didn't, don't need something similar or close to it. So we need to have both, uh, both basically, uh-

125. 30:28

     Keyword search and, um, and the semantic search together. So we'll have a hybrid search that's giving you a better result. And we're talking about the number of retrievals that we're gonna have.

126. 30:42

     I have it in the system limited to two, because in this case, you don't need more than that. But if you wanna retrieve some information, if it's a product, you might have-- You don't wanna just, uh, bring top two products that's being there for the user because you might have the same brand, like twenty product or fifty

127. 31:01

     product of the same brand. If you don't let the window, your window to pull those information, those products are never gonna be sold and never gonna be shown in the chatbot.

128. 31:11

     So it's important how many you can fetch. Uh, and then the, the

129. 31:19

     more about the retrieval that when we're using the BM25 to get the exact keywords. So the previous slide was about to get the most closest with the cosine distance that we have.

130. 31:32

     Um, because when we upload the inform-- uh, information with the, uh, in our vector, in our, in our vector database, the information with the similar meanings are going to sit next to each other.

131. 31:46

     And then when we s- users search for something and ask for something, then, uh,

132. 31:53

     our informati-- uh, uh, and our s- uh, our search is gonna convert that, uh, question, that query, uh, in that database and go find the similar closest information to that question and will return it.

133. 32:09

     So that's the reason we can't just return one. We need to return more than those, so we have multiple answers and more closer answers. But here in the sparse retrieval, we need the exact answer, so we would need to, we would need to filter it based on the language, for example.

134. 32:26

     We need to based on the ID or exact product name, SKUs, or, uh, brand name or something. So that would be the,

135. 32:37

     the keyword, uh, retrieval, uh, for us. And then how we do the ranking based on, uh, uh, which is very important because when we have the both vector and the BM25, we wanna mix these and still bring the top, uh,

136. 33:00

     top two or top five relevant answers that we need, uh, for us. So more accurate. If we, if we bring too many, if we bring top twenty here and, uh, bring-- shows top twenty answers, it's gonna be quite confusing for the user.

137. 33:16

     So you just wanna have... make sure you bring in the right amount based on your

138. 33:23

     use case, bas-- uh, based on your use case.

139. 33:27

     So if you're having products, you would bring more, you retrieve more. If it's something medical, you retrieve less. Uh, you don't want

140. 33:37

     to-- information to be all over the place. You want to accurate, more accurate information because you would have more liability, obviously, on that. Um, another point here is,

141. 33:51

     um... So this is, uh, this is more about the

142. 34:02

     re-ranker. So when we have our re-ranker here, uh, that would also, um,

143. 34:11

     bring the informa-- uh, bring that, uh... And so after that score,

144. 34:20

     uh, how many of those information that shows by default. So I'm only showing two there. Um, so I've, I've, uh, briefly talked about the agent mode versus direct RAG.

145. 34:32

     The direct RAG is, is just, uh, following the... It's a fixed pipeline. It's not gonna go calling anything. You know, the, the path is q- uh, quite clear. You have embed, uh, embedded query, you have a hybrid retriever, and you just bring up the answer.

146. 34:54

     It's good if compliance is quite important for you. But agent mode, you have a few extra tools that gonna be called. So a search is a compare product. These can bring more information back.

147. 35:08

     It's less in your control. So, um, uh, that's quite important. And this takes longer because you have another agent that's going to work. So basically, it's getting this information, and then you have agent mode based on your request is going to call.

148. 35:24

     Um, so that would, that would take slight longer, slightly longer.

149. 35:31

     Um, about the, about the telemetry and application guidelines. So, um,

150. 35:40

     can use Langfuse for-- We integrated Langfuse, uh, in our function, so we can, uh, see what kinda answers, uh, we have. It's quite easy to... Everything is local. It's no outside calling or anything, so it's easy to track the information that I was chatting with the widget anonymous.

151. 36:02

     I wasn't logged in. The FAQ, FAQ chat, that's FAQ agent is name of the, uh, the whole projects that I'm running there. So each conversation has a ID. Uh, you, you can track if there was any maintenance for this weekend, and you can track the answer for it.

152. 36:24

     You can see what model I've been using. You can see I've returned the top two answers And, uh, the-- because it was top two, it's referenced twice on that image.

153. 36:36

     Um, agent mode was off. Uh, it was a chat. Two chunk of information was returned, and it took this long milliseconds. And, uh, yeah, that's basically the information we have.

154. 36:53

     If you have external models, you can even estimate the cost that you're spending. That's another thing you can, uh, you can use that for.

155. 37:03

     And, uh, if you have different sessions, different users, you can track the logins. So, um, that's give you a good insight of how users are treating, uh, uh, with your chats.

156. 37:17

     So, um, yeah. So as I said about the user sessions and IDs, um,

157. 37:27

     and then you would have the guardrails, um, as well. So something because of u- uh, using functions for it, we can, uh, have a better control over it. So if I ask this chatbot, how do I, uh, treat flu?

158. 37:45

     I have, uh, created a section, uh, to say without going to chat, but the point of lots of these guardrail is that the issue should stop before we, uh, the prompt injection or anything that we have, uh, uh, or a question that is we're not supposed to respond.

159. 38:06

     We shouldn't even send it to the LLM to generate anything. So before even in the code, I can have... So we have created a section here, uh, to say if there is a medical escalation, bring this medical message back.

160. 38:22

     Uh, and you can see how the system prompt is quite small because I'm basically prompting everything in the code. So, uh, although the system prompt looks quite small and just few sentences, uh, I'm actually prompting way more in the, uh, code

161. 38:41

     instead of prompt. We u- I'm using the code to say what do's and don'ts and, uh, another example is the medical escalation, and you can add more based on your, uh,

162. 38:52

     concerns you had for your product. So, um, yeah. So this is an example for that. Um, can be, uh, can needs to be expanded with different keywords, with different kind of sentences, should be cured and see how many times somebody could pass this or not.

163. 39:07

     But technically, that would be example of how we should block, or you have our guardrails should block any query that's not related. Um, can do the same with the injection and so on.

164. 39:21

     Um, what else do we have here? We talked about the Langfuse and, uh, we just need to create a-- generate a key, and you have it in your, uh, system.

165. 39:35

     It would co- uh, it would work. It's just, uh, a little bit about observability that I've talked about. Um, and then, um,

166. 39:48

     then the injection as well. So the tools for prompt injection, again, it shouldn't go to the LLM and it's being stopped there. It should be stopped before, before, uh, it's being sent there.

167. 40:01

     And, uh, yeah, we're using intent regex and term dictionaries and LLM classifier to stop any of these issues. And of course, we can expand it more. But the good thing about that, because it's just a code, it doesn't, it doesn't need a long prompt.

168. 40:16

     It doesn't need anything. Uh, uh, outcome would be obvious. You know what you are blocking and what you're not blocking. So it's not like a, a LLM that sometimes likes to listen to you as your, uh, as the instruction and sometimes just escape and does its own thing.

169. 40:33

     So part of that integrating all of these in your Python before sending it is that you can have rigid test and to say, "Okay, this is what I'm blocking, and this is why," and it passes all the tests.

170. 40:46

     So you can be more sure, um, about that.

171. 40:52

     Um, so yeah, and just a small slide about, uh,

172. 40:59

     we can just create your-- the-- and I'm having it as a, um, I'm having it here as a, as a ba- uh, as a widget. It could be also done here, um, instead of, um, in a bigger chat as well.

173. 41:16

     So another thing is you have to, in your chat as a part of the rules, you should say also if somebody understand the terms that they're using. So if not, the person would not be able to start the chat, you know.

174. 41:28

     And, uh, also I gave the, the widget capability if I want. So user, you can reset the consent as well. It's just for my demo purposes. So, and then the, the widget would show here.

175. 41:46

     Uh, if I say no, this wouldn't work. If I reset the consent, uh, if I say yes, then it will start. Um,

176. 41:55

     so yeah. So that's also about the widget. And it's because I'm using React, it can be used anything else, um, um, any other, uh, framework as well. I just use it for simplicity.

177. 42:08

     Um, here saying about, okay, the models I used. So I've used two models. Uh, I pulled the Qwen two point five. Um, actually, first I've had the seven billion, but it, um, took longer.

178. 42:26

     So for a smaller server, I've changed it to, um... Let me show you the admin here. I've changed it to, uh-

179. 42:39

     of 0.5. Otherwise, I have different options, so I was trying with different, uh, models. If you have a bigger model, it's just gonna take longer, and it might be too wordy.

180. 42:50

     Interesting, uh, finding from my side was you don't need the biggest model. You-- If you can vet your data before sending it to your LLM, a smaller mo- smallest model can sort you out and get a good, uh, uh, answer for you, and, uh, it would give you less hallucination.

181. 43:10

     It would, uh... It's safer because it's not gonna make up information. If it doesn't have information, it simply is gonna say, "No, I don't have that information." So that's another way of guardrailing, um, to not, uh, to not have that issue of your LLM randomly generating something.

182. 43:29

     Um, I have BGM3, uh, embedding model. So when I upload anything here that, uh, upload anything here, that embedding models basically, that embedding models do the embedding for me.

183. 43:43

     Uh, so ba- technically, you need two models only. The smallest, uh, chat model and, uh, one, one of the smallest, uh,

184. 43:57

     uh, one of the smallest model for, uh, embedding.

185. 44:02

     And then if you have it on Docker, one click, everything's gonna be ru- uh, running. You need your, uh, Python requirements to be installed, and then you have your React working on the front end.

186. 44:17

     And then that's all you need. Uh, one more thing I should be adding here, I haven't, that Langfuse also, um,

187. 44:26

     uh, should be added here. That's gonna be running in a different port. That's missing from here. So the key takeaway for, um, this is the structure markdown first. So we use DocLink for, uh, do the heavy job, and then we use the po- uh, Postgres vector database to just store everything.

188. 44:47

     Uh, didn't use any specific framework. Uh, we just used, uh, Ollama for our chat and no other, uh, paid framework anywhere else. We used Lang, uh, Langfuse for telemetry.

189. 45:02

     That was also free to use. And this can be, uh, escalated to different products as well. It can be document, catalog or from any database. Um, thank you very much, uh, for your time.

190. 45:18

     I would like to get connected with you and, uh, if you have any questions, if you have, uh, any con- uh, concern or something that you figure that would be, uh, better to integrate, please share your idea with me either in the comments or message me on LinkedIn.

191. 45:36

     Um, yeah. Thank you so much. Uh, thank you, AI Engineer, for the opportunity, and, uh, see you next time. Cheers. [outro music]