AI Engineer World's Fair 2026
Bypassing the Multimodal Tax: Framework-Free Hybrid RAG, Raw SQL RRF, and Live UI Telemetry
Read the talk
Building an Inspectable Local RAG Chatbot
Prepare documents before employees ask questions, inspect the chunks that drive each answer, and keep retrieval, application rules, and telemetry visible in a local Python stack.
From a talk by Abed Matini
Before you start: Familiarity with Python, SQL, and the basic idea of retrieving document excerpts before generating an answer will help you follow the implementation choices.
Why process the handbook before the first question?
Drag a PDF into a chatbot and ask it to get ready for questions. Before the first question arrives, the system has already done document-processing work that may consume paid tokens. Add a vector database, keyword search, semantic search, and more tools, and a second problem appears: the application becomes harder to operate. These are the two costs Abed Matini starts with—document ingestion and fragmented infrastructure.
His working example is a local HR FAQ assistant. An administrator uploads an employee handbook; employees ask about sick leave, parental leave, and other recurring policies. The inputs can be PDFs, Word documents, presentations, or images. Rather than leaving document preparation hidden inside the chat interaction, the admin dashboard exposes the chunking strategy and the material that will become searchable.
The walkthrough follows those documents into embeddings, keyword and semantic retrieval, rank fusion, and finally an answer with telemetry and safety checks. The intended packaging is a repository that can start in GitHub Codespaces with one command. That goal shapes the architecture: keep the components small enough to run together and make their behavior inspectable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A local stack with two paths
The application uses Python and FastAPI for the backend, React for the frontend, PostgreSQL for storage, and Docker for reproducible services. Ollama serves the local language and embedding models. Matini presents CPU-only operation as sufficient for his small deployment, including staging servers or Codespaces; this is a deployment claim rather than a hardware sizing result. Langfuse records conversations and latency. Here, framework-free means avoiding a specialized RAG orchestration framework, not avoiding web frameworks or libraries.
There are two paths through the system:
- Prepare the knowledge base. Convert source documents to Markdown with Docling, split the result into chunks, and store the searchable material in Postgres. The blueprint also includes an ingestion scan.
- Answer a question. Retrieve relevant chunks for the employee’s query, pass the highest-ranked material to the language model, return its answer in the chat interface, and observe the interaction in Langfuse.
Separating these paths lets document preparation happen before an employee needs an answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make document structure visible
Uploading a document to an opaque ingestion service leaves an important question unanswered: what did the system actually read? A table may have lost its structure, or a chunk boundary may have separated a statement from its context. Saving an inspectable intermediate document makes these failures easier to find. The structure-first path converts locally, saves Markdown, applies the chosen chunking strategy, embeds the chunks, and inserts them into vector storage. This walkthrough uses Markdown post-processing; that should not be confused with Docling’s current native structural and tokenizer-aware chunkers.
Matini motivates local preparation by contrasting a short document with hundreds of pages and repeated employee use. The intended saving is to avoid unnecessary cloud document-processing work and send smaller, relevant prompts later. Local operation can avoid per-call provider fees, but it still consumes compute and storage; no cost comparison is measured here.
The admin demonstration makes the quality problem concrete. Existing indexed documents can be removed before updated HR policies are uploaded. In the whole-handbook preview, acknowledgements and fragments such as signature dates have become chunks even though they are poor answers to employee questions. Matini recommends splitting the handbook into relevant files and cleaning them before indexing.
For an FAQ assistant, one useful preparation step is to rewrite policy material as explicit question-and-answer pairs. Working-hours information, for example, can become a question followed by its policy answer. The resulting unit is useful both for retrieval and for citation: the reference points to the specific policy explanation instead of an arbitrary stretch of the handbook.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A question, its answer, and a traceable chunk
Matini uploads the prepared FAQ PDF with heading-based chunking selected. Docling identifies the question headings, and the application keeps each heading with its answer as a chunk. The admin preview then shows those question-and-answer units individually. This makes the retrieval unit match the way an employee is likely to ask for information.
The insurance question provides the first useful end-to-end check. The sample policy says the company covers 90% of individual employee premiums and 75% of dependent premiums. The chatbot returns that information, and the opened reference shows the matching FAQ excerpt. The visible source matters as much as the fluent answer: it lets the reader check which policy text supplied the numbers.
That reference also gives the developer a debugging path. If an answer is wrong or missing, inspect whether the intended chunk was retrieved before changing the prompt or model. The demo retrieves two chunks. Alongside the clean FAQ unit, it retrieves material from the larger employee handbook; that larger chunk is harder to follow and assess. A retrieval result can contain related information without being a good, readily checkable unit of evidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When headings are not the right boundary
The next upload is a general-policy document using paragraph-based chunking. Each paragraph becomes a separate chunk whether or not a heading exists. Matini compares the admin preview with the prepared source: the document’s paragraph boundaries now determine what retrieval can return. This works naturally when each paragraph already contains a coherent policy statement.
For a less organized global-travel document, the demo uses 512-character chunks with a 64-character overlap. These are character counts, not token counts. Overlap repeats text near adjacent boundaries so that a cut does not remove all local continuity. A Python implementation of that fixed-window strategy is straightforward:
python
def character_chunks(
text: str, size: int = 512, overlap: int = 64
) -> list[str]:
if not 0 <= overlap < size:
raise ValueError("Require 0 <= overlap < size")
chunks = []
start = 0
while start < len(text):
end = min(start + size, len(text))
chunks.append(text[start:end])
if end == len(text):
break
start = end - overlap
return chunks
The code keeps the final chunk and avoids creating another chunk consisting entirely of the previous chunk’s overlap.
In the preview, Matini describes the shared boundary region as two halves adding up to the overlap. But the same example exposes the limitation: a boundary cuts through the word assistance. Overlap preserves nearby characters; it does not make a boundary semantically sound. Fixed windows are a practical option when cleaning the source is difficult, rather than a universal chunk-size optimum.
Sentence grouping offers another boundary: collect a chosen number of sentences, with five used as the example. The options differ primarily in what structure they trust:
| Strategy | Boundary | Suitable source |
|---|---|---|
| Heading | Heading and associated text | Prepared FAQ |
| Paragraph | Paragraph break | General policy prose |
| Fixed window | Character count plus overlap | Unorganized text |
| Sentence group | Sentence count | Short messages |
The next demonstration uses sentence groups for a temporary update that is too small to justify extensive document preparation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn a maintenance screenshot into an answer
Suppose HR receives an email announcing weekend maintenance, but the available input is a screenshot. The useful outcome is simple: an employee should be able to ask whether maintenance is planned and receive the current notice. Matini uploads the screenshot, converts the image to text and then Markdown with an unnamed conversion model, and selects sentence-group chunking.
The admin preview exposes the extracted chunks before the screenshot becomes part of the knowledge base. An email may have neither meaningful headings nor reliably separated paragraphs. For a short, temporary notice, sentence grouping makes it possible to ingest the update quickly without first turning it into a polished policy document.
Matini asks whether there is a maintenance plan for the weekend. The answer cites the newly uploaded screenshot and reports the sample maintenance window as tomorrow from 6 p.m. to midnight. A descriptive filename would make that reference easier for an employee to recognize. After the notice expires, the administrator can archive or delete it so it is no longer visible.
The demonstration ties ingestion quality to answer quality without changing the language model. A prepared FAQ, a coherent policy paragraph, and a quickly extracted maintenance notice need different boundaries. The common requirement is to inspect what was indexed instead of assuming that uploading the entire handbook produced useful evidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep predictable work in Python
With the knowledge base prepared, the next question is how much work should require a model call. Several capabilities Matini calls agents are ordinary Python functions. His concern is local response time: he estimates that several agents repeatedly looping could leave a user waiting 20–30 seconds in his Ollama scenario. This is an illustrative estimate, without a measured workload or hardware comparison. If a function can do the work, it avoids another generation step and can be covered by a test suite.
The settings expose switches for these capabilities. Current-date lookup and arithmetic do not require a language model to invent an answer. Planned customer-service extensions include product information, prices, and calculations. A deeper search can add information, but Matini flags a cost beyond latency: the extra path may lose the references established by the original retrieval.
The selected chat model is Qwen2.5 0.5B Instruct. Matini tentatively puts its download at about 400 MB; that is an artifact-size estimate, not a runtime-memory requirement. Optional agent mode adds another model invocation and takes longer. The intended division of labor is to let code gather and constrain the information, then ask the language model to express the result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Combine similarity with exact terms
The database associates a chunk with an ID, a section, and its vector embedding. Dense retrieval finds nearby material by meaning, while keyword retrieval provides another route to relevant text. The distinction matters when the query names a product number or medication: a semantically similar item may be the wrong item. Semantic proximity cannot substitute for an exact identifier.
The dense-retrieval slide shows a cosine-distance SQL query with a 20-candidate fetch before fusion; the FAQ’s final retrieval limit is two chunks. Those limits serve different purposes: a candidate pool gives ranking more material to consider, while the final limit controls the context passed onward. A product catalog may need a wider window than this FAQ so that other products from the same brand are not excluded before they can be considered.
Matini describes the lexical side as BM25. On the dense side, the query is embedded and compared with stored embeddings using cosine distance. On the sparse side, terms and constraints such as language, IDs, product names, SKUs, and brands help locate the intended records. BM25 here is the speaker’s description of lexical retrieval; it should not be read as the name of PostgreSQL’s built-in full-text ranking function.
The two result lists must then be combined into a useful ordering—the role assigned to Reciprocal Rank Fusion in this architecture. The important application choice is how much of the combined result to keep. Matini contrasts a small answer set with a long list that would confuse the user, and recommends wider retrieval for catalogs and tighter retrieval for focused medical questions. Retrieval breadth should follow the task; narrowing a medical result set does not itself establish clinical accuracy.
After discussing ranking and the displayed result limit, Matini contrasts two execution paths:
| Mode | Path | Tradeoff |
|---|---|---|
| Direct RAG | Embed query → hybrid retrieval → answer | Predictable sequence and easier tracing |
| Agent mode | Add tools such as search or product comparison | More information, extra calls, less control |
The fixed path is attractive when compliance and traceability matter. Optional tools broaden what the application can do, but each additional call introduces more work and another place where the answer’s provenance must be preserved.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspect the conversation behind the answer
Langfuse is integrated into the application’s functions and runs locally in the demonstration. Matini opens the anonymous widget conversation under the FAQ agent project. A conversation ID connects the maintenance question to its answer, so the interaction remains traceable even though the user was not logged in.
The trace shows the selected model, agent mode off, the chat operation, returned chunks, and latency in milliseconds. In this maintenance conversation, retrieving two chunks produced two references to the same screenshot. That is a useful distinction: the number of returned chunks is not necessarily the number of independent source documents.
For external models, telemetry can also support cost estimates. Where user identities and login information are available, session and user IDs help connect interactions across visits. These fields make the trace useful for investigating both an individual answer and how people use the assistant over time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Stop an out-of-scope request before generation
Matini tests the FAQ assistant with a flu-treatment question. Instead of asking the answer model to respond medically, the application returns a configured escalation message directing the user toward a healthcare practitioner. The branch happens before answer generation. For a request the product should not answer, the application can choose a response without delegating that decision to its normal chat prompt.
This helps explain the short system prompt shown in the settings. Some do-and-don’t behavior lives in code rather than in a longer instruction string. The medical rule is an example to expand with relevant keywords, sentence variants, and attempts to bypass it. Matini proposes the same pre-generation pattern for unrelated questions and prompt-injection attempts.
After noting that Langfuse requires a generated key configured in the system, Matini returns to injection screening. He names intent regexes, term dictionaries, and an LLM classifier as possible checks. The boundary to preserve is before the answer-generating model: using a classifier for screening is distinct from allowing the normal answer path to run, although its exact placement is not specified here.
Code makes a rule’s execution predictable and its expected behavior testable. Tests can specify which requests should be blocked and why. They still need to challenge the detector, not merely confirm its intended examples: a deterministic regex can consistently miss an injection. The medical response demonstrates an application branch, while comprehensive injection resistance requires broader coverage than that one successful case.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Gate the interface with consent
The assistant can appear as an embedded widget or a larger chat view. Before a user begins, the interface asks them to accept the terms: declining prevents the chat from starting, while accepting enables it. A consent-reset control lets Matini demonstrate both paths. React is the presentation choice for this application, but the consent gate and backend flow do not depend on that particular frontend framework.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fit the model and deployment to the prepared context
Matini initially used Qwen2.5 with 7 billion parameters, then switched to the 0.5-billion-parameter model because the larger model took longer on his small server. The admin settings retain alternative model choices. He also found larger responses could be too wordy, while the smaller model could answer effectively when given vetted context. His observation that it hallucinated less and declined to answer when information was missing is not a comparative safety evaluation; the practical design lesson is to assess the model together with the evidence supplied to it.
BGE-M3 provides the document embeddings. Matini describes the core setup as one chat model plus one embedding model. That describes the retrieval-and-answer path; the earlier screenshot conversion is an additional ingestion capability whose model was not named.
Docker packages the services, with Python requirements installed and React providing the frontend. At the end, Matini catches an omission in the deployment presentation: Langfuse also needs to run, on a separate port. It belongs in the deployment inventory alongside the application, database, and model service, rather than being treated as something that appears automatically when tracing calls are added.
The resulting architecture keeps document preparation, storage, generation, and observation separate: Docling produces structured Markdown, Postgres holds the searchable material, Ollama serves the chat model, and Langfuse exposes the interactions. It does not require a specialized orchestration framework or a paid framework service. The same arrangement can support document collections, product catalogs, or database-backed assistants, provided their source preparation, retrieval breadth, and application rules are adapted to the questions they need to answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explains Markdown post-processing, structural chunking and tokenizer-aware native chunkers.
Local execution instructions and artifact details for Ollama's 398 MB quantized Qwen2.5 model.
Model specifications and examples for multilingual dense, sparse and multi-vector retrieval.
Further reading
The 2009 paper defining rank-based fusion and evaluating it on information-retrieval benchmarks.
PostgreSQL vector-search installation, indexing and examples for combining vector and full-text retrieval.
Attack patterns, screening controls and layered defenses for applications processing untrusted prompts and documents.
Updates since the talk
Current local deployment instructions, configuration requirements and operational limitations.
Read the complete timestamped transcript
- 0:01
Hello, everyone. I'm Abed Matini. I'm Senior Backend Developer at Ogilvy.
- 0:10
Thanks to AI Engineer World Fair 2026 for giving me a time slot for Online Track. And, uh, I'm going to walk you through bypassing the multimodal text today, how we can have a, have a framework for your hybrid RAG, uh, withdraw SQL RRF, and live telemetry.
- 0:32
As you can see, I have got my live demo in the right side of the screen. So, uh, any slide that I pass and if there is anything related, um, uh, related to my, uh, presentation, I can show you.
- 0:50
I have a few tabs also ready, uh, here for you to show and walk you through, uh, the slides and the demos.
- 1:04
Okay, let's begin. So we have two problem for every document... Uh, we have two problems that, uh, we try to solve today. One is when we start to, uh, chat, uh, with any LLM, and if we have to upload a document, uh, we usually gonna just drag and drop a PDF or a Word doc or an image
- 1:29
and ask the chatbot to get ready for the questions that's gonna come up after we upload those documents. The issue with that is, uh, as soon as we upload these documents, we're going to, uh, basically lose some of our tokens that we supposed to have only for,
- 1:49
uh, processing these documents without even asking a question. So we already, uh, spent some tokens there without even asking any questions. That would be the first issue. And then if we're building a chatbot and it's, uh, we wanna have vector database there, we wanna have keyword search and semantic search there, and we wanna just add more
- 2:14
tools, this is gonna be too many tools, uh, uh, combined all together. So in the production, uh, in the production basically chatbot, we're going to have, uh,
- 2:27
too many tools and it's gonna be complicated for us to manage, uh,
- 2:32
our production chatbot properly. So today I'm going to, uh, walk you through in, uh, three main, uh, sections. One is how we can, uh, upload our documents and get our documents and make it ready bef- uh, for our chatbot before, uh, user's gonna chat.
- 2:56
So on the, on the right side I ha- I do have a FAQ assistant chatbot that's running locally now for me. And, um,
- 3:09
I'm gonna show you the dashboard, the dashboard that I've created for it and how we can chat and, uh, how we get the results. So this is gonna be a, uh, supposedly FAQ, um, assistant for a sample employee handbook.
- 3:26
So imagine that you have a HR company and this HR has uploaded the, the, the, the handbook for, uh,
- 3:34
for, uh, for the employees to ask their question, any question they have about leave, about, uh, uh, like, uh,
- 3:46
different kinds of leaves or sickness or, uh, parent, uh, like, uh, parental leave or any other questions related to or, uh, possibly an HR would have as a common question that they get every day.
- 4:00
So, um, that would be one sample that we're gonna use to upload. So our documents can get, uh, can get in the different formats, or we can upload in different formats of PDF, uh, like PowerPoint documents and Word documents or even image.
- 4:18
So we're going to talk about that. We're going to talk about how we have to chunk these because we're talking about RAG here. We need to know how we can chunk this information, uh, properly based on our, the doc- uh, documents.
- 4:32
So we're going to talk about, uh, different type of chunking here as well. We can, uh, we show you in the admin how we can choose different,
- 4:43
uh, different strategies to, uh, to chunk the documents. Then we're going to see how the search is gonna work, how the embedding is gonna work, how the... our database is gonna be, how the search is gonna be based on the keyword search or the semantic search.
- 5:00
And we're gonna see about, uh, how RRF in the Python is gonna work. And then we're gonna talk about the observability, how we're gonna check the safety, how we're gonna observe, uh, all the process with Langfuse, and how, uh, we can prevent, uh,
- 5:19
prompt injection pattern or, um, any risky question that might come to LLM. I'm planning this talk for about under an hour and this, uh, the whole thing, the plan is the entire code base can be easily, uh, be running on a GitHub code space, which you can just pull from the repo and one command run, and it's
- 5:43
gonna be ready to test on GitHub code spaces.
- 5:52
Um, so the stack we're using here are Python FastAPI for the back-end, React for the front-end, our PostgreSQL for our database.
- 6:04
Uh, Docker for just easily being able to conte-- uh, have the containers and we can just re-re, uh, produce it anywhere and rerun it anywhere, for example, in code spaces or code spaces or any other, uh, environment we want.
- 6:19
Then we have Ollama, local language models, some local language models and embedding models that are running only locally that can be run on any, uh, server as well. It doesn't need the GPU.
- 6:32
The CPU is gonna be good enough for it. So, uh, any staging server, any code space, um, uh, going, going to be fine for it. And Langfuse for, uh, tracking what's going on and how we can see the chat and the latency, uh,
- 6:52
so we can improve our chats better. Um,
- 6:59
this is the end-to-end blueprint of, uh, what we're going to talk about. So we're going to have a document-- few documents as a source of truth for our chatbot.
- 7:12
So we're going to upload some documents. So what we can do with Python is we're using Doclink to convert those, uh, raw documents, uh, to Markdown files, and then we're going to chunk those Markdown files to, uh, in our, into our, uh, Postgres.
- 7:31
And then we can also do some, um, scan as well here. So, uh, that would happen as well. Then we would have some queries from the user. So the employee is asking questions, HR-related questions, and it's gonna be retrieved from the database.
- 7:46
Is the, the, the result, relevant result when the top rank would be passed through the LLM, and LLM would, uh, return the, in the, in our chatbot that we have here, uh, in a format of chat.
- 8:00
So we can see, we can get our questions, uh, answered here, and we can observe in the Langfuse what's going on in our system.
- 8:11
So, um, optimizing data ingest. So we have two different ways of, uh, optimizing. So if we first, uh, gonna just throw it into the... We have documents and we, we're just uploading it on our, um,
- 8:30
uh, cloud chatbot anywhere. The issue we just, uh, having-- The one issue we have is we're spending tokens before we doing anything. Another issue is you don't know how that document looks li-looks like there on that chunking, uh, how it's being chunked and how, uh, basically that LLM is seeing it.
- 8:50
So that would be a risk quality if, if there is a table that we don't know how it's being read there, so we kinda don't see. We're trying to, uh, have more clarity about how the, how the document's being stored in our database and how it's being retrieved.
- 9:06
So another way would be a structure first. So we use the local CPU. If we're running it here, we, we can, uh, upload our documents. Our Doclink would convert it to a Markdown file, and we would know the better chunk, uh, how the chunking is gonna work.
- 9:24
I'm gonna show you a few examples. And, uh, of course, it's gonna be a cheaper prompt if after that you wanna talk to a lo- um, an online cloud.
- 9:34
Any- but if you wanna, uh, if you wanna have it locally, then it would be actually no cost for you. Uh, so the Doclink pipeline would be,
- 9:44
uh, we, we run it ba-- uh, it's like a local, uh, locally running it on our server. The Py- the, the Python is gonna get that document. It's gonna export it to the Markdown and save that Markdown file.
- 10:00
Then we can-- then it's gonna chunk it based on different strategy and, uh, we're using embedding models to embed it to our database and insert that on, to our, uh, vector database.
- 10:16
Uh, so if we say why local? Because, uh, imagine if, if one or two pages the cost is next to nothing, but if you have two hundred pages, um, then the cost is gonna be too much.
- 10:29
And if you have a, a big, uh, company with lots of employees and they wanna just ask, keep asking the same questions or, uh, if they wanna upload documents, then it's gonna be, uh, gonna be a big cost.
- 10:46
So, um, there are four ways of chunking a document. Let me first... Before I show you the four ways, let me, um, show you the admin page. That what I'm talking about.
- 10:59
So imagine here you have, uh, we are, we're having our FAQ admin page. So, uh, imagine you have, uh, let me delete few of the documents that I have chunked before here.
- 11:16
So, uh, you wanna upload a new FAQ, uh, file acro-- uh, for your HR, as for, uh, for your HR policy. I wanna show, uh, give your chatbot, uh, data updated with our RAG system.
- 11:33
So imagine you have, um, you have some question answers here. Uh, before I do that, let me show you here what I mean. So, um, this is the heading FAQ here.
- 11:50
So, um, one way that we can feed our chatbot with information is if we actually create a question answer, uh, file. So that would be easier to digest for our, uh-
- 12:06
system to understand what's going on. So part of that is, uh, imagine we're having this entire FAQ file. So the entire, uh,
- 12:17
basically handbook. If we want to chunk this, uh, handbook and just upload it, imagine, uh, we're uploading the entire PDF. This is gonna be the way our PDF is going to look like after chunking.
- 12:30
So it started by acknowledgement, then you see a few chunk that doesn't have any meaning, and then something starts here. There are some chunks that doesn't have any meaning.
- 12:44
Uh, for example, here signature date that's basically kind of useless. So this is gonna slow down and reduce the accuracy of our chat and it's gonna increase hallucination. So, um,
- 12:59
instead of uploading, uh, straight, uh, twenty-eight pages, we can... If we can divide our documents to different chunks and just up- to different files, relevant files, and clean up that data a bit as much as we can, uh, and then upload it, then we will have a better result.
- 13:19
So, uh, one way of, uh, uploading in the FAQ case is to actually have your-- you're turning your data to question and answers. So, um, so talking about working hours, talking about different situations here, so you can have it based on questions and answers.
- 13:41
Then this way you know when you reference your documents, you're having the, uh, correct reference and it won't be any confusing.
- 13:52
And there are, uh, different way to, uh, upload your documents. One way is you have the question and answer, and then you would like to upload it. This is my FAQ PDF file, uh, same PDF file, and I would like to upload it based on, uh,
- 14:11
heading-based chunking strategy. That means, well, while I'm explaining the, the parsing where DocLink is happening in the background. So that means each, uh, chunking would be based on the heading that DocLink would find and the answer that comes based on that.
- 14:31
So each heading and answer is going to... Heading as a question and answer is going to become a chunk. So we know our chunking is clean, and it's easy to, uh, reference.
- 14:46
So when this chunking finishes in a second, we would see, okay, this is the first FAQ. It could chunk successfully to, uh, uh, based on the headings. So it recognized the heading question, uh,
- 15:04
and then the relevant answer. So each chunk has its own, uh, question and answer here. So if we say, uh...
- 15:17
If we say, for example, um, as our questions here,
- 15:29
uh, do we, do we have... Or something like this. So we should expect a kinda answer with the reference that's coming to that chunk. So we're talking about the heading chunk.
- 15:44
It's gonna understand the documents based on, uh, headings that you have. That's how it's gonna chunk each document, heading by heading. So as you can see here, um, our question was about how much...
- 16:01
This was the question we prepared, and this was the answer we prepared. Uh, sample co-occur ninety percent of individual employee premium and seventy-five percent dependent premium. What we see here as a reference, we're getting that answer.
- 16:18
And not only that, you're getting the correct answer. You're also getting the, uh, you see that it's been... This ge- answer has been generated based on this document, based on this chunk.
- 16:30
So it's easy to reference. It's easy, you know, where the issue is. If there is anything's not working, you know why. Then you can go track it why this right chunk hasn't been retrieved.
- 16:41
So in this way, you have more, uh, more, uh, clear path towards how you can debug it. Because I was, uh, in our system, I was asking for, um, return top two chunks.
- 16:58
It would, uh, suggesting me... I'll talk about all these models here. Um, it would bring top two response. So then it was also checking, uh, that sample employee handbook, the big, uh, file here.
- 17:13
Sample employee handbook. As you can see here, that's a big chunk. And if actually it's a good example because if you compare these two, uh, because we just drag, uh, uploaded the entire chunk here.
- 17:27
You can see although it's returned some information, but it's not that accurate and you don't know... Of course it went through that file and found something, but, uh, you don't know how accurate that information could be and you can't follow because the chunk is, is just, uh, uh, untrackable that easily.
- 17:50
So one way of chunking is chunking based on the headings. Second way of chunking is based on the paragraph. So technically when you upload your documents, you would say, um
- 18:06
Just chunk each paragraph you catch, have it as a separate chunk. Doesn't matter there is a heading or not heading or anything. So, um, if I wanna go back to the, to our s- uh, to our documents and say, let me first choose, I wanna do a paragraph chunking.
- 18:25
I wanna say, okay, uh, for paragraph, um,
- 18:31
let me find the paragraph. General policy. For paragraph, I wanna upload that. Um,
- 18:41
so for general policy, it's uploading here. As you would see, the whole file is just being paragraphed by paragraphs being chunked. So there is no heading, there is no, no separate information.
- 18:55
And if you look at the, uh, if you look at that information as well, if I can find it here quickly, um,
- 19:07
general, general paragraph, uh, general policy paragraph, as you can see, the information is just paragraph by paragraph. So you kinda cleaned up your data again, but di- dividing it based on a different, uh, system.
- 19:25
So, uh, instead of heading, you're talking about paragraphs. That's another way of, uh,
- 19:33
basically feeding your RAG for your information. Another way would be a fixed 512 character. You can say, okay, one of the best practices for RAG is, uh, you just divide ba- based on 512 character, and you have 64% overlap between the chunks, so it won't lose the track.
- 19:54
So if your data is just too, uh, is not organized, if there's just random data that you have and, uh, you can't clean that up, that's another way, uh, you can upload your document.
- 20:08
So that would be instead of this, you can choose another file and, um, you can say basically... There's another paragraph to, uh, for global travel, um,
- 20:25
file here. So that's gonna-- When it's gonna overlap here or when it's gonna see here the-- about this chunking strategy, you would see, okay, it started for chunking, and then it ended in the 512.
- 20:38
Uh, and then it's again, uh, using that, uh, I think it was 64, uh,
- 20:47
it was 64 overlap. Yeah. So it's using the 64 overlap here again. So, um,
- 20:55
you would see it's basically, uh, chopped it from here
- 21:02
and, uh, so from here till here is 512, and from here to here is 32, then another 32 from here to, um, here. So this is the 32, 32, 64 in total.
- 21:19
So that's how it's relating each chunks together. It's happening same for here. Uh, as you can see, it's happening same for here. It's the way that, um... But you see the issue with that is it's coming here to assist and assistance, so it's breaking.
- 21:36
It's not the best, but in some cases it works for you. Another way that, uh, we can do is, uh, sentence base. So it can be based on five.
- 21:48
It's counting based on this number of sentences, and it chop it there. There are different use case for each ones. So another interesting one is imagine that you, uh, there's some maintenance coming, you're receiving some email, and you just, uh, wanna quickly upload, um, the
- 22:11
document in your chat without, um, without... 'cause you, you don't have the... It's just a screenshot and you can't do anything. Imagine you just receive an email and says, "Okay, there is a, a maintenance plan for this day, uh, from this hour," and you just want to upload this in your chat, uh, for the, for the user.
- 22:36
So, uh, if me as employee ask, "Okay, what is... Is there any maintenance plan for this weekend?" Then I would know, uh, immediately. So another way that we can easily upload that screenshot as well, um...
- 22:53
Um, let me find the... Easily upload that.
- 23:02
Oh, sorry, that was in the, my screenshots. So the screenshot, uh, you can upload, then we're using another, um,
- 23:13
uh, model that just turns that, uh, image to text and text to .md, and then we're using this, uh,
- 23:26
uh, basically, uh, what's the name? Uh, the sentence model for the emails that can convert,
- 23:34
uh, that image to a text and then it can feed. So we can see we're using the chunking strategy, a sentence group. It's turned that image to text, and it's already, uh, good for the,
- 23:50
uh, for our chatbot, so we can, uh, uh, we can index it into our knowledge base now. So then it's gonna be a next one that's been uploaded here, screenshot.
- 24:01
We can see how it's been chunked. Because this is just an email, it doesn't have heading, it doesn't necessarily have all the paragraph chunked properly. So, and if it's a shortage of time, it's a small message, temporary change.
- 24:13
You don't need to sit and clean up the data for here. You just wanna drag and drop, update it for a weekend, and everybody knows, okay- Uh, this weekend we having a maintenance plan.
- 24:25
So, uh, when then we come here and ask, "Is there any maintenance plan for this weekend?" Then technically we should get the result of, okay, this is the maintenance plan that's been happening and the reference documents.
- 24:45
So, uh, the maintenance p- uh, this is the exact maintenance, uh, window that we got. And ac-actually the reference you can see that's a screenshot that, uh, we uploaded just now.
- 24:59
So if we do-- If we... [chuckles] Had I done the screen, uh, uh, this, the file name better, that would have been even a more, uh, relevant. And somebody says, "Okay, according to this information that I got from here, we have a maintenance plan, uh, tomorrow from 6:00 p.m.
- 25:17
onwards till, uh, midnight." So... And I can easily archive, uh, delete any of these, and then it won't be visible as well. So, uh, these are basically different ways that you can chunk.
- 25:30
Uh, that's one of the-- That's basically the one of the most important part of the whole system because most of the LLMs would just work. The, but the way you wanna feed that data and you wanna clean that data and, uh, not for, not having, uh, just drag and drop the entire handbook here and have a messy
- 25:51
data, that would help you a lot to have a more successful, uh, in this case, FAQ chatbot. Uh, yeah. So these different strategies would help you as well. And, um, yeah.
- 26:07
Let's move on to the next slide. Um,
- 26:17
so we using, uh, we using Postgres and, um,
- 26:24
we are also having a few agents that they are just, uh, Python codes. They are not really any-- We're not calling another LLM as a agent to do something for us.
- 26:36
The re- one reason for that is the speed, 'cause I'm running this locally. I'm having running this on Ollama. I don't have-- I don't need to three, four agents to loop three times, four times because then I'll have to wa- I have to wait 20, 30 seconds for it, and user would lose the interest.
- 26:54
You want, just want one natural language to run. The rest, if something can be done by, uh, a Python function, lets the Python function then run it for you.
- 27:06
Then you have also the full control over the function. It won't be any hallucination because you need to, you're gonna to write your test suite to, to just cover most of the situations.
- 27:14
So, uh, that would help you. The agents, um, I just wanna show you something here in the settings. Uh, what can it be? So I can, uh, I can, uh,
- 27:28
off and have it, uh, agents off and on as well. So I can say, uh, if somebody wants a current date, then this function easily can be called and retrieved.
- 27:39
You don't need the LLM to bring you back the current date. You don't need the LLM to calculate for you something. 'Cause also another plan for expanding this, uh, uh, expanding this chatbot is having it for, uh, customer service for that's gonna talk about the products and the prices and calculations.
- 27:58
So that's another thing, product info or even having another deeper search on the agent mode. But the point with that is then, uh, 'cause it's gonna be another search, then you would lose that referencing on that.
- 28:11
That would be important. So, um, while I'm here, I'll show you what do I have, um, uh, as a LLM actually I'm using here. I'm using the smallest Qwen 2.5.
- 28:25
So it's 0.5 billion is, uh, parameters instruct model, and it's, I think it's 400, uh, megabytes only. So it's that small, and I'm using, uh, this model also for, um, for the agent mode if I wanna run.
- 28:44
So I would run, and two mo- uh, uh, I would run another one. I have to, uh, run the agent mode. It's gonna make it slightly slower. Um, and then, um, yeah.
- 28:56
Let's go back to the slides here. Um,
- 29:02
so yeah. This, uh, way of working, you have a full visibility on your Python function. That's what's going on. So you only pass the in needed, needed information, clear information to LLM and receive a clear information.
- 29:17
So it would reduce the chance of halluc- hallucination dramatically. Um, also showing you here about,
- 29:28
uh, vectors and text, uh, on the, in the database. So, um, so when we're gonna chunk, we're gonna chunk the information. Uh, this is based on the, uh, ID and the section, um, and what kind of vector embedding are we doing.
- 29:44
And then, uh, we have our chunk. We have, we can see, uh, our, uh, basically how it's going to find the nearest neighbor here and, um, also the keyword match at the same time.
- 30:00
Because if, if you, uh, having a customer service product chatbot, you don't ha- don't need the nearest information. You need the exact information. If it comes to the number as well, if it's come to the product as well, you need the exact...
- 30:17
Uh, if it's a medical chatbot, you need the exact medication. You didn't, don't need something similar or close to it. So we need to have both, uh, both basically, uh-
- 30:28
Keyword search and, um, and the semantic search together. So we'll have a hybrid search that's giving you a better result. And we're talking about the number of retrievals that we're gonna have.
- 30:42
I have it in the system limited to two, because in this case, you don't need more than that. But if you wanna retrieve some information, if it's a product, you might have-- You don't wanna just, uh, bring top two products that's being there for the user because you might have the same brand, like twenty product or fifty
- 31:01
product of the same brand. If you don't let the window, your window to pull those information, those products are never gonna be sold and never gonna be shown in the chatbot.
- 31:11
So it's important how many you can fetch. Uh, and then the, the
- 31:19
more about the retrieval that when we're using the BM25 to get the exact keywords. So the previous slide was about to get the most closest with the cosine distance that we have.
- 31:32
Um, because when we upload the inform-- uh, information with the, uh, in our vector, in our, in our vector database, the information with the similar meanings are going to sit next to each other.
- 31:46
And then when we s- users search for something and ask for something, then, uh,
- 31:53
our informati-- uh, uh, and our s- uh, our search is gonna convert that, uh, question, that query, uh, in that database and go find the similar closest information to that question and will return it.
- 32:09
So that's the reason we can't just return one. We need to return more than those, so we have multiple answers and more closer answers. But here in the sparse retrieval, we need the exact answer, so we would need to, we would need to filter it based on the language, for example.
- 32:26
We need to based on the ID or exact product name, SKUs, or, uh, brand name or something. So that would be the,
- 32:37
the keyword, uh, retrieval, uh, for us. And then how we do the ranking based on, uh, uh, which is very important because when we have the both vector and the BM25, we wanna mix these and still bring the top, uh,
- 33:00
top two or top five relevant answers that we need, uh, for us. So more accurate. If we, if we bring too many, if we bring top twenty here and, uh, bring-- shows top twenty answers, it's gonna be quite confusing for the user.
- 33:16
So you just wanna have... make sure you bring in the right amount based on your
- 33:23
use case, bas-- uh, based on your use case.
- 33:27
So if you're having products, you would bring more, you retrieve more. If it's something medical, you retrieve less. Uh, you don't want
- 33:37
to-- information to be all over the place. You want to accurate, more accurate information because you would have more liability, obviously, on that. Um, another point here is,
- 33:51
um... So this is, uh, this is more about the
- 34:02
re-ranker. So when we have our re-ranker here, uh, that would also, um,
- 34:11
bring the informa-- uh, bring that, uh... And so after that score,
- 34:20
uh, how many of those information that shows by default. So I'm only showing two there. Um, so I've, I've, uh, briefly talked about the agent mode versus direct RAG.
- 34:32
The direct RAG is, is just, uh, following the... It's a fixed pipeline. It's not gonna go calling anything. You know, the, the path is q- uh, quite clear. You have embed, uh, embedded query, you have a hybrid retriever, and you just bring up the answer.
- 34:54
It's good if compliance is quite important for you. But agent mode, you have a few extra tools that gonna be called. So a search is a compare product. These can bring more information back.
- 35:08
It's less in your control. So, um, uh, that's quite important. And this takes longer because you have another agent that's going to work. So basically, it's getting this information, and then you have agent mode based on your request is going to call.
- 35:24
Um, so that would, that would take slight longer, slightly longer.
- 35:31
Um, about the, about the telemetry and application guidelines. So, um,
- 35:40
can use Langfuse for-- We integrated Langfuse, uh, in our function, so we can, uh, see what kinda answers, uh, we have. It's quite easy to... Everything is local. It's no outside calling or anything, so it's easy to track the information that I was chatting with the widget anonymous.
- 36:02
I wasn't logged in. The FAQ, FAQ chat, that's FAQ agent is name of the, uh, the whole projects that I'm running there. So each conversation has a ID. Uh, you, you can track if there was any maintenance for this weekend, and you can track the answer for it.
- 36:24
You can see what model I've been using. You can see I've returned the top two answers And, uh, the-- because it was top two, it's referenced twice on that image.
- 36:36
Um, agent mode was off. Uh, it was a chat. Two chunk of information was returned, and it took this long milliseconds. And, uh, yeah, that's basically the information we have.
- 36:53
If you have external models, you can even estimate the cost that you're spending. That's another thing you can, uh, you can use that for.
- 37:03
And, uh, if you have different sessions, different users, you can track the logins. So, um, that's give you a good insight of how users are treating, uh, uh, with your chats.
- 37:17
So, um, yeah. So as I said about the user sessions and IDs, um,
- 37:27
and then you would have the guardrails, um, as well. So something because of u- uh, using functions for it, we can, uh, have a better control over it. So if I ask this chatbot, how do I, uh, treat flu?
- 37:45
I have, uh, created a section, uh, to say without going to chat, but the point of lots of these guardrail is that the issue should stop before we, uh, the prompt injection or anything that we have, uh, uh, or a question that is we're not supposed to respond.
- 38:06
We shouldn't even send it to the LLM to generate anything. So before even in the code, I can have... So we have created a section here, uh, to say if there is a medical escalation, bring this medical message back.
- 38:22
Uh, and you can see how the system prompt is quite small because I'm basically prompting everything in the code. So, uh, although the system prompt looks quite small and just few sentences, uh, I'm actually prompting way more in the, uh, code
- 38:41
instead of prompt. We u- I'm using the code to say what do's and don'ts and, uh, another example is the medical escalation, and you can add more based on your, uh,
- 38:52
concerns you had for your product. So, um, yeah. So this is an example for that. Um, can be, uh, can needs to be expanded with different keywords, with different kind of sentences, should be cured and see how many times somebody could pass this or not.
- 39:07
But technically, that would be example of how we should block, or you have our guardrails should block any query that's not related. Um, can do the same with the injection and so on.
- 39:21
Um, what else do we have here? We talked about the Langfuse and, uh, we just need to create a-- generate a key, and you have it in your, uh, system.
- 39:35
It would co- uh, it would work. It's just, uh, a little bit about observability that I've talked about. Um, and then, um,
- 39:48
then the injection as well. So the tools for prompt injection, again, it shouldn't go to the LLM and it's being stopped there. It should be stopped before, before, uh, it's being sent there.
- 40:01
And, uh, yeah, we're using intent regex and term dictionaries and LLM classifier to stop any of these issues. And of course, we can expand it more. But the good thing about that, because it's just a code, it doesn't, it doesn't need a long prompt.
- 40:16
It doesn't need anything. Uh, uh, outcome would be obvious. You know what you are blocking and what you're not blocking. So it's not like a, a LLM that sometimes likes to listen to you as your, uh, as the instruction and sometimes just escape and does its own thing.
- 40:33
So part of that integrating all of these in your Python before sending it is that you can have rigid test and to say, "Okay, this is what I'm blocking, and this is why," and it passes all the tests.
- 40:46
So you can be more sure, um, about that.
- 40:52
Um, so yeah, and just a small slide about, uh,
- 40:59
we can just create your-- the-- and I'm having it as a, um, I'm having it here as a, as a ba- uh, as a widget. It could be also done here, um, instead of, um, in a bigger chat as well.
- 41:16
So another thing is you have to, in your chat as a part of the rules, you should say also if somebody understand the terms that they're using. So if not, the person would not be able to start the chat, you know.
- 41:28
And, uh, also I gave the, the widget capability if I want. So user, you can reset the consent as well. It's just for my demo purposes. So, and then the, the widget would show here.
- 41:46
Uh, if I say no, this wouldn't work. If I reset the consent, uh, if I say yes, then it will start. Um,
- 41:55
so yeah. So that's also about the widget. And it's because I'm using React, it can be used anything else, um, um, any other, uh, framework as well. I just use it for simplicity.
- 42:08
Um, here saying about, okay, the models I used. So I've used two models. Uh, I pulled the Qwen two point five. Um, actually, first I've had the seven billion, but it, um, took longer.
- 42:26
So for a smaller server, I've changed it to, um... Let me show you the admin here. I've changed it to, uh-
- 42:39
of 0.5. Otherwise, I have different options, so I was trying with different, uh, models. If you have a bigger model, it's just gonna take longer, and it might be too wordy.
- 42:50
Interesting, uh, finding from my side was you don't need the biggest model. You-- If you can vet your data before sending it to your LLM, a smaller mo- smallest model can sort you out and get a good, uh, uh, answer for you, and, uh, it would give you less hallucination.
- 43:10
It would, uh... It's safer because it's not gonna make up information. If it doesn't have information, it simply is gonna say, "No, I don't have that information." So that's another way of guardrailing, um, to not, uh, to not have that issue of your LLM randomly generating something.
- 43:29
Um, I have BGM3, uh, embedding model. So when I upload anything here that, uh, upload anything here, that embedding models basically, that embedding models do the embedding for me.
- 43:43
Uh, so ba- technically, you need two models only. The smallest, uh, chat model and, uh, one, one of the smallest, uh,
- 43:57
uh, one of the smallest model for, uh, embedding.
- 44:02
And then if you have it on Docker, one click, everything's gonna be ru- uh, running. You need your, uh, Python requirements to be installed, and then you have your React working on the front end.
- 44:17
And then that's all you need. Uh, one more thing I should be adding here, I haven't, that Langfuse also, um,
- 44:26
uh, should be added here. That's gonna be running in a different port. That's missing from here. So the key takeaway for, um, this is the structure markdown first. So we use DocLink for, uh, do the heavy job, and then we use the po- uh, Postgres vector database to just store everything.
- 44:47
Uh, didn't use any specific framework. Uh, we just used, uh, Ollama for our chat and no other, uh, paid framework anywhere else. We used Lang, uh, Langfuse for telemetry.
- 45:02
That was also free to use. And this can be, uh, escalated to different products as well. It can be document, catalog or from any database. Um, thank you very much, uh, for your time.
- 45:18
I would like to get connected with you and, uh, if you have any questions, if you have, uh, any con- uh, concern or something that you figure that would be, uh, better to integrate, please share your idea with me either in the comments or message me on LinkedIn.
- 45:36
Um, yeah. Thank you so much. Uh, thank you, AI Engineer, for the opportunity, and, uh, see you next time. Cheers. [outro music]