VoiceVision RAG - Integrating Visual Document Intelligence with Voice Response

When a document contains text, tables, and images, what should a retrieval system store—and what evidence should it give the model that answers the question? Suman Debnath’s workshop works through that problem from retrieval mechanics to a spoken response. The retrieval workflow came first; the agent layer was added afterward using Strands Agents, a lightweight, open-source framework that had recently launched at the time of the recording.

The workshop is organized around a notebook rather than a separate slide deck. Debnath, introducing himself as an AWS principal machine learning advocate focused on language, RAG, and fine-tuning, points participants to section eight of his repository, Agentic Voice-based RAG. It contains equivalent notebooks with and without saved outputs: one for working through the cells independently, the other for comparing expected results. Explanatory diagrams live inside the notebook. Participants were also offered AWS credits through a survey or directly from the speaker.

Start with extraction. A custom script, document-processing framework, or OCR service such as Textract separates the document into images, tables, and text. Metadata preserves each entity’s source page. In the first architecture, a multimodal embedding model encodes those entities into vectors, and a vector database stores them. The same embedding model encodes the user’s question. Search returns relevant entities, which join the original question in a multimodal LLM’s input. The answer model must accept images because the retrieved evidence may include them.

A second architecture inserts summarization before embedding. Image captioning produces a textual description of an image; tables and text receive their own summaries. A text embedding model indexes those summaries. At query time, the system retrieves summary text and sends it to a text-only LLM. The retrieval representation and the answering evidence are both summaries.

The third architecture retains summary-based search but changes the handoff. A dictionary maps each summary to its original image, table, or text. After retrieval, the system resolves those references and supplies the original entities to a multimodal LLM.

What you search need not be what you ask the generator to read. In the third option, summaries locate evidence; they do not replace that evidence in the final prompt.

Two implementation details matter here. First, preprocessing decides how an object enters the pipeline: if a table is extracted as an image, downstream components receive an image. The embedding model does not retroactively correct that classification. Second, a single model icon in an architecture diagram may conceal two sequential operations—one model creates a summary, and another embeds it. Keeping those roles separate makes the data flow easier to reason about.

These pipelines become harder to apply when the PDF is itself a collection of photographs or scans. Debnath gives examples of highway number-plate captures and photographed government forms later assembled into PDFs. Extraction can work on such material, but its suitability depends on the documents. If an existing pipeline already answers the relevant questions well, his advice is to keep it.

The alternative is to preserve the page as a visual object. Separating its text, tables, and illustrations can lose the relationships that make them meaningful together. Debnath compares this to sending members of a family to different places and asking someone else to recognize that they belong together. An audience member identifies the technical issue directly: the relationships between objects on the PDF matter.

IKEA assembly instructions make the limitation concrete. A pictorial person assembling parts may communicate an entire step with little or no text. OCR alone cannot recover the action merely by recognizing characters. A useful retrieval representation must capture enough visual meaning to find the page that answers the question.

To introduce ColPali, Debnath uses a book-navigation analogy. Given an unfamiliar book and a question, a reader first looks for its structure—index, appendix, chapters—then reads the relevant portion. He illustrates this with a deep-learning book he recommends from Professor Simon. The retrieval objective is similar: locate the useful evidence before composing an answer.

Debnath places ColPali’s introduction around July 2024. Its starting representation is simple: a hundred-page PDF becomes a hundred page images. There is no requirement to extract text, tables, and illustrations into separate retrieval entities. Each page is divided internally into patches and represented by multiple vectors. The workshop’s small diagram uses fifteen patches, giving fifteen vectors per page and 150 across ten such pages. Its discussion of a 32 × 32 grid refers to a much denser representation: 1,024 image patches, as described in the paper, rather than the teaching diagram’s fifteen.

The background intuition begins with separate capabilities: transformer-based language models process text, while image models historically used architectures such as CNNs. Video adds a time dimension to visual inputs. Combining vision and language requires more than putting their outputs next to each other. A photograph of a dog and the sentence describing a dog sitting in a field should have related representations. Two independently trained encoders could each emit ten numbers and still place those representations in unrelated spaces. Equal vector length is necessary for a direct comparison, but it does not establish semantic alignment.

Debnath explains alignment through contrastive learning. Training includes matched image–text pairs and unrelated pairs. A loss penalizes a matching pair whose representations remain far apart, and backpropagation updates the model weights. The purpose is to make semantic correspondence visible in the learned representation. This is background intuition for cross-modal learning, rather than a complete specification of ColPali’s training objective.

A prompting digression follows the same practical concern with specificity. Listing foods someone dislikes does not tell you what dinner to prepare. Likewise, asking for a dog sitting on a chair specifies a scene; saying the dog should not sit on the floor leaves a desk, chair, and many other possibilities open. Debnath recommends stating the desired result explicitly. He also attributes the advice partly to a shortage of negative training examples, but the concrete lesson here is that excluding one outcome does not identify the intended one.

The encoding path combines a vision encoder, projection layers, and a transformer. Projection makes representations compatible with the next stage and, ultimately, produces comparable retrieval vectors. Crucially, the application supplies the full page image; it does not call the model separately on manually cropped squares. ColPali’s patch representations are contextualized through the language model, so they should not be understood as independent crop embeddings.

After training, document ingestion runs without a question. It creates and stores the page representations in advance. At query time, the demonstrated workflow accepts a text question, tokenizes and encodes it, and compares its representations with those stored for the pages. ColPali’s output at this stage is a retrieval representation, not the final answer. The complete query diagram shows the tokenizer, text embeddings, transformer, projection, and final representation.

Both paths require preprocessing. PDF rendering tools may produce images with very different resolutions—Debnath contrasts 800 × 800 with 50 × 50—so the processor prepares inputs in the form expected by the model. Standardizing input shape is part of the encoding pipeline, though resizing cannot restore visual detail that was absent from a low-resolution source.

Consider the workshop’s simplified page with a 2 × 2 grid: four patch vectors. The question “What is AI?” is represented by three illustrative query tokens. Compare every query-token vector with every patch vector using a dot product. The result is a matrix with one row per query token and one column per page patch. For each row, keep the largest value: that token’s strongest match anywhere on the page. Sum those row maxima to obtain the page score. Different query tokens can find their strongest evidence in different regions.

For query-token vectors qᵢ and page vectors dⱼ, MaxSim is:

The notebook explanation presents an illustrative page-one score of 2.58. Repeat the calculation for the other pages, then rank them. Asking for the top five chunks now means retrieving five whole pages, even though patch-level interactions determine their scores.

The scoring operation can be expressed directly in Python with PyTorch:

import torch


def maxsim(query_vectors: torch.Tensor,
           page_vectors: torch.Tensor) -> torch.Tensor:
    # query_vectors: [query_tokens, dimensions]
    # page_vectors:  [page_tokens, dimensions]
    similarities = query_vectors @ page_vectors.T
    best_patch_per_token = similarities.max(dim=1).values
    return best_patch_per_token.sum()

The inputs are the retrieval vectors produced by the model. This is called late interaction because document vectors already exist before the query arrives; their detailed interaction with query tokens happens during scoring.

The database must support this multivector comparison. Debnath chooses Qdrant for its MaxSim support. He has not surveyed every database and mentions an OpenSearch extension only tentatively. The operative requirement is specific: store multiple vectors for a page and score their interaction with the query’s multiple vectors. Once the scores exist, selecting the highest-ranked pages completes retrieval; no agent is needed for this step.

The notebook imports the retrieval model and Qdrant client, with Qdrant running locally in Docker. Its dataset is chapter thirteen of a science textbook, chosen because it includes both diagrams without text and pages mixing illustrations with prose. Hugging Face authentication supplies access to the model download. Debnath recommends keeping the token in a .env file rather than embedding it in notebook code; he describes the displayed token as a dummy.

Device selection checks for GPU, Apple MPS, and CPU execution. The MacBook demonstration selects mps, visible in the notebook output. Debnath recommends accelerated hardware for this vision workload but notes that CPU execution is possible. Memory still constrains batching: he recounts an oversized run that repeatedly crashed and rebooted his laptop, eventually prompting an IT replacement before he recognized the workload as the cause.

The demonstrated checkpoint is ColPali v1.3. Load both the processor and model, and cache the downloaded files locally so subsequent runs do not download them again. The processor prepares the inputs; the model produces the embeddings.

The Qdrant container exposes a local port and persists vectors in a local storage directory. A client connects to the service, and the notebook creates the class-ten-science collection only if it does not already exist. The dashboard provides a way to inspect that collection as ingestion proceeds. Its vectors have 128 dimensions.

An audience question about increasing the dimension leads to a useful distinction. Debnath compares a richer representation with knowing more facts about a person: knowing only his employer and marital status does not tell you whether he plays cricket. But the collection dimension must match the embedding model’s output. For this checkpoint, 128 is the retrieval projection dimension; changing the database setting to 256 does not produce a richer embedding. A different dimension requires a compatible model representation as well as database support.

The collection also needs a multivector configuration with the MaxSim comparator. In Python, the essential configuration is:

from qdrant_client import QdrantClient, models

client = QdrantClient(host="localhost", port=6333)
collection_name = "class_ten_science"

if not client.collection_exists(collection_name):
    client.create_collection(
        collection_name=collection_name,
        vectors_config=models.VectorParams(
            size=128,
            distance=models.Distance.COSINE,
            multivector_config=models.MultiVectorConfig(
                comparator=models.MultiVectorComparator.MAX_SIM,
            ),
        ),
    )

The comparator implements the row-maximum aggregation needed to rank a page from its multiple vectors. Creating this collection prepares storage; it does not yet embed or insert any pages.

The ingestion function walks a directory of PDFs and renders each page as an image. Its bookkeeping list, described as all images, keeps the document ID, page number, and RGB image together; rendered data is stored in a local PDF-data directory. With one PDF, every record has document ID zero, while page numbers advance from zero. Keeping these identifiers attached to the image lets search results resolve back to the correct source page.

Debnath reports that batches of roughly ten or twelve images exhausted his laptop’s approximately 16 GB of memory. He recommends beginning with two or three images per batch on a laptop. For each batch, the processor standardizes the images, the model generates their multivector representations, and the notebook inserts them into the collection. The demonstration corpus contains ten pages. Each page’s vectors need to remain grouped with its page metadata so that retrieval returns pages rather than unrelated patch records.

The first concrete retrieval question is: What are the different trophic levels? Its path is now short:

1. Process and encode the text question.
2. Submit its token vectors to Qdrant.
3. Request five page results.
4. Resolve and display the returned page images.

The notebook’s visualization includes a page containing the trophic-level material. At this point, ColPali has finished its job. It has located evidence, not written an answer. The next component receives the original question and those page images.

The generator can be any suitable multimodal model. The notebook first offers a local option through Ollama, avoiding a requirement for AWS access. A helper converts images to Base64 for the demonstrated input format, then the generate call receives the images and the full question. It does not receive the query embedding: that representation was useful for search, while generation needs the question’s actual text. Debnath cautions that the local model’s response quality may differ.

The Bedrock path uses a different adapter to construct the multimodal request expected by the Converse API. It supplies the retrieved images, question, and model ID. The workshop uses Claude Sonnet 3.7 and mentions Claude Sonnet 4 as an alternative. These are the recording’s model choices; the important interface boundary is unchanged: retrieval returns page evidence, and a separate vision-capable generator reads it. The notebook frame shows the image-handling functions, model call, and a generated explanation organized around producers and consumers.

To make the workflow agentic, consolidate the retrieval cells into a function: accept a text query and return matched image paths. This preserves the existing retrieval implementation while giving the agent a clear callable capability. The function’s contract is deliberately narrow; the generator still handles the answer.

Strands follows a model-first design: give a model access to tools and let it select actions, rather than constructing extensive backstories and control scaffolding. Debnath points to the project site, launch material, and videos, then describes integrations with Bedrock, direct Anthropic access, and LiteLLM. His DNA metaphor gives the framework two strands: a model and its tools.

A separate demonstration makes that contract visible before applying it to visual retrieval. After installing the agent and tools packages, Debnath imports an agent, a Bedrock model, and built-in file-reading, file-writing, and speech tools. Ollama is also supported. He describes Claude 3.7 as the default in the version being demonstrated; defaults are version-dependent, so explicit model configuration keeps that choice clear. The agent receives a request to read a local textbook, summarize it, save the summary, and speak the result.

The resulting tool sequence reads the file, writes output, and invokes speech. The visible terminal marks file_read, file_write, and speak, while the explorer shows speech_output.mp3. The audible summary describes light entering through the cornea, the lens focusing it on the retina, the iris regulating incoming light, and accommodation changing focus. This demonstrates several separate capabilities coordinated by one model-driven agent.

The next demonstration moves a capability behind an MCP server. Debnath wraps Manim in a server and uses the Strands agent as its client. A request for a cubic-function scene leads to a tool call, code execution, and a generated mathematical animation reminiscent of 3Blue1Brown. The spoken request begins with a term of 2x³ but does not specify the complete expression. The architectural point is the same as with built-in tools: the agent can call an external capability without embedding that capability’s implementation in its own reasoning loop. A minimal agent uses the framework’s default model unless one is supplied explicitly.

Back in the textbook workflow, the imported tool decorator exposes the custom retrieval function to Strands. An image-reader tool handles the image input formatting previously performed by custom generation wrappers. The agent is configured with a model, a RAG system prompt identifying the available tools, and two capabilities: retrieve page paths and read the images. Asking the trophic-level question now delegates the orchestration to the agent.

Speech adds another tool to that sequence. The agent retrieves relevant pages, uses the image reader to supply visual evidence to the answer model, and invokes speak for the final response. Debnath asks for a natural female voice. After a delay, retrieval finishes and speech begins explaining trophic levels as feeding positions in a food chain and the flow of energy through an ecosystem. He suspects an internet delay, but does not establish its cause.

The live experiment then tests how much behavior can be controlled through the user request. Debnath removes the system prompt and asks for a male voice. While earlier speech continues, he suggests limiting answers to fifty or a hundred words to shorten playback, and points back to repository part three for the traditional image/table/text preprocessing architecture. A subsequent run produces another spoken answer without the earlier system prompt.

Producing speech and selecting the requested voice are separate outcomes. After another male-voice request, Debnath acknowledges that the tool is retaining its default female voice. His practical correction is to configure the speech tool’s specification or persona for more deterministic behavior, with the system prompt as another possible place to express the preference. A natural-language request does not by itself establish that a tool’s underlying configuration changed. The successful spoken answer remains useful, while the voice-selection experiment exposes a boundary between model instructions and tool settings.

Asked about production use, Debnath describes work with an unnamed insurer’s driver-license and insurance-policy images. OCR worked on that material, and he says ColPali also worked well. He characterizes embedding ingestion as heavy and query-time retrieval as fast in his experience, without giving measured quality, latency, corpus size, or deployment scale. Those missing quantities matter when moving from this local example to capacity planning.

The scaling discussion turns to indexing. A large collection need not require a naive scan over every stored vector. Debnath explains hierarchical nearest-neighbor navigation as moving from coarse search to closer neighbors, using tree pruning as an analogy. HNSW is a graph-based index, and that general intuition does not establish how this notebook’s multivector search scales. A page contributes many patch vectors, so the number of pages alone understates the storage and indexing workload.

Would visual retrieval replace traditional RAG? Debnath’s answer is no. Start with the least complicated effective architecture and evaluate it on the actual documents. If converting a PDF into text preserves what a reader needs to answer questions, rendering and embedding every page may add unnecessary work. If a reader needs to see the illustrations and embedded text together, page-image retrieval becomes more compelling. Many vectors per page increase the burden, though more efficient vision models could change the tradeoff.

An audience member asks whether nonoverlapping patches can cut a paragraph or table in an unhelpful place. Debnath answers with training intuition: the model has already encountered patch-based visual inputs during training, including pictures and document pages, and can learn relationships across them. Patching is inherited from vision models rather than invented for retrieval. Contextualized patch representations help explain why the model can use more than an isolated square, but the discussion does not measure patch-boundary failures or show that they cannot occur.

Debnath proposes fine-tuning or training on a small dataset while varying patch size, if resources permit. He has not performed that experiment, and his suggested starting value of four is not developed into a precise configuration. The more useful evaluation question is semantic specificity: if a document repeatedly mentions positional embeddings, can retrieval find the page that explains them? He uses the Attention Is All You Need paper as an example. MaxSim aggregates evidence across query tokens, rather than ranking solely from one token’s match to one patch; that mechanism motivates the desired behavior but does not guarantee it for every query.

A hybrid system is another possible response to mixed data. Debnath describes a teammate’s ongoing exploration prompted by a customer adding a substantially different dataset to an existing requirement. Some datasets use ColPali, others use traditional retrieval, and metadata helps route incoming questions to the appropriate retrieval space. Their experiment keeps one model per dataset. It does not switch between visual retrieval for the first five pages of one document and text retrieval for the next five. The unified interface sits above separate retrieval choices.

The textbook demonstration uses an existing checkpoint without Debnath performing task-specific fine-tuning. He treats fine-tuning as a future exercise, not a prerequisite for trying the model. For a more demanding visual evaluation, he returns to IKEA manuals: sparse pictorial instructions make it easier to see whether retrieval captures information that OCR text alone leaves out. The useful test is whether the returned page contains the visual evidence needed to answer, and whether the downstream model actually uses it.

For participants continuing with their own projects, Debnath closes with an offer to contact him on LinkedIn for AWS credits, including additional credits where needed.