If we want them to do Knowledge Work, design them as Knowledge Agents — Benjamin Clavié, Mixedbread

An early retrieval-augmented generation system could search a document and let someone ask questions about a PDF. It provided an interface for asking questions, but Clavié considers it limited as a worker. Benjamin Clavié, who works on retrieval at Mixedbread, begins with the next step: agents that search, interpret results, call tools, and carry out programming tasks. Their success creates a tempting assumption that the same design will work across other professions.

Legal research, financial analysis, and finding medication information all require an agent to find knowledge and use it. Coding belongs to that larger category, too. Clavié's concern is that agent designers have learned disproportionately from one unusually convenient form of knowledge work, then tried to generalize its tools and task structure.

His working definition starts with the transformation the worker performs: information goes in; an actionable judgment or decision comes out. The input can be ambiguous and scattered. A lawyer must turn it into findings about a case and perhaps an argument on the client's behalf. Finding documents supplies the material. Understanding their consequences produces the work.

Clavié also offers a deliberately circular shorthand: if the problem needs search, it is a knowledge problem; knowledge problems need search. Lawyers, academics, actuaries, software engineers, and researchers have developed ways to organize this work. Those established workflows give agent designers more examples to learn from than programming alone.

Code supplies both clues and a prepared task

Code gives an agent several advantages at once:

• Stable references: Identifiers, files, and paths often persist long enough to connect a request to its implementation.
• Searchable structure: Keywords and method definitions give grep literal strings to find.
• Narrow assignments: A feature request or ticket usually arrives after a human has reduced a broader need to a particular change.

These advantages reinforce each other. A small task points toward a small part of a codebase, and that codebase contains names the agent can search.

Clavié makes the hidden preparation visible by imagining a much broader request: discover a new programming paradigm, then implement it in an application. Ordinary coding-agent requests rarely demand that kind of invention and decomposition. The human has already done some of the intellectual work before the agent starts searching.

A matching phrase can answer the wrong question

Now consider an agent searching a contract for thirty days. The phrase could describe a deadline, a grace period, or a retention rule. A search could even return a medication schedule from another domain. The words match in every case, but their role changes. The agent needs to establish what the duration governs and whether that meaning fits the question.

The search therefore starts from intent rather than a known identifier. A legal question might require finding international norms, identifying their conditions, and deciding whether those conditions hold in this case. The original request may name none of those dependencies. The agent must discover what it needs to know before it can retrieve enough evidence to answer. This is why Clavié treats non-code knowledge as especially contextual and meaning-driven.

Clavié turns to a broad history of knowledge work to explain two developments:

• Tools for finding information: Writing, library catalogues, bibliographies, library classification, and search engines make growing collections easier to navigate.
• Organizations for dividing work: Reliance on one gifted expert gives way to institutions such as monasteries and universities, bureaucracies, and specialized firms.

The examples illustrate two responses to the same pressure: there is more knowledge than one person can conveniently find and use alone.

A hospital makes specialization concrete. Doctors, senior doctors, nurse practitioners, nurses, and healthcare assistants handle different levels of tasks. The organization improves the allocation of work alongside whatever improvements its tools provide. Expecting one expert to perform every step would discard that organizational advantage.

The two developments form one feedback loop. New knowledge demands better tools; better tools allow new workflows and roles. Workers must learn those tools before the new workflows become effective. Someone accustomed to visiting a library needs to understand that a search engine accepts a typed query without a physical visit. More efficient workers then create more knowledge, starting the cycle again.

Cheap search changes which questions are worth asking

A tool can make a task economically practical even when the task was already possible. Clavié imagines finding a manuscript in an uncatalogued library taking two or three weeks, compared with ten minutes using a catalogue. These are illustrative timings: their point is that expensive retrieval reserves information for urgent questions, while cheap retrieval lets people investigate smaller uncertainties.

Maps provide another example: information about where to go changes the practicality of exploration. Multimodal search makes the same argument for archives containing millions of PDFs. A collection people would otherwise leave untouched can support useful work once finding evidence costs less. Tool quality changes the set of workflows people are willing to adopt.

Clavié next compares retrieval tools on a bounded deep-research benchmark: specific questions over a large text collection. The leaderboard includes two BM25 configurations, optimized and unoptimized. BM25 performs lexical search—retrieval based on the words in queries and documents—but the method's name does not specify a single implementation or quality level. A poorly tuned baseline cannot tell you how much the method can accomplish.

The badly optimized tool reaches about 60% accuracy. Clavié considers that too unreliable for sustained use: if someone repeatedly needs to check or redo the work, doing it themselves becomes attractive. Better retrieval moves accuracy through the seventies and eighties toward 90%. Near the top, he treats small score differences as run variance, so another fraction of an accuracy point is less persuasive than a clear efficiency gain.

Similar accuracy can require much less searching

The result Clavié emphasizes is 90.2% accuracy with 20% fewer tool calls. Finding useful evidence sooner means fewer rounds of searching and processing results. He connects that to lower token use and resource costs, then contrasts a workflow requiring 25 calls with one requiring 8. Efficiency matters because the cheaper workflow is easier to keep using.

The cost claims need their conditions kept attached. Clavié also describes spending about 5% of the unoptimized baseline's cost, but the call counts alone do not establish that ratio. Nor does 20% fewer calls necessarily mean 20% lower total cost: calls and model turns can consume different amounts of work. The practical decision is to compare answer quality with the effort needed to achieve it, rather than choose a tool from accuracy alone.

A bounded benchmark still understates the difficulty of an open-ended client problem. Clavié describes a client bringing a situation to a law-firm partner. The partner identifies the facts, likely consequences, and areas of law that need investigation. Before anyone searches, someone must convert the situation into researchable questions.

Paralegals and assistants investigate those questions using tools they have learned to use. They return memos and notes to the partner. The partner may research a clarification, but mostly relies on those findings to assemble the response. The memo is an important intermediate product: it gives the person responsible for the answer organized research to reason over.

This returns to the coding-ticket comparison. The programmer often performs the framing step before handing work to an agent. A client with a broad problem may provide only the situation and desired outcome. Clavié argues that a knowledge agent must therefore own the decomposition step and delegate research to subagents. Simply giving the answering agent a search tool leaves the organization of that work unresolved.

When the tool limits even a human

A PDF-based enterprise benchmark provides both PDFs and OCR versions: text extracted from document images through optical character recognition. With BM25, Clavié reports that a human and a model reach the same performance ceiling, even when the human can continue searching. He interprets that equality as a retrieval limitation: the tool fails to expose the needed evidence. It does not establish that the model reasons as well as the human.

Multimodal retrieval changes what evidence is available. Mixedbread's search tool can work with the PDF's visual content, including tables, rather than depending entirely on OCR text. Clavié reports a large accuracy improvement from this change. The causal distinction matters: more thinking or more searches cannot reliably recover information that the representation and retrieval process fail to supply.

The better tool still leaves the agent at a reported 88.9% accuracy, compared with a 99.4% human result using oracle evidence—the correct documents supplied for each question. That condition matters: the human comparator is relieved of retrieval misses. It is different from the BM25-constrained human discussed earlier. Mixedbread's separate benchmark account identifies the 99.4% result as human performance with an oracle retriever and lists the human using BM25 at 82.2%; its other configurations should not be substituted for the recording's reported 88.9%.

The agent can inspect results and continue over multiple turns; Clavié recalls a budget of about ten turns. Better evidence and an iterative loop help, but the remaining difference from the human oracle-evidence result does not isolate reasoning ability. The agent still has to find and organize its evidence, while the oracle comparator already has the right documents.

Give the answering agent a research product

Mixedbread's search-agent approach applies the earlier law-firm pattern. The main agent breaks the question into important aspects and writes queries for them. Searcher agents investigate those aspects independently, find results, and return short memos. The main agent then answers from the organized findings. Research and synthesis have separate places in the workflow.

Clavié reports a 3.5-percentage-point accuracy gain. He describes its significance through the oracle gap: the distance between a search system's result and the result obtainable with ideal evidence. The useful question is how much performance retrieval and research organization leave unused, rather than how impressive the raw score looks.

Clavié rounds the gap from about ten points before searcher agents to about six afterward, describing roughly a 40% reduction. Using the recording's stated 88.9% agent score, 99.4% human oracle-evidence score, and 3.5-point gain gives a 10.5-point initial gap, with one-third closed by the gain. His rounded 40% description should therefore stay approximate. This is a reduction in the gap to an ideal-evidence comparator, rather than a controlled measure of human versus agent research ability. The experiment still shows additional improvement from organizing research after improving retrieval.

Clavié closes with a practical reason to study librarians, paralegals, legal firms, and medical organizations: humans have already developed ways to divide knowledge work. The benchmark examples show how to apply that lesson. Improve a tool when its limitations impose a ceiling; change the workflow when better evidence still leaves performance unused. Tools should serve an observed need.

An available tool is not automatically a usable tool

Tools and agents need to be designed together. Clavié observes that agents can default to grep-style queries because grep appears widely in training material. That familiarity does not make the query appropriate for every collection. A model or harness needs to understand the available search primitives:

• Grep: Find literal patterns in searchable text, including the stable names common in code.
• BM25: Perform lexical retrieval over an indexed text representation.
• Semantic search: Formulate a meaning-based query when literal clues do not adequately express the information need.

The agent must know when to use each primitive and how to ask it a useful question.

PDFs make the limitation concrete. Searching extracted text with BM25 is possible—the earlier benchmark does exactly that—but it offers different access from multimodal retrieval of the document's visual content. An agent looking for evidence in a table needs tools and queries suited to that evidence. Familiar text-search habits should not determine what the system can see.

A larger window still needs someone to choose what belongs in it

Context is finite and costs money. Clavié imagines a hundred-million-token window to emphasize that even a dramatic expansion would leave the problem of organizing legal knowledge across jurisdictions and specialties. The example is a scale argument, rather than a measured inventory of legal text. More capacity does not determine which facts matter or which dependencies the agent must investigate.

His proposed division of work gives those decisions explicit places. The main agent decomposes the task; searchers gather and organize relevant knowledge; memos bring the findings back for synthesis. Increasing context can supply more room, but this architecture decides what work happens before information reaches the answering agent. That is the design problem knowledge agents must solve.