AI Can't Make Up a Number It Can't Look Up - Vinoo Ganesh, Kepler

An analyst asks for Palantir’s 2023 revenue. The search system retrieves the company’s 2025 annual filing, which includes the earlier year’s figure. The model picks up 2025 revenue and links to the filing. A reviewer who opens the citation will find the right company and a genuine revenue number. Catching the error takes one more step. The reviewer has to check which year that number belongs to.

Vinoo Ganesh, Kepler’s CEO and co-founder and formerly head of business engineering at Citadel, uses this example to distinguish three ways a financial answer goes wrong. Retrieval can select the wrong document. Extraction can select the wrong fact inside a relevant document. And even with the right inputs, a model can make an arithmetic error. A document citation helps investigate the first two failures. It says little about whether the calculation was performed correctly.

Choosing the right source can itself require judgment. For a question about what a stock is worth, a firm might want its own internal analysis or the estimate of an analyst it particularly trusts. Ganesh sees room for a model to exercise judgment about where to look and how to approach a question. Once it chooses a source and a fact, though, ordinary code can supply the value and perform the arithmetic.

Kepler moves much of the extraction work to ingestion. Code processes unstructured documents, contextualizes individual facts and assigns them unique identifiers in a structured store. In the Palantir example, the useful unit is the revenue figure associated with a particular reporting year. An identifier for the filing as a whole would leave the model with the same choice between years every time someone asked the question.

At query time, the model plans the work. It chooses which source and entity to retrieve and refers to stored facts by ID. Code looks up the values, performs any required math and writes the numbers into the answer. This puts a substantial processing step up front. The payoff, Ganesh says, is that Kepler can reuse the extracted facts without paying the token cost of reading a 10-K to extract the same number at search time.

The distinction matters when diagnosing a failure. Looking up an existing ID returns the value stored under that ID. The model can still choose the ID for the wrong year, and a lookup will faithfully return that year’s revenue. If the ID is missing altogether, however, the lookup returns no value. The application now has a specific failure it can act on before showing a number to the analyst.

A fact lookup starts with a known identifier. Finding the right identifier still calls for search, especially when the evidence spans public filings, internal research and analyst commentary. Ganesh says that letting a model direct the whole search encourages it to pursue an early plausible hit. Exploring numerous sources through model calls is expensive, and that first hit may already look good enough to build an answer around.

Kepler combines approximate nearest-neighbor search with BM25 keyword retrieval to collect up to about 200 candidate documents. It combines their rankings using reciprocal rank fusion, narrowing the set to roughly 60. Cross-encoders then score the query together with individual document chunks. The ordering lets inexpensive retrieval methods cast a wide net and reserves the more expensive joint scoring for passages that survived the cut. The selected fact’s ID gives code a precise value to fetch once search has done its job.

A wide search has to stay inside the caller’s authorized corpus. Ganesh gives a concrete failure case. Omit the tenant filter and a Citadel user could search Millennium’s documents. Accept the caller’s assertion about which firm they represent and an impersonator could get the same result. Both failures happen before the model evaluates relevance. The system needs to establish whose credentials authorize the request and which documents that identity can retrieve.

The same question applies when code fetches a document by ID. A search filter offers little protection if a direct read can bypass it. Ganesh also calls out stale results and queries against an index that has yet to be populated. Those are different explanations for poor evidence: in one case an old document remains discoverable, and in the other the search may have had nothing to examine. A reviewer investigating a missing or surprising fact needs to know what the caller could access and what was actually indexed.

Kepler’s source IDs are hashes that point to stored facts. The application can check whether a reference resolves and expose that connection as a clickable chip on the number itself. Ganesh describes three possible outcomes for a citation. The system verifies it, corrects it during search or refuses it. For a calculated figure, the relevant evidence includes the source inputs and the arithmetic that produced the result. A link to a filing alone leaves the reader to reconstruct both.

Ganesh reports that refusals fell sharply in Kepler’s production data after the system required the model to resolve its references. He defines a refusal here as an inability to produce a number because the pointer could not be resolved. He gives no exact rate in the talk. He also says refusals persisted and that the model could still generate stray numbers during inference. Kepler therefore retains a final post-processing check on what reaches the reader.

That distinction gives an engineering team a practical vendor test. Start with one displayed number and click through. For an extraction, can you establish the metric, period and source? For a calculation, can you follow the inputs and reproduce the result? Then inspect what the application shows when the reference fails to resolve. The behavior at that point tells you whether the evidence actually controls what the analyst sees.