# AI Can't Make Up a Number It Can't Look Up - Vinoo Ganesh, Kepler

Vinoo Ganesh, Kepler

[Talk page](https://ai.engineer/talks/RdQCY_-SXbA-ai-cant-make-up-number-it-cant) · [Video](https://www.youtube.com/watch?v=RdQCY_-SXbA)

## Who Writes the Number? How Kepler Grounds Financial AI in Facts It Can Look Up

Vinoo Ganesh follows a financial figure from a company filing to an analyst’s screen, showing where a model can exercise judgment and where a database lookup, a calculation or a refusal should decide what the analyst sees.

### The part a competitor copied

[0:12](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=12s)

Three months before this appearance, Vinoo Ganesh, CEO and co-founder of Kepler, had argued at AI World’s Fair that institutions needed a way to verify AI-generated work. The day before this session, he had shared a stage with Jack, who had run Citadel’s biggest book and founded Candlestick Capital. Jack still distrusted AI. Meanwhile, by Ganesh’s count, roughly $270 million in venture capital had gone into formal verification layers in those three months.

Ganesh says a venture capitalist even passed Kepler’s pitch deck to a competitor, whose product appeared with the same tagline. What amused him was the part they copied. Kepler does run a verification pass on its output. But by the time that pass encounters a revenue figure, other parts of the system have already determined which documents the user can search, which fact the figure points to and which component gets to write the number. Those earlier decisions make the final check possible.

### An investment firm wants originality and an accountable set of numbers

[1:42](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=102s)

Ganesh often hears builders of retrieval-augmented generation systems predict that better models will eventually eliminate hallucinations. He sees a complication specific to investing. A firm earns alpha in part because its analysts can look at information available to other investors and arrive at a different judgment. That capacity to propose an unexpected explanation is valuable. The revenue figure an analyst uses to support it, on the other hand, has to survive review. And a model with access to public information alone misses the firm’s own research, which may be precisely where its distinctive judgment resides.

Firms were buying retrieval platforms, building their own systems and running agent pilots. One customer described itself to Ganesh that morning as AI-first on the strength of a single copilot agent. Yet fundamental investing had scarcely undergone the transformation those purchases suggested. Ganesh puts the obstacle in human terms. Before people will stand behind a piece of work, they need to know whom they can hold responsible when it goes wrong.

He cites figures of 80 percent of finance executives trusting numbers and 31 percent finding errors. The product feedback he recounts puts a finer point on the problem. A system demos well and behaves like a smart intern. Give it the right structured information and much of the answer comes back correctly. The reviewer still owns the last ten percent and needs some way to locate the mistake.

![Three review excerpts describe subtle errors and the work of auditing them. Below, accuracy is 90.8% with structured financial data and 19.8% with web search alone. A separate panel reports 46% as the best result for finding a 10-K line item.](https://ai.engineer/api/talk-essay-media/415595c6aa4ec6c8992cfa6333846e91b8c6c83337ce97b7610f0b9fbb4ca355)

500 reviews: nobody trusts the numbers.

### Three ways to get Palantir’s revenue wrong

[4:12](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=252s)

The first failure is choosing the wrong document. Even defining the right one takes judgment. If the question is what a stock will be worth, the firm might want its own internal analysis. It might instead want an estimate from an outside analyst whose work it particularly trusts. Both documents could discuss the correct stock and contain a plausible answer. Retrieval has to take account of whose assessment the user is asking for.

The second failure occurs inside a relevant source. Suppose the question concerns Palantir’s 2023 revenue and the system retrieves its 2025 Form 10-K. The filing includes the earlier year’s figure, so retrieving it is defensible. But the model can pick up the 2025 revenue and cite the filing. A reviewer following that citation will find Palantir, a genuine revenue figure and the correct document. To spot the error, the reviewer has to match the selected figure to its reporting year. A citation filter can check that association after generation, by which point the system has already produced the incorrect claim.

The third failure is arithmetic. A model can select the intended figures and still predict the wrong next token when asked to combine them. Ganesh’s rejoinder to the observation that deterministic models exist is that the CPU already knows how to add two numbers. Once the operands and the operation are known, code can perform the calculation directly. Better document search would do nothing to fix this particular error.

![Three columns connect each failure to its fix with a downward arrow. Retrieval finds the source, a filter checks copied numbers against their citations, and code performs the arithmetic.](https://ai.engineer/api/talk-essay-media/c690b42708595a7f021edb93da80802723f16f482feee1dca405dad770b35f12)

Wrong numbers come from three jobs we keep handing the model.

### Extract the fact once, then let the model plan with its ID

[5:42](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=342s)

Kepler pays the extraction cost before an analyst asks a question. A substantial preprocessing step takes unstructured documents, puts individual facts in context and gives them unique identifiers in a structured store. In the Palantir example, the unit worth identifying is revenue for a particular reporting year. An ID that names only the 10-K would leave every subsequent reader to choose between the two years again.

At query time the model acts as the analyst planning the work. It determines which source to consult, which entity to retrieve and what to do with the result. It refers to facts using pointers. Code retrieves the values and performs the math, and code writes the numbers that appear through this path. This division also changes the economics of repeated questions. Reading a 10-K with a model to extract a figure at search time consumes tokens each time. A fact extracted at ingestion is available for subsequent lookups.

The reader encounters this architecture in a small interface detail. A sentence describing how revenue changed from one amount to another has clickable chips on the figures. The analyst can select an amount and inspect how Kepler obtained it. The connection exists because the system used a reference to produce the figure in the first place. Kepler also runs a post-processing check, the part Ganesh says the competitor noticed.

![A five-step diagram traces an Apple revenue question across model and code lanes. The model plans and writes prose. Code retrieves sales, computes growth, and verifies the two clickable figures in the answer.](https://ai.engineer/api/talk-essay-media/82ad89bdd0dce74390d87d068304b28c502001c21d7bd1f48fca65eb662054e8)

Code writes every number and pointer. The model reads them.

### A portfolio manager’s research process suggests where search belongs

[7:12](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=432s)

As head of business engineering at Citadel, Ganesh worked across the firm on software that could help generate alpha. He saw talented portfolio managers pair their judgment with deterministic tools. A buy-side researcher might build a discounted cash flow model, or DCF, using data already in hand. For information the researcher lacks, the next step could be a GLG or Guidepoint call, finding an expert or speaking with management. Knowing what remains to be learned is part of the skill.

Kepler makes a similar distinction. It stores what it can represent deterministically. For material it cannot handle that way, it preprocesses the information and uses hybrid storage that includes turbopuffer alongside other mechanisms. Reranking helps it choose among possible sources. Once it has a pointer, fetching the associated item can be as ordinary as calling a get method. Run a model-led search ten times with the same query and its choices can vary. Fetch an existing key and the store returns the value under that key. A bad choice of key will still faithfully retrieve the wrong year.

![A table compares exact lookup for public filings with hybrid search for client documents. Rows show what each collection holds, how Kepler retrieves and cites a result, and how the data is organized.](https://ai.engineer/api/talk-essay-media/7df1517d1b6c9181642379531ee252937dd47a67facdb48732f4f0a44f9a237b)

Look up what is tagged. Search only what isn’t.

### From 200 candidate documents to individual passages

[9:12](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=552s)

Finding the key is a search problem. Even in a collection of SEC documents, the system has to decide both which source deserves attention and which candidates it can stop considering. Kepler starts with approximate nearest-neighbor search, or ANN, alongside BM25 keyword retrieval. The two methods give it a broad set of candidates, usually up to about 200 documents. Reciprocal rank fusion combines their rankings and narrows the set to roughly 60 documents.

Cross-encoders then score the query together with an individual document chunk. The unit of comparison has changed. A first pass can surface a filing about the right company. The later pass can examine how closely a particular passage bears on the question. This ordering reserves the more expensive joint scoring for material that survived the wider search.

Ganesh contrasts that with letting an LLM direct the entire investigation. Once the model finds a plausible first document, moving forward with it is cheap. Exploring many additional sources costs more even if one of them would better answer the question. Kepler builds breadth into retrieval before asking the system to narrow its attention.

### Scope the search and the direct read to the caller

[10:46](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=646s)

Broad retrieval immediately raises two implementation questions for client documents. Whose OAuth token authorizes the request, and what can that user read? Ganesh identifies five ways the document path can fail. First, omit the tenant filter and a Citadel user could search Millennium’s documents. Second, accept a caller’s claim about which firm they represent and an impersonator can query another firm’s research. Correctly ranking the resulting documents only makes the disclosure more effective.

Third, search can return out-of-date information. The caller may be entitled to see a document that has since been superseded. Fourth, a direct read can skip the rules that governed search. Fetching one document by its ID still requires an access decision. An allowlist applied only when assembling search results leaves that second path exposed. Fifth, a query can hit an index that has yet to be populated. An empty result in that case tells the analyst little about whether the firm possesses the information.

![A five-row table pairs each possible file leak with its safeguard, from a missing tenant filter to a query reaching a partially rebuilt index.](https://ai.engineer/api/talk-essay-media/37e296d06016f4d1a402f6a881c983cb022d61a377ba405a06f7b42a7fed205c)

Five ways one client could see another’s files, each closed in code.

These failures can look much the same from the answer screen. An expected fact is missing, or an unexpected one appears. To investigate, a builder needs to know who authorized the request, which documents were eligible to come back and what was actually indexed. For a firm whose internal analysis helps it generate alpha, constraining the corpus is part of answering the question correctly.

### A source hash gives verification something to check

[12:16](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=736s)

Kepler’s source IDs are hashes. Preprocessing creates the references, and the model can use one to obtain a fact if the referenced item exists in the database. Ganesh’s analogy is a HashMap. Ask for a key that has no entry and the lookup gives you nothing. That outcome matters because it turns an unsupported reference into a condition application code can detect before presenting a sourced figure.

Kepler gives a citation three possible outcomes. It can verify the citation against a stored fact and present the figure with its reference. It can correct the citation during search. Or it can refuse the figure when it cannot resolve the pointer. The guarantee applies to facts in the corpus. Whether a retrieved fact is the most relevant one for this investor still depends on the source and period choices made earlier.

![Three cards show what readers see for each citation outcome: a green $416,161M chip for an exact match, a pale chip marked “fixed” for a correction, and plain 99% text for a refused citation.](https://ai.engineer/api/talk-essay-media/867c014d4f7e90b908dc3981b23b86d23d677d0c79eb3de92a89dee9b12630ae)

Every citation ends up verified, corrected or refused.

Ganesh suggests the same design for legal precedents. Store the cases that exist and let the model search among them. An invented case name then has no stored precedent to point to. His concern is what happens to adoption when practitioners repeatedly encounter convincing inventions. In his assessment, much financial AI use had settled on formatting PowerPoint presentations, while a trustworthy cradle-to-grave DCF remained out of reach. He argues that firms would have adopted such a system quickly if it already worked.

### Production refusals fell, and generated numbers still needed a check

[15:48](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=948s)

In Kepler’s production data, a refusal meant that the model could not resolve a pointer and therefore could not produce the requested number. Ganesh reports that refusals dropped sharply once the model had to resolve its own references. He does not say how large the drop was in percentage terms.

Refusals persisted. Ganesh also acknowledges that a probabilistic model can generate a stray number during inference. A lookup can prevent a nonexistent key from supplying a value, but the model is still capable of emitting numeric text. Kepler keeps a final post-processing check for that reason. The structured retrieval path gives that check something concrete against which to test the output.

### Evaluate a vendor by following one displayed number

[16:18](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=978s)

Ganesh’s proposed vendor test starts at the number an analyst is expected to trust. Click it. If it was extracted, follow it to the source fact and check that the metric and period match the claim. If it was calculated, inspect the source inputs and the arithmetic. A figure can involve both operations, in which case the reader needs to follow the extraction into the calculation. A document link attached after the answer was written leaves that work to the analyst.

Then ask how the application enforces what the interface appears to promise. Does the model reference facts through pointers? Is every citation checked? Is arithmetic checked against its sources? What keeps a client file from reaching the wrong customer? Ganesh saves the most revealing question for last. What happens to a number that came from nowhere? A database lookup returns nothing when the requested value is absent. An analyst should be able to see that refusal just as clearly as the source of a verified figure. When Ganesh says AI is “out of the search phase,” this is the infrastructure he is asking builders to deliver: a way to carry either outcome from the database to the person making the decision.

![A table pairs five vendor questions with Kepler’s answers about pointers, citations, arithmetic, file access, and unretrieved numbers. The highlighted final row reports that 12.8% of final-answer numbers were uncited.](https://ai.engineer/api/talk-essay-media/57c3591559feb364cf46ac9472b384e96e34d5ac52217df9e937a864a498afd4)

Five questions to ask before you trust an AI number.

## Key takeaways

- Diagnose a wrong number at three separate points: source selection, fact selection within that source and arithmetic. The Palantir example can pass a document-level citation check while failing the reporting-year check.
- Give reusable facts identifiers during ingestion. Let the model choose what to retrieve, then use code to fetch values and compute results.
- Apply the caller’s document permissions during both candidate search and direct reads by ID. Check index freshness and population when investigating surprising results.
- Test a displayed figure in both directions. Follow a valid reference through extraction and calculation, then inspect what the user sees when a reference cannot resolve.

## Slides

### “Honestly, I estimated those.”

[1:43](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=103s) [A chat excerpt shows a model admitting it estimated share counts. Underlined passages highlight the admission and the difference between its estimate and the figures in the 10-K.](https://assets.ai.engineer/sites/aiecode2025/sha256/304d156d48c5896d6852254eb06a0c64497a28f55f822946a510d43834661b04.webp)

“Honestly, I estimated those.”

A frontier model, asked where the share counts came from in the dilution model it had just built from a 10-K.

Its first answer had looked perfect.

1 in 3 analyst tasks the best model gets wrong. It never says which.

Source: Vals AI, Finance Agent v2 leaderboard, Oct 12, 2026.

where did you get your RSU and PSU number

Honestly, I estimated those — I couldn't get the RSU/PSU activity table to render from the SEC filing either. Same problem as the options table. Do you have that section of the 10-K as well? If you can share it I'll plug in the actual numbers.

You're right to call that out — my estimates were wrong. The actual 10-K data is right here:

Options outstanding as of December 31, 2025: 127,679,804 at a weighted-average exercise price of $36.07

I had estimated 106M at $42.90 because I couldn't get the stock option activity table to render from the SEC HTML extraction, so I was backing into it from the EPS dilutive effect. The actual numbers are meaningfully different on both counts — more options outstanding, at a lower exercise price. That means more dilution than my initial spreadsheet showed.

Let me rebuild the Excel with the correct data from the filing.

Kepler
2

### Every firm is trying. It works until numbers matter.

[2:43](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=163s) [Four rows pair firms’ attempts to use AI with the problems they encountered. Two survey figures below describe unchecked numbers in reports and errors that reached investors or the board.](https://assets.ai.engineer/sites/aiecode2025/sha256/f1dc2dd712fa85d873289dd478780ff66f1c6f3459eb07f809a47bb5fa3a5576.webp)

THE PROBLEM
Every firm is trying. It works until numbers matter.

We rolled out a chat assistant.
Great for emails.
Nobody trusts it with a number.

We connected it to our files.
It read 20 files.
The answer was in file 4,312.

We ran an agent pilot.
It ran 40 steps.
Step 3 was wrong. Nobody noticed.

We built our own.
New models ship every week. Workflows break.
We patch instead of building.

88% of finance execs trust AI numbers in reports without checking them.
31% found AI errors had already reached investors or the board.

Source: Workiva, 2026 MidYear Executive Benchmark Survey.

Kepler
3

### 500 reviews: nobody trusts the numbers.

[3:33](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=213s) [Three review excerpts describe subtle errors and the work of auditing them. Below, accuracy is 90.8% with structured financial data and 19.8% with web search alone. A separate panel reports 46% as the best result for finding a 10-K line item.](https://assets.ai.engineer/sites/aiecode2025/sha256/415595c6aa4ec6c8992cfa6333846e91b8c6c83337ce97b7610f0b9fbb4ca355.webp)

THE PROBLEM
500 reviews: nobody trusts the numbers.

“[It] demos like a dream and delivers like an intern on day one.”

“The errors were subtle enough to be dangerous — slightly incorrect line items, all adding up to correct subtotals.”

“Fixing this would require careful cell-by-cell auditing that takes longer than just inputting the numbers yourself.”

The model isn’t the problem. What it can see is.

90.8%
right with structured financial data

19.8%
right with web search alone

Source: Daloopa, FinRetrieval benchmark (Claude Opus accuracy), Jan 2026.

46%
is the best a frontier model does at finding the right 10-K line item.

The wrong picks look just as plausible.
Range: 38–46%.

Source: Kepler line-item extraction eval, reported in Anthropic customer profile, May 2026.

Kepler
4

### Wrong numbers come from three jobs we keep handing the model.

[4:16](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=256s) [Three columns connect each failure to its fix with a downward arrow. Retrieval finds the source, a filter checks copied numbers against their citations, and code performs the arithmetic.](https://assets.ai.engineer/sites/aiecode2025/sha256/c690b42708595a7f021edb93da80802723f16f482feee1dca405dad770b35f12.webp)

WHY IT HAPPENS
Wrong numbers come from three jobs we keep handing the model.

Find the source, copy the number, do the math. A wrong number looks exactly like a right one, so Kepler takes all three jobs away from the model.

FAILURE 1
Wrong source
Retrieval missed the document, so the model filled the gap with a number that sounds right.

FIX
Retrieval finds the source
Filings are read as tagged facts and client files are searched in turbopuffer. 3.0M filing facts are citable.

FAILURE 2
Right source, wrong number
The document was found, but the scale ($M vs $B), the unit ($ vs shares) or the copying slipped.

FIX
A filter checks every citation
Each cited number is compared with its source as the answer streams. 32,128 were corrected or refused.

FAILURE 3
Mental math
Growth rates, margins and multiples were worked out in the model’s head.

FIX
Code does the math
Compute tools do the arithmetic and register each result. 32.7% of all citations are calculations.

Source: Kepler production event logs, Apr–Jul 2026.

Kepler
5

### Code writes every number and pointer. The model reads them.

[5:47](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=347s) [A five-step diagram traces an Apple revenue question across model and code lanes. The model plans and writes prose. Code retrieves sales, computes growth, and verifies the two clickable figures in the answer.](https://assets.ai.engineer/sites/aiecode2025/sha256/82ad89bdd0dce74390d87d068304b28c502001c21d7bd1f48fca65eb662054e8.webp)

HOW KEPLER WORKS
Code writes every number and pointer. The model reads them.

“What was Apple’s FY2025 revenue, and how fast did it grow?” Every step sits in one of two lanes, and only code touches a number.

MODEL
Language only

CODE
Every number

1 Plan
Needs FY25 and FY24 revenue, then percent change.

2 Retrieve
Tagged 10-K fact: Total net sales 416,161 ($M).

3 Compute
Percent change = 6.4%, stored under a hash-derived id.

4 Read
The model reads both pointers into its prose. It types no number.

5 Verify
Filter matches each marker to its registered value.

READER SEES
Revenue was $416,161M up 6.4% from FY2024. Click a chip to open the filing line.

Kepler
6

### Look up what is tagged. Search only what isn’t.

[7:35](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=455s) [A table compares exact lookup for public filings with hybrid search for client documents. Rows show what each collection holds, how Kepler retrieves and cites a result, and how the data is organized.](https://assets.ai.engineer/sites/aiecode2025/sha256/7df1517d1b6c9181642379531ee252937dd47a67facdb48732f4f0a44f9a237b.webp)

RETRIEVAL · FINDING THE SOURCE
Look up what is tagged. Search only what isn’t.

Job 1, finding the source. Kepler picks the path by how the data is shaped, and ranks only where there is nothing exact to look up.

PUBLIC FILINGS · LOOKUP	CLIENT DOCUMENTS · SEARCH

What it holds
10-Ks, 10-Qs, transcripts, ownership filings (Form 4, 13F, 13D/G)	Decks, memos and spreadsheets from SharePoint, Box, Google Drive, Egnyte, OneDrive, Outlook and uploads

How it’s found
Exact lookup by company, line item and period. Nothing is ranked.	Hybrid search in turbopuffer: vectors plus BM25 in the client’s own namespace, then a rerank

What gets cited
The tagged fact: value, unit, scale, period. Total net sales 416,161, $M, FY25.	A chunk anchor: the spreadsheet cell, table row or verbatim quote

How much
3.0M filing facts available as anchors	One namespace per organization and one per user

Why it matters: a lookup can’t return the wrong company’s revenue. Search can miss, so it gets the most engineering.

Kepler
7

### Each search pulls up to 200 candidates and hands the model 10.

[9:24](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=564s) [Four narrowing boxes show the retrieval stages: up to 200 candidates, 100 after rank fusion, 20 documents, and 10 results after reranking. Each box aligns with its stage description.](https://assets.ai.engineer/sites/aiecode2025/sha256/001ef323dca7f2054af40b25e44c0814c8c37cd1d194e8f385964ef33d84e6d3.webp)

RETRIEVAL · FINDING THE SOURCE
Each search pulls up to 200 candidates and hands the model 10.
Each stage throws candidates away, so the first stage fetches wide on purpose. Missing the document costs the whole answer.

Up to 200
ANN + BM25 in turbopuffer
Both run at once, 100 each (3× top_k). zembed-1 embeds the query first.

100
RRF fusion, k = 60
The two ranked lists merge by rank, not raw score.

20 docs
Best chunk first
At most 5 chunks per document, so one file can’t crowd out the rest.

10
zerank-2 cross-encoder
Reads query and chunk together and reorders. Up to 25 when asked.

About 4.5 s per search: embed 0.2 s, turbopuffer + Postgres re-check ~2.0 s, rerank ~2.3 s.
Lesson: one wide rerank beat several narrow ones, and a bigger pool is the cheapest recall gain.
Source: Kepler engineering measurements.

Kepler
8

### Five ways one client could see another’s files, each closed in code.

[10:44](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=644s) [A five-row table pairs each possible file leak with its safeguard, from a missing tenant filter to a query reaching a partially rebuilt index.](https://assets.ai.engineer/sites/aiecode2025/sha256/37e296d06016f4d1a402f6a881c983cb022d61a377ba405a06f7b42a7fed205c.webp)

RETRIEVAL · FINDING THE SOURCE
Five ways one client could see another’s files, each closed in code.
Client documents sit in a shared search index, so we listed the ways a leak could happen and closed each one in code.

HOW A LEAK COULD HAPPEN	WHAT CLOSES IT
1  A developer forgets the tenant filter
One filter builder adds it to every query. Org namespaces carry an org_id predicate; a personal namespace is itself the boundary.

2  A caller claims to be another firm
Org and user come from the signed token and are stored on the chat. Only auth code can create a tenant scope.

3  Search returns a deleted or unowned file
Every hit is re-checked in Postgres (tenant, ownership, not deleted) before the model sees it.

4  A direct read skips the search rules
Search and read share one allow-list, so a read can’t return what a search would hide.

5  A query lands on a half-built index
Index rebuilds go v1 → v2 with an atomic flip, so queries only ever see a complete index.

Kepler
9

### Code writes the pointers. The model only reads them.

[12:24](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=744s) [An arrow connects the model’s bracketed citations to the reader’s version of an Apple revenue sentence. The filter rewrites $416.2B as “$416,161M · fixed” and displays 6.4% in a green chip.](https://assets.ai.engineer/sites/aiecode2025/sha256/35b48a7fc05077395d96c985dcaf3450dc292c5a03972e80a7a5b6ba040df745.webp)

VERIFICATION · CHECKING EVERY NUMBER
Code writes the pointers. The model only reads them.
Job 2, copying the number. Tools write a pointer to each value, and a filter checks every one the model reads back.

WHAT THE MODEL STREAMS
Apple’s FY2025 revenue was [src-7f3a : $416.2B | a-12], up [calc-… : 6.4% | r-1]
↓ citation filter: hold, look up, compare
WHAT THE READER SEES
Apple’s FY2025 revenue was $416,161M · fixed, up 6.4%

src-7f3a
Source id. A tool registered it; ids are hash-derived, so the model can’t invent one.

$416.2B
Display. What the model wants to show. Here it was rewritten to the filing’s exact form.

a-12
Anchor. The exact fact, cell, row or quote inside the source.

Match rule: within half a unit of the last digit shown, capped at 5%. $2B accepts 2.04B, rejects 2.49B. $ vs shares always fails.

Kepler
10

### Every citation ends up verified, corrected or refused.

[13:26](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=806s) [Three cards show what readers see for each citation outcome: a green $416,161M chip for an exact match, a pale chip marked “fixed” for a correction, and plain 99% text for a refused citation.](https://assets.ai.engineer/sites/aiecode2025/sha256/867c014d4f7e90b908dc3981b23b86d23d677d0c79eb3de92a89dee9b12630ae.webp)

VERIFICATION · CHECKING EVERY NUMBER
Every citation ends up verified, corrected or refused.
The filter always tries to repair a citation before giving up. Here is one example of each outcome.

VERIFIED
Exact match
The model cites $416,161M and the filing fact says 416,161 ($M).
READER SEES
$416,161M
Shown as written, as a green chip.

CORRECTED
Right number, wrong form
The model cites $416.2B. The value matches, but the filing states it in millions.
READER SEES
$416,161M · fixed
Rewritten to the source’s form. Also fixes a paraphrased quote or an anchor pointing at the wrong nearby fact.

REFUSED
No honest match
The model cites 99% net margin. Compute registered 26.9%.
READER SEES
99%
Chip removed, number left as plain text, reason logged: value mismatch.

Apr–Jul 2026: 21,147 corrected · 10,981 refused. The filter fixes about two citations for every one it removes.
Source: Kepler production event logs, Apr–Jul 2026.

Kepler
11

### Five questions to ask before you trust an AI number.

[16:53](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=1013s) [A table pairs five vendor questions with Kepler’s answers about pointers, citations, arithmetic, file access, and unretrieved numbers. The highlighted final row reports that 12.8% of final-answer numbers were uncited.](https://assets.ai.engineer/sites/aiecode2025/sha256/57c3591559feb364cf46ac9472b384e96e34d5ac52217df9e937a864a498afd4.webp)

TAKEAWAYS
Five questions to ask before you trust an AI number.
You can use these on any AI tool, including ours. The last row is where our own guarantee stops.

ASK THE VENDOR	KEPLER’S ANSWER
1  Does the model write pointers, or only read what tools wrote?
Reads only. Tools write every pointer.

2  Is every citation checked against its source before I see it?
Yes, in the stream: 21,147 corrected and 10,981 refused.

3  Who does the arithmetic and lays out the tables?
Code. 32.7% of all citations are calculations.

4  What stops my files reaching another client?
Typed tenant scopes, plus a Postgres re-check on every hit.

5  What happens to a number nobody retrieved?
Not caught by verification. 12.8% of final-answer numbers were uncited; we log them and measure recall.

Source: Kepler production event logs, Apr–Jul 2026. Uncited: 14,347 of 112,229 numbers in audited final answers.

Kepler
13

## Full transcript

[0:12](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=12s) Hi, everyone. My name is Vinu Ganesan. Um, excited to be here today. Uh, I'm the CEO and co-founder of Kepler. Um, this is gonna be kind of an interesting talk because it comes from what happened three months ago. So three months ago, uh, I think there's a little bit of an overlap, but I gave a talk at AI World Fair, uh, really around verification, um, and why that m- most institutions need verification in some form of the trust ecosystem of AI. Well,

[0:42](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=42s) yesterday, most of you saw Jack and I were on stage talking about why Jack, who ran Citadel's biggest book and started Candlestick Capital, still doesn't trust AI. Since that three-month period, there's been about a hund- two hundred and seventy million dollars of VC money that got poured into formal verification layers. A VC actually took Kepler's own pitch deck and gave it to a competitor and produced something that has our same tagline. I'm not bitter, just a little. But the

[1:12](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=72s) funny thing is they copied the wrong part. And so what I wanna do is actually give you, the practitioners today, you can't just tack verification on at the end. And so, uh, when I gave that chat or that talk a few, um, months ago, uh, Anthropic had just published an article on how Kepler does what it does and why it's really powerful and, and really unique. So what I wanna do is get into the technical side of how we do the thing that we do, and ultimately, this is driven by one very simple insight: Wall

[1:42](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=102s) Street still doesn't trust AI. Most people think that's 'cause of this. Uh, AI will estimate things, it will hallucinate things. A frontier model will make stuff up. And ultimately, it is, and that's a component of it. But the actual core underlying problem is we have a structural problem in this technology that we're all so desperate to use. Meaning, if you actually chat with people that are building RAG platforms, most of what they'll say is, "Well, the models will eventually get good or get better to the point that they will

[2:12](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=132s) eliminate hallucination." That is a nonsensical term. You know another word for hallucination? Alpha. Alpha is how a firm makes money. The creativity that you have and that your analysts have is a mechanism through which you can be a unique firm. Otherwise, there'd be only one company on Earth. So when you ask these types of simple questions around where data comes from, there is a piece of this that the model will just make things up, and there's a piece of this that your firm's internal data is never actually reflected

[2:42](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=162s) in this setup. So what are we seeing right now? We're seeing everyone with a checkbook buy everything they can. Every RAG platform, some firms are trying to build, some firm, some firms are running agent pilots. I had a, a customer meeting this morning where someone said they were actually an AI-first firm, which means they have one co-pilot agent running all this stuff. So why are we not seeing the complete switch of fundamental investing becoming effectively, why are we not all Jane Street? Why, why

[3:12](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=192s) has that not happened yet? And so the challenge really comes from this idea of in order for you to produce work product that you believe in, you have to trust it, meaning you have to blame somebody who gets it wrong. So we see numbers like this all the time, where eighty percent of finance execs trust numbers, but then thirty-one percent find those errors. So what do we actually need? We need a situation where the reviews that we're getting for AI apps, as builders ourselves, do more than

[3:42](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=222s) just RAG-ify things. And, and these are quotes we've actually pulled from specific, uh, I don't wanna say our competitors, but other people, where things demo super well, and they behave like a really smart intern. Or things are right if you just hook up the right structured information, but that last ten percent still gets things wrong. So what I wanna do is... We're lucky we have great partners and folks like TurboPuffer, and I gave a similar chat yesterday about this, but how do we actually do the verification? How do we

[4:12](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=252s) architecturally build a system that does not hallucinate numerically? So let's talk about it. Now, wrong numbers or wrong facts come from three sources. Either one, you retrieved the wrong document. So I pulled a document, some random thing from somewhere else. How do you fix this? Well, by retrieving the right document. But that in itself is complicated. Is the right document my own firm's internal analysis of what this stock is gonna be worth? Is it, let's say, uh,

[4:42](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=282s) some analyst who I really trust more than anyone else? That's piece one. Now, failure two. We get the right source, and we pull the wrong number out of that source. Let's say we're trying to figure out what the revenue for, I don't know, Palantir was back in twenty twenty-three. Well, if I look at the twenty twenty-five 10-K that says Palantir's twenty twenty-three revenue was this, I better make sure that that number corresponds to twenty twenty-three and not twenty twenty-five. So the fix here is really we have a filter that actually checks each

[5:12](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=312s) citation. But this is still a problem because that filter only applies after the fact, and formal verification doesn't actually help here because the thing has already been produced. Three is the mental math. Models don't do math. They predict the next token. Now, for those of you who are gonna say we have deterministic models, great, we do. We also have one CPU cycle that's good at adding two numbers together. Why don't I use that CPU cycle? So third failure is let's use deterministic tools that we already have

[5:42](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=342s) to actually compute numbers and entities rather than relying on some external tool. So here's what we do. This is the magic behind Kepler, at least for piece one. There's a whole bunch of other stuff. We are architected fundamentally differently. Code writes every number, and what code does and preprocessing does is we assign pointers to unique identifiers, meaning facts, meaning something like a number in a 10-K. This operates in a giant pre-processing and contextualization step,

[6:12](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=372s) such that all of this exists going from unstructured data to a structured data store. Then the model is effectively a smart analyst. It figures out how to go about solving the problem, but not actually being the thing to solve the problem itself. So it plans. It says, "I'm gonna go here and retrieve this source and retrieve this entity and do this thing." Code and math do the retrieval. By actually pulling a number out of a ten K at com- at search time, that's a hugely

[6:42](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=402s) expensive token cost. And Anthropic and OpenAI should thank you for paying all those tokens and token maxing, but most of your firms don't like that, at least anymore. I know there were people getting bonuses from token maxing for a while. Um, now, the next piece of this is really once a model has actually read both these pointers, it can't structurally generate numbers. Meaning unless the model actually has a mechanism of dereferencing that pointer, that number or that fact cannot be generated in Kepler's output.

[7:12](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=432s) Now, we do run a post-processing step, which is the verification step, and this is the piece that was kind of funny, where the last company just copied the verification piece, but all the structural stuff beforehand didn't make it. So what you as a consumer actually see is you see something like the revenue was this amount to this amount, and you physically can click on these chips and see exactly how we got the information. Now, in my past life, I had an opportunity. I, I was head of business engineering at Citadel, so I worked across, um, the firm to figure out effectively what do we

[7:42](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=462s) need to produce software that helps us generate alpha. You know the crazy thing with Citadel? PMs are so talented and so smart, but they work when they're paired with deterministic tools and technology. That's kind of the magical insight behind fundamental investing. Fundamental investor is so good at understanding data, but that data has to be right. And so when you build a system like Kepler, what you effectively have to do is look up the right information and then be clever about the

[8:12](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=492s) information you don't have. So let me give you the parallel. Most folks and most buy side researchers or buy side investors try to use tooling or build DCFs on their own, and then they find and search for information they don't have. They schedule GLG or GuidePoint calls. They find experts. They speak to management. The same system actually applies here. So what we do is we pick the path, and we store the information deterministically that we can actually store deterministically. When we have

[8:42](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=522s) information we can't store deterministically, we run similar pre-processing steps to grab information, store that information in a hybrid storage mechanism, which is a combination of Turbo Puffer, who's awesome by the way, and a bunch of other, um, mechanisms. And then we ultimately use a combination of re-ranking with literally old school like pointer dereferencing like dot get methods to pull things into our system. Now, why this matters is a, a lookup cannot get it

[9:12](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=552s) wrong. Search can. If you're running the same ten searches with the same query, the probabilistic nature of models means there's inherent entropy. That's actually an opportunity, and so that's what we've changed here. Now, how we do this is we break this down, and this occurs across at the SEC world as well. Um, we are frequently throwing candidate documents away, meaning we consistently have to look at not only what source is the right source for what information, but

[9:42](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=582s) also which document should we stop looking at kind of globally. So, uh, we're in ANN plus BM twenty-five to get back a set of documents, it's usually up to two hundred. Uh, we do reciprocal, uh, rank-- we effectively get sixty documents back is probably the best way to think about this. Um, then we do cross encoders. We try to figure out exactly the query and the actual document chunked together and get those results that way. What that allows us to do is we can search really wide. You know what

[10:12](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=612s) LLMs aren't good at doing? Searching really wide. What they try to do is create a plan and find either the fastest or easiest breadth-first search mechanism, uh, actually maybe depth-first search mechanism of actually, like, getting the first document and moving forward. So we allow this to search really wide and then narrow from there. The reason LLMs don't like searching wide is because it's expensive. It's because looking across a bunch of different sources is significantly worse than getting the reward function

[10:42](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=642s) back with your first hit and going from there.

[10:46](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=646s) Okay. So, um, I wanna talk a little about security here 'cause it's super important. Kepler was built with a security-first mindset and security-first mentality. Now, client documents, and this is specifically targeted towards the builders here, this is the thing that no one's chatting about. No one is actually talking about the security of what OAuth token is used and what user scope mechanism of retrieving documents, which is actually pretty scary. So let's talk about how things go wrong in these systems that people are trying to build and how you can fix it.

[11:16](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=676s) First and foremost, you forget to filter the tenant. All of a sudden, Citadel can search all of Millennium's documents. You will be in a lot of trouble. Second, a caller claims to be another firm, meaning I can claim that I am impersonating this other entity and querying documents that way. Third, search returns information that's out of date. We've had this problem kind of globally in even Google search results. Fourth, direct read skips the search rules, meaning you're not actually querying the full corpus of information or, like, there's not an allow list. You only get

[11:46](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=706s) back the single document that you did the search. Five, you hit an index that isn't populated. The way to actually solve all these problems, and there's individual things listed here, is actually scoping searches with security built in. That's kind of another critical piece that I think a lot of people miss when they're trying to build these systems themselves. Inevitably, people will start caring about the security associated with querying this information. Alpha is the most elusive thing out there, especially for folks on Wall Street. So making sure you have

[12:16](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=736s) a good mechanism of actually constraining the set of documents that come back with each query is really important. Okay, now let's talk about verification, which is the last piece here. This is probably what we're known most for, but we do the other stuff I mentioned as well. Um, how we do this is by pre-processing information and storing it deterministically, we have systems like code that actually write individual pointers. The model can only then

[12:46](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=766s) read those pointers and grab information back if they exist in the database. If I grab something out of a HashMap that doesn't exist, I don't get anything back. And so, this combination is what allows us to actually make sure the information that we retrieved actually came from the source that it came from. Now, this is a, a kind of hidden version of what our, um, source IDs look like, but these are hashes, and these are very simple things that you can do in your own system and your own setup to ensure that you're pulling back facts that in fact

[13:16](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=796s) do exist. Cool. So that's a little bit about how we do verification. Now I wanna chat a little bit about what this enables kind of globally. This enables us to verify facts that come out of our grounded corpus of data. Now, you as a builder or you as a startup creator or you as someone who is trying to figure out, how do I trust my outputs, really just need to do a couple of things. You need to show people that as you're creating these agents that

[13:46](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=826s) are theoretically looking for alpha or whatever they'd be looking for, you're using information that is actually relevant and concrete, and most importantly, accurate to the set of people that you're trying to service. So we end up with a contract where citations are either verified, corrected at, uh, search time, or s- flat out refused. Now this is crazy if you think about it. If a model cannot de-resolve that pointer, it cannot print that number out.

[14:17](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=857s) So that's kind of an insane thing. It means that the whole... And not just a number. Let's talk about facts. Like, we see these court cases all over of precedents being hallucinated. Well, why can't we create a system that just has the precedents stored deterministically and allows us to search through that list and not have a random court case that never existed all of a sudden start existing? So this techn- this, this mechanism and this technology actually becomes fairly globally applicable. And the reason I'm on stage telling you how we do what we do is

[14:47](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=887s) because I'm terrified that Wall Street never trusts this technology because of the fear that so many other companies or so many build-it-at-home folks have been, uh, instituting in these organizations. It, it's, it's gotten to the point where the vast majority of finance AI tools right now are being used to format PowerPoint presentations. We still haven't actually built a DCF cradle to grave, despite what most people are saying, and if we had, every single company on Wall Street would've adopted that overnight.

[15:18](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=918s) So this is my fear. My fear comes from a very simple fact. For us to actually get value out of this technology, we need to be able to build technology that people actually can trust and can verify. That isn't just true in finance, it's true in a bunch of different industries. So please, take what you need to from this presentation and get people using this technology. Otherwise, we're all gonna be out of a job. Now, I wanna chat through what happened when we saw this. So when we went through this whole process, um, this is actually some of our, our real production

[15:48](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=948s) data here. Um, refusals are when the model does not-- is not able to produce a number, meaning the model will say, "I couldn't de-reference this pointer, therefore, I could not produce this number." We saw that drop super drastically. Now, the reason that it didn't drop to zero is because models are still probabilistic, meaning it can still make up numbers after the fact during inference. So there is another pre-pro- uh, post-processing step after this that we, we have as

[16:18](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=978s) well. But my point with this slide is just by creating a system where the models have to de-reference their own pointers, you can actually create a very powerful, very verifiable AI platform. And that to me is my ask to all of you. As you go through the building process, as you go through the scoping process, any tool that you use, ask them if you can actually verify the numbers. Ask them if you can see where the sources are, not in terms of some after the fact annotation, but click on the number, whether it's a

[16:47](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=1007s) calculation, whether it's an extraction, or some combination of the two to make sure you trust the insight. And all of this ultimately is gonna be the unlock. So you as a vendor or you as a builder, ask these questions. Does the model actually write pointers? Does it actually create a mechanism for verifying the output? Is every citation source checked? Is arithmetic source checked? Do you have a mechanism for stopping files that reach clients or customers that it shouldn't reach?

[17:18](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=1038s) And most importantly, what happens to a number that came from nowhere? What happens when a, when a fact comes from nowhere? And how do I actually handle these kinds of, of hallucinations? So this whole platform is actually to make a very simple statement very obvious, which is we can have the smartest models in the world. We can hit some crazy cool piece of technology, but we still have the same database problems that we've had from the beginning, which

[17:48](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=1068s) are I cannot... If I search through a, a, a row and that column d- and that value doesn't exist, I won't get a retrieval back. We're now out of the search phase of AI. We need to be building infrastructure. We need to be building scaffolding and harnesses and substrates or whatever the cool VC term is these days that actually enable us to make use of this technology. So this is my challenge and my ask to all of you. Um, I hope you go out there and build something compelling, but I hope you build something compelling that you can trust. Thank

[18:18](https://www.youtube.com/watch?v=RdQCY_-SXbA&t=1098s) you.
