# Do Not Let Models Invent Numbers - Vinoo Ganesh, Kepler

Vinoo Ganesh, Kepler

[Talk page](https://ai.engineer/talks/dZgA6btnb_s-do-not-let-models-invent-numbers) · [Video](https://www.youtube.com/watch?v=dZgA6btnb_s)

## Do Not Let Models Invent Numbers

Vinoo Ganesh explains how Kepler separates model planning from financial data retrieval and arithmetic, then checks numerical output against a citation registry before showing it to an analyst.

![Kepler separates retrieval, deterministic arithmetic and a filter that checks numerical citations.](https://ai.engineer/api/talk-essay-media/cc713274c73d6c63adc82bfd9bd929b54b85c17b7648a789212392802339d548)

Kepler separates retrieval, deterministic arithmetic and a filter that checks numerical citations.

### A source link cannot explain what “17” means

[0:12](https://www.youtube.com/watch?v=dZgA6btnb_s&t=12s)

A sentence saying revenue grew 12.4% to $418.2 billion gives a financial analyst two figures to defend. Both need a traceable source, and the growth rate needs a defensible calculation. Vinoo Ganesh, introducing himself as Kepler’s CEO, starts from that working requirement: an analyst should be able to show a client or portfolio manager where a number came from. His experience leading compute at Palantir and business engineering at Citadel informs the architecture that follows.

Attaching a source after generating an answer leaves several independent ways to fail:

- **Wrong source:** Retrieval can find a semantically relevant opinion rather than reliable evidence. Ganesh’s deliberately absurd example is a personal “YOLO blog” predicting Palantir will reach $100,000 a share. Citing it would identify the opinion without making it dependable.
- **Wrong interpretation:** A 10-K or 10-Q can contain the number 17, but that token alone says almost nothing. It could mean millions, billions, yen, dollars, revenue from an earlier year, or simply a page number.
- **Wrong arithmetic:** Growth rates, margins and multiples require operations on selected inputs. Generating a plausible numerical answer does not provide the fidelity of executing a defined calculation.

These failures need different controls. Better search can improve source selection, but it cannot recover a missing currency or table scale by itself. Correctly extracting two values does not ensure that their growth rate was calculated correctly. Kepler’s governing decision is therefore to make the model cite numbers while other software supplies and computes their values.

### The model plans; the registry supplies the numbers

[4:11](https://www.youtube.com/watch?v=dZgA6btnb_s&t=251s)

Kepler divides the work among retrieval, human-written deterministic code and a post-processing filter. Retrieval finds evidence. Predetermined code performs arithmetic. The filter removes numerical output that cannot be verified. The distinction between human-written calculation code and code improvised by the model matters here: the permitted computation belongs to the platform’s implementation.

![Language models plan analyses while Kepler’s tools retrieve facts, calculate values and register citations.](https://ai.engineer/api/talk-essay-media/24120dd5288cbdb0d1ad0da2c7348f69def63788ce62cc1db05be80b04d27349)

Language models plan analyses while Kepler’s tools retrieve facts, calculate values and register citations.

The citation registry begins before a user asks a question. Ganesh says Kepler processed more than 30 million SEC documents and registered numerical facts extracted from filings. Those facts include financial-statement values and figures embedded in management discussion and analysis, such as a written statement that revenue grew 37%. A registered citation gives a fact an anchor back into its source document. Production output must resolve to verifiable evidence rather than merely carry a document link.

Consider the question “What is Google’s adjusted EBITDA?” The model’s useful job is to work out what answering it requires. Relevant facts may span multiple years of 10-Ks. An investor presentation or earnings call may disclose an adjustment, and a later restatement may change a previously reported value. The model plans those retrieval and reconciliation steps with knowledge of the available tools; tools then retrieve facts, compute results, build tables and register the resulting information.

The registry also preserves meaning. In Ganesh’s hypothetical example, a Palantir 2025 10-K contains a 2023 revenue figure of $1 billion. The fact must retain 2023 as its data period even though the document is from 2025. And the visible cell might contain only `1`, with “billions” elsewhere in the table. Losing either relationship produces a numerically plausible but incorrectly scoped answer.

Planning does not stop a model from inventing additional figures while composing its response. The final filter exists because that behavior remains possible: if the platform cannot deterministically establish that a number came from a trusted source, it strips the number. The guarantee is enforced on the output path, after the model has had an opportunity to generate unsupported content.

### Carry value, unit, scale and period together

[8:18](https://www.youtube.com/watch?v=dZgA6btnb_s&t=498s)

The total-net-sales example makes the extraction problem concrete. A value around 416,000 is not yet an interpretable financial fact. It needs a unit, a scale and a period alongside the value. These fields determine whether a downstream calculation compares like with like. The important relationship in a filing is often between a cell and information elsewhere on the page, rather than between adjacent words in a text chunk.

![Kepler combines structured filing data with search over unstructured documents.](https://ai.engineer/api/talk-essay-media/42f13cfe526848a5bd2de2786c1d207f0beee699a0c29cf26b8dbc0b3d6914cd)

Kepler combines structured filing data with search over unstructured documents.

Structured filing data helps, but it cannot cover the whole research workflow. Ganesh challenges the idea that XBRL solves extraction completely, estimating that it is incorrect around 10% of the time; he does not define the error measure behind that estimate. More fundamentally, investment teams also use their own memos and internal documents, which do not all arrive as structured financial records.

Kepler therefore structures data where structure is available and searches where it is not. The intended corpus includes public filings alongside internal sources such as SharePoint, Box, Egnyte, OneDrive and Outlook. A turbopuffer namespace supports retrieval across that information, while the surrounding Kepler machinery remains responsible for interpreting facts and verifying numerical output.

### Search widely, then rank for relevance and trust

[9:48](https://www.youtube.com/watch?v=dZgA6btnb_s&t=588s)

Retrieval combines two search mechanisms over the namespace. Approximate nearest-neighbor search, or ANN, retrieves through embeddings; BM25 supplies lexical matching. Ganesh describes using 1,280-dimensional embeddings. The architecture searches broadly through both mechanisms before narrowing the candidates, rather than asking a single retrieval score to settle every question about relevance and source quality.

The candidates pass through reciprocal reranking and another ranking stage that accounts for source trust, with a cross-encoder used for reranking before results return to the model. Source trust matters when apparently equivalent facts disagree. An earnings-call transcript may report revenue differently from the 10-K. Treating these as two pieces of information to reconcile preserves the conflict; collapsing them into one interchangeable search result hides it.

![Parallel ANN and BM25 retrieval feed reciprocal rank fusion and cross-encoder reranking.](https://ai.engineer/api/talk-essay-media/78756c21140a53527767f1cb65a53b8256b6b6b922e48c8869d19cc0ab59996e)

Parallel ANN and BM25 retrieval feed reciprocal rank fusion and cross-encoder reranking.

In the diagram, ANN and BM25 run at the same time. Each retrieves three times the requested candidate count. Reciprocal rank fusion combines their lists using a rank constant of 60. The cross-encoder stage selects 20 documents, and the diagram shows 10 returned to the model, with a maximum of 25.

Keeping this pipeline current has an operating cost. New information can require regenerated embeddings, and Ganesh emphasizes the expense of doing that repeatedly. He presents efficient embedding generation, including the possibility of hosting the embedding model, as an important part of making the retrieval strategy practical. The ranking design depends on being able to refresh the searchable information economically.

Parallel retrieval feeds a narrower evidence set
ANN and BM25 retrieve candidates independently. Ranking combines and narrows those candidates before evidence returns to the model.
- Searchable namespace — Information available to the retrieval pipeline.
- ANN search — Embedding-based approximate nearest-neighbor retrieval.
- BM25 search — Lexical retrieval over the namespace.
- Reciprocal reranking — Ranks the broadly retrieved candidates.
- Further reranking — Accounts for trusted sources; uses a cross-encoder.
- Model receives evidence — The selected retrieval results return to the model.
- Searchable namespace → ANN search (Search through embeddings)
- Searchable namespace → BM25 search (Search lexical matches)
- ANN search → Reciprocal reranking (Supply candidates)
- BM25 search → Reciprocal reranking (Supply candidates)
- Reciprocal reranking → Further reranking (Pass ranked candidates)
- Further reranking → Model receives evidence (Return selected results)

### Authorize each hit and preserve each fact’s identity

[11:18](https://www.youtube.com/watch?v=dZgA6btnb_s&t=678s)

Internal research introduces a separate requirement: a relevant document may still be forbidden to the person asking. Ganesh uses two Citadel desks as the example. Desk One must not see Desk Two’s research simply because both belong to the same firm. Isolation therefore has to reach the document level.

![A registered citation separates the source identifier, displayed value and underlying anchor.](https://ai.engineer/api/talk-essay-media/752557d2bd5d42e9bffab5a247fa452297fb77ddc710b2e7e5b42de455ea29e2)

A registered citation separates the source identifier, displayed value and underlying anchor.

Every retrieval hit is reauthorized in Postgres. The checks cover the citation, value and document through the computations, and builders add filters as queries are assembled. Only the authentication component can create a scoped token. These controls make permission checks part of the path that retrieves and uses information, rather than relying solely on an initial search scope.

A fact also needs an identity that survives its presentation. Kepler assigns unique identifiers to facts, with source IDs resolving to stored values and metadata in a deterministic engine. Entity recognition and resolution help determine which entity a fact belongs to. The display value can then be formatted for a requested presentation, such as thousands of dollars, while retaining the underlying fact and its source.

The user-facing invariant is that every number and fact can be clicked to inspect where it came from and how it was obtained. Ganesh connects this to reproducibility and describes an idempotent guarantee: repeated queries yield the same results. The deterministic values and calculations explain the intended basis for that guarantee; the talk does not specify how repeated queries are scoped when source data or restatements change.

### Resolve the reference, repair selectively, strip when unproven

[13:46](https://www.youtube.com/watch?v=dZgA6btnb_s&t=826s)

A model may already know a historical revenue figure from training. That familiarity does not establish that it recalled the right year, scope or value, and it does not supply a traceable extraction path. Ganesh uses revenue from 2005 to illustrate why even a convincing historical answer must pass through the evidence system.

![Kepler checks numerical citations against the registry while streaming a response.](https://ai.engineer/api/talk-essay-media/d5f02600d31c66a963e78d30e1f1c49ab11932c61c0c4cee80e4919ee9387fb0)

Kepler checks numerical citations against the registry while streaming a response.

Ganesh characterizes roughly a third of Kepler’s citations as “the math the model didn’t do.” The point is that a substantial share of cited output represents work delegated away from the model, rather than facts accepted from its training memory. Those citations can arise while working with web results, spreadsheets and filing exhibits as well as other retrieved material.

During inference, references are resolved against the registry. Ganesh’s analogy is dereferencing a pointer: a reference identifies an entry, and the registry supplies the information needed to trace it. If no matching entry can support a numerical claim, the filter strips the number. A fluent sentence is insufficient to let an unresolved value through.

That decision has a visible product cost. Kepler gives up some of the speculative freedom people enjoy in a conversational model. Ganesh explicitly says it is a poor fit for creative marketing content or for users who want the model to invent projections freely. The same restriction is useful when building a discounted cash flow model, or DCF, whose inputs must be accurate and traceable.

Removal is not always the first or only response. A post-processing step can attempt citation repair: when information cannot be reliably traced, an agentic process may try to establish which source supports it. Ganesh says Kepler does this in selected cases but leaves the selection rules and repair mechanism proprietary. The disclosed distinction is between attempting to recover a valid source relationship and allowing an unsupported number to remain.

### What the filter caught in ordinary model output

[16:40](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1000s)

Kepler’s slide headline says the filter refused 10,981 citations. The bars add up to that total only when they include 799 events labeled “passed unverified (soft).” Kepler classifies 3,810 events as numeric mismatches. The largest category, 4,542 quotes the filter could not find in their sources, illustrates how a traceability check can fail without establishing that a number is wrong. The chart cites production event logs from April through July 2026. Ganesh describes evaluations involving Claude or ChatGPT, but does not explain how those evaluations relate to the logs. The chart gives neither the total number of citations checked nor counts by model.

![Horizontal bars count citation events by reason. The headline calls 10,981 citations refused, but the bars include 799 marked “passed unverified (soft).” Five red bars total 3,810 numeric mismatches.](https://ai.engineer/api/talk-essay-media/569c5597299de884eed812379fb82db101240d73e30db71db93d1722badcf8db)

The slide’s headline counts 10,981 citations as refused, including 799 labeled “passed unverified (soft).” Kepler identifies 3,810 numeric mismatches in its April–July 2026 production event logs. The chart gives no total for all citations checked.

These results motivate the architectural objection to plain retrieval-augmented generation. Connecting a language model to a vector database can deliver relevant context, but relevance alone does not verify the numerical answer produced from it. A financial research system also needs controls over source identity, the meaning of extracted values, arithmetic and the final response. Retrieval is one component of that system.

The sharpest correction in the talk concerns who owns the guarantee. Ganesh begins to describe the model as structurally incapable of producing hallucinated numbers, then corrects himself: the platform enforces that restriction. The model still hallucinates. The filter and deterministic execution path are intended to prevent its unsupported numerical inventions from reaching the user.

### Click through the calculation—and keep the guarantee in scope

[18:32](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1112s)

The product example asks for Apple’s fiscal 2025 revenue and how quickly it grew. The screen shows net sales of $416,161 million in fiscal 2025 and $391,035 million in fiscal 2024. Dividing the $25,126 million increase by the fiscal 2024 figure gives 6.4% growth after rounding. Clicking the revenue figure highlights the filing entry. Opening the growth calculation shows the two inputs, each with its own source.

![An Apple revenue example links displayed figures to filing evidence and the steps of the calculation.](https://ai.engineer/api/talk-essay-media/d40a6f2a9127cc0b49d2758ce369c10d343db59b7c776d5666f7bbef6b062285)

An Apple revenue example links displayed figures to filing evidence and the steps of the calculation.

That inspection path targets a familiar manual workflow. Analysts still extract information from S-1s and move figures from PDFs into Excel because producing a usable financial model requires more than a plausible answer. Kepler’s aim is to automate that transfer and calculation work while preserving the information needed to check it.

The closing guarantees remain specific:

- **Citations:** Models select references, and the platform verifies citations before presenting numerical output.
- **Arithmetic:** The compute layer performs calculations using deterministic code.
- **Isolation:** Tenant and document controls limit which research a user can retrieve.
- **Recall:** Kepler measures retrieval recall, acknowledging that finding the needed evidence is a separate concern from checking a value already retrieved.

Those controls do not amount to an autonomous hedge fund or a replacement for everyone making investment decisions. Ganesh expects more “quantamental” work, combining quantitative methods with fundamental research. The intended division of labor leaves people with the investment ideas they want to develop and gives AI the repetitive retrieval and assembly work they would rather avoid. Traceable numerical execution makes that handoff more useful without turning an investment judgment into a guaranteed outcome.

## Key takeaways

- A financial figure needs more than a source link: its value, unit, scale and period must stay attached to the right fact.
- Kepler assigns planning to the model, retrieval and arithmetic to tools, and numerical output checks to a post-processing filter.
- Hybrid retrieval finds candidates; further ranking handles relevance and source trust, while Postgres reauthorization checks whether each hit is permitted.
- Unresolved numerical claims are stripped, with selective citation repair attempted in some cases. This trades speculative freedom for traceability.
- The model can still hallucinate. The claimed protection belongs to the platform’s execution and filtering path, and does not guarantee an investment outcome.

## Resources

- [How Kepler built verifiable AI for financial services with Claude](https://claude.com/blog/how-kepler-built-verifiable-ai-for-financial-services-with-claude) — The architecture case study Ganesh points readers toward for more detail on Kepler’s use of model planning and deterministic infrastructure.
- [Context Is the Easy Part](https://kepler.ai/blog/2026/02/13/context-is-the-easy-part/) — Ganesh develops the surrounding data-engineering requirements: provenance, versioning, validation, reproducibility and observability when retrieved context changes.

## Related talks

- [How Kepler Built Verifiable AI for Financial Services](https://ai.engineer/talks/Tt2kX2sgQio-kepler-built-verifiable-ai-financial-services) — Develops the companion concepts of atomic provenance, scoped determinism and derivation chains, including how firm-specific processes affect verification.
- [Respect The Process](https://ai.engineer/talks/CLttOU7n6sI-respect-process) — Applies a comparable separation to sustainability calculations: agents propose changes through typed tools, while controlled deterministic execution validates consequential operations.

## Slides

### Do Not Let Models Invent Numbers

Vinoo Ganesh introduces how Kepler retrieves, computes and verifies figures, with turbopuffer as one component of its architecture.

[0:12](https://www.youtube.com/watch?v=dZgA6btnb_s&t=12s) [Kepler title slide describing how it retrieves, computes and verifies every figure, presented by Vinoo Ganesh.](https://assets.ai.engineer/sites/aiecode2025/sha256/4a4d52653bf8ce23a7ad4d2244cad12e3ba60d9c997cacc4ef11ff495dbdfd9c.webp)

Do Not Let Models Invent
Numbers

Kepler

How Kepler retrieves, computes and verifies every figure

Vinoo Ganesh CEO, Kepler · with turbopuffer · AIE

$416,161

Kepler × turbopuffer

KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize

AI Engineer
NEW YORK

OCTOBER 13, 2026
EXPO SESSION
Shannon Room

### Every number must be traceable

Red circles and separate “source?” labels flag both the 12.4% growth rate and $418.2B revenue figure for verification. Ganesh argues that attaching a source after generation does not ensure the numbers are correct.

[1:43](https://www.youtube.com/watch?v=dZgA6btnb_s&t=103s) [A revenue statement has its growth rate and revenue figure circled in red, each with a source question.](https://assets.ai.engineer/sites/aiecode2025/sha256/4e3cc806524f0967906cc5355e88102e1fe4e96348abb5449079c6d7be2bf4eb.webp)

An analyst never shows a client a number they cannot trace

Revenue grew 12.4% to $418.2B.
source?
source?

Kepler
Kepler x turbopuffer · AIE
02
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
EXPO SESSION
Shannon Room

### Plausible numbers fail in three ways

Attaching a source leaves three ways for numbers to go wrong: retrieval can miss the source, extraction can confuse scale or units, and model arithmetic can miscalculate growth rates, margins and multiples.

[2:36](https://www.youtube.com/watch?v=dZgA6btnb_s&t=156s) [Three failure modes: wrong source; right source with incorrect scale, unit or transcription; and mental math for growth rates, margins and multiples.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c3a8b956b89232051867ab03d6a7c5d213d56530e72bd8a323849a4e6dcaa0f.webp)

Plausible numbers fail in three ways

Kepler

Wrong source
Retrieval missed, the model filled the gap.

Right source, wrong number
Scale ($M vs $B), unit ($ vs shares), transcription.

Mental math
Growth rates, margins and multiples computed in the model's head.

Kepler x turbopuffer · AIE
03
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
EXPO SESSION
Shannon Room

### The model cites numbers. It never writes them.

Kepler routes numbers through three stages: retrieval finds the source, human-written deterministic code does the math, and a post-processing filter checks every receipt. The model cites the resulting numbers.

[4:01](https://www.youtube.com/watch?v=dZgA6btnb_s&t=241s) [Three connected boxes show retrieval finding the source, code doing the math, and a filter checking every receipt.](https://assets.ai.engineer/sites/aiecode2025/sha256/e3b507585b41a8aa0f001fcb80f8c18704b20e8e1ac1e983a537521d01ece86d.webp)

The model cites numbers. It never writes them.

Kepler

1
Retrieval finds the source

2
Code does the math

3
A filter checks every receipt

Kepler x turbopuffer · AIE
04
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
EXPO SESSION
Shannon Room

### This runs in production, so we can measure it

Kepler’s Apr–Jul 2026 production logs record 3.0M filing facts exposed as citable anchors. Its filter refused 10,981 citations and corrected 21,147 to the source’s exact form.

[4:27](https://www.youtube.com/watch?v=dZgA6btnb_s&t=267s) [Production metrics show 20,248 registered citations, 3.0 million citable filing facts, 10,981 refused citations, and 21,147 citations corrected to the source's exact form.](https://assets.ai.engineer/sites/aiecode2025/sha256/13c71aab58929e647a2e86862efda625d1d5394d71862e9dbee92bee78be5dfc.webp)

This runs in production, so we can measure it

Kepler

20,248
citations registered by tools

3.0M
individual filing facts shown to the model as citable anchors

10,981
citations the filter refused

21,147
citations the filter corrected to the source's exact form

Source: Kepler production event logs, Apr–Jul 2026.

Kepler x turbopuffer · AIE
05
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
EXPO SESSION
Shannon Room

### Language models plan; deterministic software handles every number

The LLM plans and writes prose while tools retrieve data, compute, and register typed sources. A streaming citation filter checks the model’s output against that registry, strips unverified numbers, and returns clickable citation chips.

[5:46](https://www.youtube.com/watch?v=dZgA6btnb_s&t=346s) [Architecture connects a user question to an LLM agent, tools, a citation registry, a streaming citation filter, and clickable verified numbers. Tools read public and private data.](https://assets.ai.engineer/sites/aiecode2025/sha256/e85da03dafbc429bb40ac389d1f7834c79a0a4a6c5e65eb7cee780cad10bf346.webp)

Language models plan; deterministic software handles every number

Kepler

User question
asks
LLM agent
plans, writes prose

tool call
Tools
retrieve, compute, build tables

registered source
Citation registry
typed sources

lookups
Streaming citation filter
verified chips
User
numbers as clickable chips

tokens

reads
Public data
filings, transcripts, ownership (structured)

Private data
client libraries (turbopuffer)

Kepler x turbopuffer · AIE
06
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
EXPO SESSION
Shannon Room

### Structured where the data is structured, search where it is not

Kepler tags public filing facts with value, unit, scale and period. Private decks, memos and spreadsheets flow into a turbopuffer namespace for hybrid vector and BM25 search.

[8:21](https://www.youtube.com/watch?v=dZgA6btnb_s&t=501s) [Public filing facts have value, unit, scale and period; private decks, memos and spreadsheets are indexed for vector and BM25 search.](https://assets.ai.engineer/sites/aiecode2025/sha256/d868ba7826810160a9c7e63cdf401c0bba940381975f18266a6bb54d43a65301.webp)

Kepler

Structured where the data is structured, search where it is not

PUBLIC
10-K · Consolidated statements of operations
Total net sales
416,161
↑ A tagged fact: value, unit, scale, period

PRIVATE
Deck
Memo
Spreadsheet
index

turbopuffer namespace
hybrid search:
vector + BM25

SharePoint, Box, Google Drive, Egnyte, OneDrive,
Outlook, uploads

Kepler x turbopuffer · AIE
07
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
EXPO SESSION
Shannon Room

### Search wide, then rerank with a cross-encoder

Parallel ANN and BM25 searches each fetch 3 × top_k. Reciprocal rank fusion combines them; a cross-encoder reranks the top 20 documents before 10 results (max 25) return to the model.

[9:46](https://www.youtube.com/watch?v=dZgA6btnb_s&t=586s) [Query flows through concurrent ANN and BM25 searches, reciprocal rank fusion, candidate selection and cross-encoder reranking before results return to the model.](https://assets.ai.engineer/sites/aiecode2025/sha256/52c49253c3aa8e61cd973b429a81c17935793c60405f89e2ed4ca90480628f1d.webp)

Search wide, then rerank with a cross-encoder

Query zembed-1,
1280-dim
1

ANN + BM25
concurrent, 3x
over-fetch
ANN
BM25
3 × top_k each

RRF fusion k = 60
100 / namespace

Rerank candidates
best chunk first,
max 5 per doc
100

zerank-2 cross-
encoder
top 20 docs

Returned to the
model
10 (max 25)

Vector and BM25 searches run at the same time
Each arm over-fetches 3 × top_k
Results fused with reciprocal rank fusion, k = 60

Kepler
Kepler x turbopuffer · AIE
08
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
EXPO SESSION
Shannon Room

### Tenant isolation you cannot forget to apply

The query crosses four nested safeguards: signed-token identity, auth-created tenant scopes, a shared filter builder, and Postgres reauthorization. A shared allow-list keeps reads from returning documents that search would hide.

[11:28](https://www.youtube.com/watch?v=dZgA6btnb_s&t=688s) [Four nested safeguards protect document access: signed-token identity, unforgeable tenant scopes, a shared query filter builder and database reauthorization. Search and read share an allow-list.](https://assets.ai.engineer/sites/aiecode2025/sha256/e970392d5188a64f5a6d80e3b611313e8bd9179b5575a3e457a650d759b13b59.webp)

Tenant isolation you cannot forget to apply

Kepler

1 Identity: org and user come from the signed token, stored on the chat.
2 Unforgeable scopes: only auth can create a tenant scope; other code can only hold one.
3 One filter builder: one builder adds the tenant filter to every query. A namespace's kind comes from its name, never from comparing it to the caller.
4 Re-check: every hit is re-authorized in Postgres (tenant, ownership, not deleted) before the model sees it.

query
Document

Search and read share one allow-list, so a read cannot return what a search would hide.

Kepler x turbopuffer · AIE
09
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
EXPO SESSION
Shannon Room

### Anatomy of a citable number

Kepler ties each displayed number to a tool-registered source and an anchor identifying the supporting fact, cell, row or quote. Clicking the number reveals its source; the display value supports customized units.

[12:35](https://www.youtube.com/watch?v=dZgA6btnb_s&t=755s) [A citation contains a registered source identifier, display value and anchor to an exact fact, cell, row or quote.](https://assets.ai.engineer/sites/aiecode2025/sha256/3ddc9471b012667af70ff3114626a0b4397ca114385e6647c5eb360c52c29720.webp)

A number is citable only if a tool registered it

Kepler

[ src-7f3a : $416,161 | a-12 ]

source id a source a tool registered
display what the user reads
anchor the exact fact, cell, row or quote

Revenue was $416,161 million

Kepler x turbopuffer · AIE
10
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
EXPO SESSION
Shannon Room

### Calculations account for a third of citations

Calculations lead Kepler’s citation categories: 6,624, or 32.7% of all citations in its Apr–Jul 2026 production logs. These are computations the model did not perform; SEC filings and web results follow.

[13:43](https://www.youtube.com/watch?v=dZgA6btnb_s&t=823s) [Horizontal bars compare citation categories, with calculations leading at 32.7 percent, followed by SEC filings and web results.](https://assets.ai.engineer/sites/aiecode2025/sha256/c91f35bbef682a0deae82041b4f338cff165b286db543b046b83006c3c7474df.webp)

A third of all citations are math the model did not do

Calculation (compute): 6,624 — 32.7% of all citations
SEC filing: 5,353
Web result: 4,140
Spreadsheet cell: 789
Stock price: 603
Earnings transcript: 557
Investor slides: 548
Client document (SharePoint): 462
Earnings data: 402
User-provided number: 375
Filing exhibit: 176
Uploaded document: 72
Stock split: 66
Drug price: 41
News: 40

Source: Kepler production event logs, Apr–Jul 2026.

Kepler
Kepler x turbopuffer · AIE
11
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
Engineering the future of AI

### Checking citations against their sources

The filter buffers each receipt until it closes, then checks its anchor against the source registry. Matching citations become green chips; unmatched citations become plain text. Checks depend on the source type.

[14:38](https://www.youtube.com/watch?v=dZgA6btnb_s&t=878s) [A sequence diagram shows a filter consulting a source registry before rendering a green citation chip or plain text; a table lists checks for different anchor types.](https://assets.ai.engineer/sites/aiecode2025/sha256/fd888b711e69dc0dc9c4b54e8c12e03d86b94435a9227e1bc38c61a374f6f8ce.webp)

Every receipt is checked against its source, mid-stream

Kepler

LLM tokens
Filter
Registry
User

[src-7f3a: ...
holds until the receipt closes
look up source
anchor (a-12)
match: green chip
no match: stripped to plain text

ANCHOR — CHECK
Filing fact — Unit compatible; value matches at shown precision
Spreadsheet cell — Cell holds that value
Filing table cell — Parsed table holds that value
Ownership row (Form 4, 13F, 13D/G) — Figure is on that row
Quote — Quote is verbatim in source; every number shown is in the quote

Kepler x turbopuffer · AIE
12
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
EXPO SESSION
Shannon Room

### Repairing citations before removing their markers

Kepler streams prose while buffering open citations. Its filter tries format corrections, matching sibling facts and verbatim quote repairs before stripping an unsupported citation marker and leaving plain text.

[15:56](https://www.youtube.com/watch?v=dZgA6btnb_s&t=956s) [A table maps exact matches, formatting differences, sibling facts and paraphrased quotes to verified or corrected chips; unmatched citations become plain text.](https://assets.ai.engineer/sites/aiecode2025/sha256/120aec5bce34bbbd90334dfd65b668515e6e8b8a6b4956e66b1c30f5c8bf4f4a.webp)

Before stripping a citation, the filter tries to repair it

Prose streams straight through. The filter buffers only while a receipt is open, then runs this ladder.

WHAT THE FILTER FINDS | WHAT IT DOES | USER SEES
Exact match | Renders the citation as a chip | verified chip
Same value, different format | Rewrites the display to the source's exact form: $416.2B becomes $416,161 (in millions) | corrected chip
Value matches a sibling fact | Re-points the anchor to the fact in the same source that does match | corrected chip
Paraphrased quote | Snaps the quote to the source's exact wording | corrected chip
No honest match | Strips the marker and logs why | plain text

Kepler
Kepler x turbopuffer · AIE
13
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
EXPO SESSION
Shannon Room

### Citation rejection counts

Kepler’s Apr–Jul 2026 production logs show 10,981 refused citations, including 3,810 numeric mismatches caught before users saw them. Missing quotes were the largest category; numbers absent from quoted text led the numeric failures.

[16:39](https://www.youtube.com/watch?v=dZgA6btnb_s&t=999s) [Bar chart reporting 10,981 refused citations, including 3,810 numeric mismatches, with counts by rejection reason.](https://assets.ai.engineer/sites/aiecode2025/sha256/1d7ff30a0f63a542495b8bd8c690950339eee47990ab8872ca669d9da68d4b96.webp)

Kepler

The filter refused 10,981 citations. 3,810 were wrong numbers.

Quote not found in source: 4,542
Number not in the quoted text: 2,357
Source never registered: 1,412
Value does not match source: 1,034
Quote not found, passed unverified (soft): 799
Malformed marker: 341
Fact the model was never shown: 320
Raw id leaked into prose: 75
Spreadsheet cell does not exist: 72
Wrong unit ($ vs shares, currency): 27
Nested marker: 2

3,810
numeric mismatches caught
before a user saw them

Source: Kepler production event logs, Apr–Jul 2026.

Kepler x turbopuffer · AIE
14
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
EXPO SESSION
Shannon Room

### Deterministic arithmetic and citation checking

Deterministic code computes results and registers them under hash-derived IDs. If the model cites a 26.9% result but writes 99%, the filter strips the citation and logs a value mismatch.

[17:56](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1076s) [Four-step value mismatch example above a diagram connecting revenue inputs to deterministic percent-change computation and a registered result.](https://assets.ai.engineer/sites/aiecode2025/sha256/ebc97ac977b366c5a1bc2e1faf730ebed2b4ed8cd6e677e8c7a3d36162315ff8.webp)

The model never does arithmetic

1. Compute registers a net margin of 26.9% under a hash-derived id
2. The model cites that id but writes 99%
3. The filter checks 99% against 26.9% and strips the citation
4. The reason is logged: value mismatch

Revenue FY25
Revenue FY24
operands
Compute: percent change
deterministic math
result
6.4%
hash-derived id, the model cannot forge it

Statements and cap tables are laid out in code. An uncited zero is missing data, not a real zero.

Kepler x turbopuffer · AIE
15
KEPLER.AI

Interface text:
Kepler
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
EXPO SESSION
Shannon Room

### Tracing Apple revenue and growth to sources

The Kepler demo traces Apple’s $416,161M FY2025 revenue to a highlighted filing entry. Clicking 6.4% opens the growth calculation, whose two revenue inputs each link to their own source.

[18:31](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1111s) [Three product screenshots show Apple's FY2025 revenue answer, a highlighted filing value, and a 6.4% growth calculation with separately sourced inputs.](https://assets.ai.engineer/sites/aiecode2025/sha256/8653cba38f560d17a9cf2f575cf4527bb205401750ce99a2850af67aacd02246.webp)

Kepler
What was Apple's FY2025 revenue, and how fast did it grow?

Left screenshot:
What was Apple's FY2025 revenue, and how fast did it grow?
Read SEC filings for Apple Inc.
Apple's FY2025 net sales were $416,161M, up 6.4% from $391,035M in FY2024. Both figures are from Apple's 10-K for the fiscal year ended September 27, 2025.

Filing excerpt:
Columns: 2025 | 2024 | 2023
Net sales:
Products: $307,003 | $294,866 | $298,085
Services: 109,158 | 96,169 | 85,200
Total net sales: 416,161 | 391,035 | 383,285
Cost of sales:
Products: 194,116 | 185,233 | 189,282
Services: 26,844 | 25,119 | 24,855
[illegible]

Center screenshot:
What was Apple's FY2025 revenue, and how fast did it grow?
Read SEC filings for Apple Inc.
Apple's FY2025 net sales were $416,161M, up 6.4% from $391,035M in FY2024. Both figures are from Apple's 10-K for the fiscal year ended September 27, 2025.

FY2025 revenue growth: 6.4%
=(B2-B3)/B3 = 0.06426 → 6.4%
Input | Amount (USD)
FY2025 net sales 10-K | 416,161,000,000
FY2024 net sales 10-K | 391,035,000,000
FY2025 revenue growth | 6.4%

Right screenshot:
FY2025 revenue growth: 6.4%
Input | Amount (USD)
FY2025 net sales 10-K | 416,161,000,000
FY2024 net sales 10-K | 391,035,000,000
FY2025 revenue growth | 6.4%

Apple Inc.
CONSOLIDATED STATEMENTS OF OPERATIONS
(In millions, except number of shares, which are reflected in thousands, and per-share amounts)
Years ended
Columns: September 28, 2024 | September 30, 2023 | September 24, 2022
Net sales:
Products: $294,866 | $298,085 | $316,199
Services: 96,169 | 85,200 | 78,129
Total net sales: 391,035 | 383,285 | 394,328
Cost of sales:
Products: 185,233 | 189,282 | 201,471
Services: 25,119 | 24,855 | 22,075
Total cost of sales: 210,352 | 214,137 | 223,546
Gross margin: 180,683 | 169,148 | 170,782

click the chip
exact fact highlighted in the filing
every input of the calculation has its own source

Kepler x turbopuffer · AIE
16
KEPLER.AI

Interface text:
Left screenshot: 9:02 AM; Source; 10-K; AAPL 10-K FY2025; Exhibits (76); 1 of 4; 156 / 300
Center screenshot: Apple FY2025 revenue growth analysis; Source; COMPUTED; 6.4%; B4; A; B; C; D; E; F; row numbers 1–4
Right screenshot: COMPUTED; 6.4%; B3; 391,035,000,000; A; B; row numbers 1–4; 10-K; Exhibits (85)

Recording overlay:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
EXPO SESSION
Shannon Room

### Where the guarantees end, and five rules to reuse

In Kepler’s Apr–Jul 2026 audit, 87.2% of numbers in final answers had citations. Verification catches wrong numbers but can miss omissions; the five rules combine source checks, arithmetic in code, tenant isolation and recall.

[19:45](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1185s) [Closing slide reports 87.2% cited and 12.8% uncited numbers, explains verification limitations, and lists five rules for citations, computation, tenant isolation and recall.](https://assets.ai.engineer/sites/aiecode2025/sha256/0965c7fc54bc37384705813f55444f439c6c81105e260bd1f6d260ae9b8617f0.webp)

Kepler

Where the guarantees end, and five rules to reuse

Numbers in audited final answers
97,882 cited (87.2%)
14,347 uncited (12.8%)

Uncited numbers are logged in final answers and deleted between agents.

Soft passes look verified to users, so evals count them as failures (799 in this window).
Verification stops wrong numbers, not missing ones. Recall still matters.

1 Make the model cite numbers, not write them.
2 Verify every citation against its source, in the stream.
3 Push arithmetic and table layout into code.
4 Isolate tenants with types, then re-check every hit.
5 Measure recall, and count every fallback.

Thank you · Vinoo Ganesh · Kepler · turbopuffer
Source: Kepler production event logs, Apr–Jul 2026.

Kepler x turbopuffer · AIE
17
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
EXPO SESSION
Shannon Room

[20:28](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1228s) [Closing slide showing cited versus uncited numbers, verification limitations, and five rules for reliable numerical answers.](https://assets.ai.engineer/sites/aiecode2025/sha256/1fedcc5cb4d8d2b24024a60c77af238c24369cc00dd627df23bbd1f5c91be195.webp)

Kepler

Where the guarantees end, and five rules to reuse

Numbers in audited final answers
97,882 cited (87.2%)
14,347 uncited (12.8%)

Uncited numbers are logged in final answers and deleted between agents.

Soft passes look verified to users, so evals count them as failures (799 in this window).
Verification stops wrong numbers, not missing ones. Recall still matters.

1 Make the model cite numbers, not write them.
2 Verify every citation against its source, in the stream.
3 Push arithmetic and table layout into code.
4 Isolate tenants with types, then re-check every hit.
5 Measure recall, and count every fallback.

Thank you · Vinoo Ganesh · Kepler · turbopuffer
Source: Kepler production event logs, Apr–Jul 2026.

Kepler x turbopuffer · AIE
17
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
AI Engineer

### Appendix

An appendix divider marks the end of the presentation as Ganesh thanks the audience.

[20:45](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1245s) [Dark green Kepler slide titled Appendix.](https://assets.ai.engineer/sites/aiecode2025/sha256/2d4d12e33d0c607a74f0080f03293b02f3d851347ddf7fa1a5980994d58bb4ad.webp)

Appendix

Kepler

Kepler x turbopuffer · AIE
18
KEPLER.AI

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
Engineering the future of AI

## Full transcript

[0:12](https://www.youtube.com/watch?v=dZgA6btnb_s&t=12s) Hey, thanks for coming. My name is Vinoo Ganesh. I'm the CEO of Kepler. Uh, we have a few of my teammates here as well. I'm excited to be here to chat with you. So the initial inception of this session comes from the World's Fair for AIE, where we discussed at a high level, uh, Kepler as a whole and how we do what we do. Um, and our big claim to fame right now, and there's a new bunch of other stuff, is Anthropic did a full deep dive into our architecture, our technology, and the biggest question that I

[0:42](https://www.youtube.com/watch?v=dZgA6btnb_s&t=42s) got after the World's Fair presentation was, "Well, how do you actually do this?" Um, so our partners and friends at turbopuffer have given me this session to talk about how we use turbopuffer as a component of this overarching ecosystem. Um, also shameless plug, Jack and I are doing a talk on the main stage in about twenty-five minutes, so we're gonna power through this, and if you wanna hear about fundamental investing in the age of AI, uh, feel free to come to that twenty-five minutes from now. Um, so let's talk a little bit

[1:12](https://www.youtube.com/watch?v=dZgA6btnb_s&t=72s) about-- So a quick background on myself. Um, again, I'm Vinoo. I'm the CEO of Kepler. Uh, I began my career at Palantir. I led the compute layer there, um, then built my own data as a service company, and then was head of business engineering at Citadel. Um, so a lot of what we learned here in building, uh, Kepler comes directly from the experiences of understanding how to build and harness or a substrate or whatever the cool new word is in pretty aggressive environments where accuracy and fidelity

[1:42](https://www.youtube.com/watch?v=dZgA6btnb_s&t=102s) matter pretty aggressively. So let's talk a little bit about what Kepler does. Um, I know we're, um, we've been in a lot of places. We're excited to be part of Gemini 4's, um, blog post today as well. Uh, they have a new model out. Um, so the core tenet is, um, an analyst. If you're actually working in finance, you never show a client or your PM a number that they can't explicitly validate or trace. So when you see something like revenue grew twelve point four percent to four

[2:12](https://www.youtube.com/watch?v=dZgA6btnb_s&t=132s) hundred and eighteen point two billion, these are actual numbers that need to be correct. The challenge is we're all operating off of probability machines. LLMs are at their core probability machines. The same thing that makes them so creative and cool is the same thing that makes them very difficult to use in mission-critical situations. So the natural thing you see is people that attach sources after the fact. Now, this is cool, but here's the problem that happens when you just attach a source after the fact.

[2:42](https://www.youtube.com/watch?v=dZgA6btnb_s&t=162s) First, LLMs don't actually read your documents, despite what many people think. What they do is they chunk a bunch of things together, they do a semantic search, and they retrieve information that may be semi-relevant if you're lucky. So the first big challenge that a lot of people have is we have the wrong source. Like we've pulled information from Seeking Alpha or Vinoo's random YOLO blog that says Palantir will be worth a hundred thousand dollars a share. Or we have the right source and the wrong number. For

[3:12](https://www.youtube.com/watch?v=dZgA6btnb_s&t=192s) example, when you read a ten K or a ten Q, you can imagine a world where I see the number seventeen. Well, what does that correspond to? Is it seventeen million? Seventeen billion? Is it seventeen million yen? Seventeen million dollars? Is it seventeen million was the revenue of this company in the year twenty twenty-three in the ten K that's of twenty twenty-five? Or is seventeen the page number? Like, all of these things are considerations that occur when we're actually working on any kind of

[3:41](https://www.youtube.com/watch?v=dZgA6btnb_s&t=221s) environment that requires verifiability or numerical accuracy. Finally, there's a piece of mental math. Growth rates, margins, multiples. Models are not actually doing math. Now, in some cases, they write code that does do math, but models themselves are not actually doing math in a way that ensures fidelity and accuracy. So what we end up having to develop is an architecture where the model cites the numbers but never writes them. At its core, and the magic

[4:11](https://www.youtube.com/watch?v=dZgA6btnb_s&t=251s) that powers Kepler, is retrieval finds the right source. Thank you, turbopuffer, which we'll talk about. Code does the math, and code meaning the model is not writing the math, but predetermined, uh, deterministic code that we humans have written. And we have a post-processing filter. So let's break this down. So what we did is we processed the entire corpus of SEC data. Now, that's thirty plus million documents. Um, our friends at turbopuffer will be hopefully working with us

[4:42](https://www.youtube.com/watch?v=dZgA6btnb_s&t=282s) fairly soon on how we actually surface this to all of you. Um, but we ran a pre-processing step, and we registered what we're calling citations. And so citations are effectively the deterministic numerical numbers that we have pulled out of filings. Things like revenue numbers, things like an income statement, a balance sheet, things like MD&A that actually says, "Hey, my revenue grew thirty-seven percent," in, like, human written text. There's about twenty thousand of those. There's three

[5:12](https://www.youtube.com/watch?v=dZgA6btnb_s&t=312s) individual filing facts that we've revealed as citable anchors, which effectively means our ability to tell you where a number came from comes down to the ability to be able to click on a number and actually reference back a core document where that came from. When we're running in production, we actually refuse citations that we can't verify. Meaning if I have a number that's not cited, the model is structurally incapable of producing this output. Finally, uh, the number of citations that the

[5:42](https://www.youtube.com/watch?v=dZgA6btnb_s&t=342s) filter has correctly extracted, we got about twenty-one thousand. I'll chat through this in a little bit more detail.

[5:48](https://www.youtube.com/watch?v=dZgA6btnb_s&t=348s) How Kepler works and the piece in Anthropic, if you want more information, Google Kepler Anthropic, you can read about, is the language models themselves are planning. Deterministic software handles every number or even more generally every single fact. So let's break this down. A user asks the question. The LLM then creates a plan with full knowledge of tooling, deterministic retrieval steps, uh, citation filter, bounding boxes, and this whole thing. Now let's actually make this specific. Let's say

[6:18](https://www.youtube.com/watch?v=dZgA6btnb_s&t=378s) I, I'm saying, "What is the adjusted EBITDA of, you know, Google?" Think about what's involved in get-- in answering that question. I have to first retrieve a bunch of facts across 10-Ks arguably over potentially different years. I have to figure out, uh, you know, in an investor presentation and an earnings call what new information was disclosed or if a restatement came out and now I have to actually update a number. All of that is what the model thinks through. From there, we have tools that actually retrieve, compute, build

[6:48](https://www.youtube.com/watch?v=dZgA6btnb_s&t=408s) tables, and register this entire infrastructure into a citation registry. That citation registry is what allows us to not only give magnitude or metadata associated with numbers or facts, but type those facts. So a simple example here. Let's say I do look at, uh, Palantir's twenty twenty-five ten-K. In there it says Palantir's twenty twenty-three revenue was, you know, let's say a billion dollars. All of those numbers that I just

[7:18](https://www.youtube.com/watch?v=dZgA6btnb_s&t=438s) said need to be contextualized. That billion dollars, even though it appears in a ten-K from twenty twenty-five, has to be annotated and treated as twenty twenty-three's data point. I said billion. Well, if you looked at a ten-K or a ten-Q, it usually says one. The billions is up in the table somewhere far away floating around. So we have to somehow take these complex objects and actually give them the annotations to be able to pursue meaningful and traceable output that's actually correct. Now, after the

[7:48](https://www.youtube.com/watch?v=dZgA6btnb_s&t=468s) fact, models like saying stuff. That is what they do. So what happens when a model actually invents a number that we cannot deterministically prove came from a trusted source? We have to have a pros proc-- a post-processing step to actually strip this information out. And so we do this, and we give you the full traceability in the mo-- I'm gonna claim this, I think it's true, in the most granular form of any tool on the market, which means you can click on everything in the Kepler platform, and I'll show you screenshots that

[8:18](https://www.youtube.com/watch?v=dZgA6btnb_s&t=498s) actually allow us to prove this architecture works globally. So, um, let's talk a little bit about how this works in the realm of turbopuffer, who's, uh, been nice enough to give us this stage to have this conversation. First and foremost, look at a ten-K. Let's say we see total net sales. That fact, that four hundred and sixteen thousand dollar, four hundred and sixteen thou... It's a good point, right? We don't know what that means. That four hundred and sixteen thousand number, um, needs to have facts associated with it, things like a value, a

[8:48](https://www.youtube.com/watch?v=dZgA6btnb_s&t=528s) unit, a scale, a period. Now here's the problem. For those of you who work in this space, everyone can say XBRL solves all this issue. We don't need to worry about pulling information out of documents. That's true, if XBRL weren't incorrect like some ten percent of the time. But even more so, how many people are just looking at public information when they're making investment decisions? Fairly few. They have their own OneDrive, their own Bipsync, their own various investment memos that they've

[9:18](https://www.youtube.com/watch?v=dZgA6btnb_s&t=558s) put together, and a slew of internal information. So for our architecture to work, we have to work not just across ten-Ks and ten-Qs, but also on top of your internal data, your SharePoint, your Box, your Egnyte, your OneDrive, your Outlook. So we structure the data where the data's structured, and then we implement search where it isn't. And what that allows us to do is create a namespace in turbopuffer that actually combines both the BM25 and hybrid search with the full guarantees of the Kepler platform.

[9:48](https://www.youtube.com/watch?v=dZgA6btnb_s&t=588s) Here's how we do it, and this is probably the slide. I know people are taking photos. This is the magical slide. I'm not worried about revealing this because this is a very hard problem and it moves quickly. Um, so we use ZM-- uh, ZeroEntropy to, um, zero embeddings. We used twelve eighty dimensions here. Uh, ANN, which is approximate nearest neighbors plus BM25 is how we actually grab information across... Look, everyone's taking photos. This is amazing.  Um, this is how we actually grab this information across our namespace. We do do

[10:18](https://www.youtube.com/watch?v=dZgA6btnb_s&t=618s) reciprocal re-ranking to actually make sure that we pull the right usually sixty K or sixty documents back, and then we do another form of re-ranking. And this is a trusted form of re-ranking where we pull back the sources that we actually trust more than others. For example, different sources will have different pieces of information that claim to be the same thing. An earnings transcript, an earnings call transcript where a presenter says, uh, "My revenue was this," that disagrees with what exists in the ten-K. It's actually

[10:48](https://www.youtube.com/watch?v=dZgA6btnb_s&t=648s) a-- it's not a bug, it's a feature. There's two different pieces of information. I have to actually de-conflict across these. So we need to be aware of that as well. We use a cross-encoder to actually do the re-ranking, and then we return that result to the model. This step in this process is predicated on two things. It's predicated on, one, our ability to rerun embeddings as new information comes in incredibly cheaply. So generating embeddings is not a cheap process. If you can do it and you can do it efficiently,

[11:18](https://www.youtube.com/watch?v=dZgA6btnb_s&t=678s) great. But if you're hosting your own embeddings model, it becomes... Unless you're hosting your own embeddings model, it becomes really expensive and very difficult. So strategy here is we search wide, then we re-rank. Now there's tenant isolation that also becomes really important. Now when we actually hit any kind of a document, we have a bunch of different pieces that are important. For example, it would be very unfortunate if Citadel Desk One saw Citadel Desk Two's research, right? So we have to have security built into the document level as well.

[11:48](https://www.youtube.com/watch?v=dZgA6btnb_s&t=708s) So we-- the first thing we do is if you start at that middle query that goes to the document, um, every hit is reauthorized in Postgres, meaning we check the citation, the value, the document across every one of these computations. Every builder adds a new filter. So we're building up these complex filters over time that actually allow us to make sure that as we're querying information and querying data, we're not leaking data inadvertently across a bunch of different sources. Only Auth itself can create a scoped

[12:18](https://www.youtube.com/watch?v=dZgA6btnb_s&t=738s) token to actually create this information. And of course, the user and aura come from assigned device. So this is the way that we ensure security, and this is the way that we ensure your alpha stays within your firm's ecosystem. Alpha is an interesting question which we can talk about in a world where everyone has access to LLMs as well. Cool. Now, how do we make sure a number can appear in the Kepler platform? First and foremost, we assign unique identifiers to each fact in our platform, meaning we need to have identifiers that

[12:48](https://www.youtube.com/watch?v=dZgA6btnb_s&t=768s) both can do entity recognition and entity resolution. Each one of these source IDs corresponds to a bunch of metadata and a fact stored in a, in a deterministic engine. In order for us to show you the output in a way that's customizable, we need to have a display value. So if you want your whole Kepler platform in thousands of dollars or in yen or in any of these, we need to be able to post-process and show you the actual output. And we have to show you

[13:18](https://www.youtube.com/watch?v=dZgA6btnb_s&t=798s) how we got it and where we got it from. So the invariant in Kepler is you can click on every number, every fact in the platform and see the actual source that we got it from. And that's how we actually give you numbers that are truly citable, deterministic, and reproducible. This is also the idempotent guarantee that we're able to give to people. By having this guarantee, we can run the same query over and over again wherever and get the same results.

[13:46](https://www.youtube.com/watch?v=dZgA6btnb_s&t=826s) So here is what we, uh, have to deal with. When we ran this whole corpus of data, um, a third of all of our citations are the math that the model didn't do. What that means is pulling every one of these citations out, whether it's web results, spreadsheets, whether it's coming from a filing exhibit, the model itself can make up a bunch of things from its own corpus or its own, um, training set that are really, really difficult to work around, especially when you go to historical data.

[14:16](https://www.youtube.com/watch?v=dZgA6btnb_s&t=856s) For example, looking up a revenue number from two thousand and five likely lives within the model's training corpus. But it-- if we retrieve it from that corpus, we can't guarantee that it was pulled correctly from the right year and right scope and that we can actually trace it back. So just off of a whole, a third of this are math that the model-- th- yeah, a third of all citations are the math the models did not do. Now, how do we actually do this live? As tokens move forward and as we're flowing kind of like through doing inference,

[14:46](https://www.youtube.com/watch?v=dZgA6btnb_s&t=886s) um, we give these filters and these references to an LLM that has to actually resolve these. Think of this as the LLM dereferences the pointer. We then have a registry where the sources themselves are looked up, and we pull information out of that registry to give you the traceability. Now, the really powerful thing here is when we don't have a match, we strip that number. That's like a very aggressive thing to do. It means your LLM will not be that wonderful thought partner that you thought it would be, where it would make up,

[15:16](https://www.youtube.com/watch?v=dZgA6btnb_s&t=916s) speculate, and project. This means at this moment, and I'll say this, Kepler is not a super useful tool if you rely on model hallucinations. Meaning if you're trying to be creative or you're trying to, I don't know, write marketing content, Kepler is not the right product to use. That means there are certain industries in finance where opportunistically selling, like banking, probably don't rely on numerical accuracy because your job is to sell an offering. It's not to create a DCF that's backed by accurate, traceable information.

[15:46](https://www.youtube.com/watch?v=dZgA6btnb_s&t=946s) So by checking against our sources, we've effectively created a ground truth retrieval engine across any corpus of data, including public equities as well as your own internal data. Now, this is annoying, and this means the platform cannot be as magically powerful as many people want it to be. So how do we actually ensure whether we should strip a citation or whether we should try to repair it? And so the core insight here is we have a post-processing step that actually tries to do a citation repair,

[16:17](https://www.youtube.com/watch?v=dZgA6btnb_s&t=977s) which effectively means if we find information that we can't reliably trace, should we try to repair that and do an on-the-fly agentic determination that says this information came from this source? The answer is in certain spaces, we actually do do that. And that's probably the second piece of this presentation that we can keep a little bit, um, kinda like wishy-washy, which is how Kepler maintains its IP.

[16:40](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1000s) Cool. Um, now, when we actually run these queries, there are some numbers that I think are worthwhile to share because most people in this industry right now are wrapping a lot of financial research with an LLM, and they're saying, "Great, everything is right. As long as things are, you know, RAGged properly, I should be good to go." So our filter across native, just like running it on Claude or ChatGPT, um, refused ten thousand nine hundred and eighty-one citations from our evals, meaning ten thousand, about I guess three thousand eight

[17:09](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1029s) hundred and ten of these were just straight up wrong numbers. This is the scariest thing ever if you live in financial analytics. This means hooking up Claude to, uh, or OpenAI or any of these models to just a native RAG tool or a vector database actually isn't enough. Now, there's really cool things that you can do if you're looking at just semantic search, and that's it. But if you're looking at anything that requires numerical accuracy and auditability, you actually can't trust regular RAG platforms. This is a scary thing for a lot of startups that

[17:39](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1059s) are effectively ChatGPT wrappers. I'm not saying that in a negative way. It's the reality of where we are right now. So a new kind of architecture has to emerge, and that occur-- architecture has to combine both the auditability and traceability with the deterministic elements of, of course, like numerical accuracy.

[17:58](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1078s) Um, models don't do math. This is pretty simple, and I think it's actually a pretty critical, um, insight here. Uh, any time that we see some kind of situation where the model is, uh, forced to do math or if it can't reason through something, we effectively make that impossible. By having that pros-post-processing step, the model can't actually write something it can't cite. Meaning the model is structurally incapable... And I wanna be careful. The platform is structurally incapable of producing hallucinated numbers.

[18:28](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1108s) The model can still hallucinate, and it definitively does.

[18:32](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1112s) Cool. Let's talk a little bit about what Kepler looks like, and this is a tiny sliver of what our platform does when we show the deterministic traceability and all that. Um, happy to chat through, uh, this is if it's, if it's useful. But a simple question like Apple's 2025 revenue and how fast it grew, every number being clickable means by clicking on these, you see not only the 10-K or the 10-Q or the 8-K or the investor presentation that it came from, but you see the actual calculations that are done behind the scenes. Things like, I did these two numbers, and

[19:02](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1142s) I have a 6.4%, um, increase, all becomes prese-present and traceable on the platform. So this architecture, I think, will become the standard. Um, the opportunity in front of people that are building financial research tools or operating in the realm of AI is right now limited by the lack of architecture that actually supports the workflows you want to do. You can set up fully managed agents right now that try to ask for ideas based on, you know, generated information,

[19:32](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1172s) but you still can't produce a DCF, which is why we still have analysts down the street from here who are working right now on stripping information out of S-1s and taking information from a PDF and putting it into Excel. Our goal is to end that. And so, um, I do wanna end on kind of one very important thing, which is our guarantees do end somewhere. We make models cite numbers, not write them. We verify every citation. Uh, we push arithmetic to the compute layer, not to

[20:02](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1202s) the model layer. We isolate tenants, and we measure recall. This means we can actually guarantee that the ground truth research that you do in your platform is unique and powerful. It does not mean that we can create a full-scale hedge fund that's replacing everyone, at least right now. And so I see the move, and we'll talk about this in, I guess, 30 seconds as we go, uh, to the next place. But, um, the move that I'm seeing is folks will be more quantamental. This is the future. This is the opportunity. The more that we can use unique

[20:32](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1232s) retrieval mechanisms in conjunction with, of course, deterministic retrieval, um, we'll end up in a really powerful place where AI can finally do the stuff that we don't wanna do instead of sitting there and doing the ideating that we do wanna do. With that, um, there's an appendix here, but thank you so much. I appreciate the time.
