# Why Wall Street Still Doesn’t Trust AI

Vinoo Ganesh · AI Engineer NYC 2026 · October 14

[Watch recording](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20975s)

[05:49:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20975s) Vinoo Ganesh

[05:49:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20977s) I'm the CEO and co-founder of Kepler.

[05:49:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20983s) This is going to be kind of an interesting talk

[05:49:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20985s) because it comes from what happened three months ago.

[05:49:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20988s) So three months ago, I think there's a little bit of an overlap,

[05:49:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20991s) but I gave a talk at AIE World Fair really around verification

[05:49:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=20995s) and why most institutions need verification

[05:50:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21000s) in some form of the trust ecosystem of AI.

[05:50:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21004s) Well, yesterday, most of you saw Jack and I were on stage talking about why Jack, who ran Citadel's biggest book and started Candlestick Capital, still doesn't trust AI.

[05:50:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21015s) Since that three-month period, there's been about $270 million of VC money that got poured

[05:50:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21023s) into formal verification layers.

[05:50:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21025s) A VC actually took Kepler's own pitch deck and gave it to a competitor and produced something

[05:50:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21031s) that has our same tagline.

[05:50:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21032s) I'm not bitter, just a little.

[05:50:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21034s) But the funny thing is they copied the wrong part.

[05:50:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21036s) And so what I want to do is actually give you, the practitioners today, you can't just

[05:50:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21042s) tack verification on at the end.

[05:50:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21043s) And so when I gave that chat or that talk a few months ago,

[05:50:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21050s) Anthropic had just published an article on how Kepler does what it does

[05:50:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21053s) and why it's really powerful and really unique.

[05:50:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21056s) So what I want to do is get into the technical side of how we do the thing that we do.

[05:51:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21061s) And ultimately, this is driven by one very simple insight.

[05:51:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21064s) Wall Street still doesn't trust AI.

[05:51:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21067s) Most people think that's because of this.

[05:51:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21070s) AI will estimate things.

[05:51:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21071s) It will hallucinate things.

[05:51:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21073s) A frontier model will make stuff up.

[05:51:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21075s) And ultimately, it is, and that's a component of it.

[05:51:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21078s) But the actual core underlying problem is we have a structural problem in this technology

[05:51:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21083s) that we're all so desperate to use.

[05:51:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21086s) Meaning, if you actually chat with people that are building RAG platforms, most of what

[05:51:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21091s) they'll say is, well, the models will eventually get good or get better to the point that they

[05:51:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21094s) will eliminate hallucination.

[05:51:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21096s) That is a nonsensical term.

[05:51:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21099s) You know another word for hallucination?

[05:51:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21101s) Alpha.

[05:51:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21102s) Alpha is how a firm makes money.

[05:51:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21104s) The creativity that you have and that your analysts have

[05:51:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21107s) is a mechanism through which you can be a unique firm.

[05:51:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21111s) Otherwise, there should be only one company on earth.

[05:51:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21113s) So when you ask these types of simple questions

[05:51:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21116s) around where data comes from,

[05:51:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21117s) there is a piece of this that the model will just make things up,

[05:52:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21120s) and there's a piece of this that your firm's internal data

[05:52:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21123s) is never actually reflected in this setup.

[05:52:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21127s) So what are we seeing right now?

[05:52:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21129s) we're seeing everyone with a checkbook

[05:52:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21131s) buy everything they can.

[05:52:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21132s) Every RAG platform,

[05:52:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21134s) some firms are trying to build,

[05:52:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21135s) some firms are running agent pilots.

[05:52:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21138s) I had a customer meeting this morning

[05:52:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21140s) where someone said

[05:52:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21141s) they were actually an AI-first firm,

[05:52:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21143s) which means they have one co-pilot agent running.

[05:52:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21145s) All this stuff.

[05:52:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21146s) So why are we not seeing the complete switch

[05:52:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21150s) of fundamental investing becoming effectively,

[05:52:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21152s) why are we not all Jane Street?

[05:52:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21154s) Why has that not happened yet?

[05:52:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21156s) And so the challenge really comes from

[05:52:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21159s) this idea of in order for you to produce work product

[05:52:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21162s) that you believe in, you have to trust it,

[05:52:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21165s) meaning you have to blame somebody who gets it wrong.

[05:52:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21168s) So we see numbers like this all the time,

[05:52:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21170s) where 80% of finance execs trust numbers,

[05:52:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21173s) but then 31% find those errors.

[05:52:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21175s) So what do we actually need?

[05:52:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21178s) We need a situation where the reviews

[05:53:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21181s) that we're getting for AI apps, as builders ourselves,

[05:53:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21184s) do more than just ragify things.

[05:53:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21187s) And these are quotes we've actually pulled from specific, I don't want to say our competitors,

[05:53:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21193s) but other people, where things demo super well and they behave like a really smart intern.

[05:53:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21198s) Or things are right if you just hook up the right structured information, but that last

[05:53:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21202s) 10% still gets things wrong.

[05:53:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21205s) So what I want to do is, we're lucky, we have great partners and folks like turbopuffer,

[05:53:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21209s) and I gave a similar chat yesterday about this, but how do we actually do the verification?

[05:53:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21214s) How do we architecturally build a system that does not hallucinate numerically?

[05:53:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21219s) Let's talk about it.

[05:53:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21221s) Now, wrong numbers or wrong facts come from three sources.

[05:53:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21225s) Either one, you retrieved the wrong document.

[05:53:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21228s) So I pulled a document, some random thing from somewhere else.

[05:53:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21232s) How do you fix this?

[05:53:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21233s) Well, by retrieving the right document.

[05:53:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21236s) But that in itself is complicated.

[05:53:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21238s) Is the right document my own firm's internal analysis of what this stock is going to be worth?

[05:54:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21243s) Is it, let's say, some analyst who I really trust more than anyone else?

[05:54:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21248s) That's piece one.

[05:54:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21249s) Now, failure two.

[05:54:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21251s) We get the right source, and we pull the wrong number out of that source.

[05:54:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21254s) Let's say we're trying to figure out what the revenue for, I don't know, Palantir was

[05:54:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21258s) back in 2023.

[05:54:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21261s) Well, if I look at the 2025 10K that says Palantir's 2023 revenue was this, I better

[05:54:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21266s) make sure that that number corresponds to 2023 and not 2025.

[05:54:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21271s) So the fix here is really we have a filter that actually checks each citation.

[05:54:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21275s) But this is still a problem because that filter only applies after the fact.

[05:54:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21280s) And formal verification doesn't actually help here because the thing has already been produced.

[05:54:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21285s) Three is the mental math.

[05:54:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21287s) Models don't do math.

[05:54:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21288s) They predict the next token.

[05:54:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21291s) For those of you who are going to say we have deterministic models, great, we do.

[05:54:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21294s) We also have one CPU cycle that's good at adding two numbers together.

[05:54:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21298s) Why don't I use that CPU cycle?

[05:55:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21300s) So third failure is let's use deterministic tools that we already have to actually compute

[05:55:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21306s) numbers and entities rather than relying on some external tool.

[05:55:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21311s) So here's what we do.

[05:55:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21312s) This is the magic behind Kepler, at least for piece one.

[05:55:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21315s) There's a whole bunch of other stuff.

[05:55:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21316s) We are architected fundamentally differently.

[05:55:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21319s) Code writes every number, and what code does and preprocessing does is we assign pointers

[05:55:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21324s) to unique identifiers, meaning facts, meaning something like a number in a 10K.

[05:55:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21330s) This operates in a giant preprocessing and contextualization step,

[05:55:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21334s) such that all of this exists, going from unstructured data to a structured data store.

[05:55:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21339s) Then the model is effectively a smart analyst.

[05:55:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21342s) It figures out how to go about solving the problem,

[05:55:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21346s) but not actually being the thing to solve the problem itself.

[05:55:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21349s) So it plans.

[05:55:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21351s) It says I'm going to go here and retrieve this source and retrieve this entity and do

[05:55:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21355s) this thing.

[05:55:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21356s) Code and math do the retrieval.

[05:55:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21358s) By actually pulling a number out of a 10k at search time, that's a hugely expensive

[05:56:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21365s) token cost.

[05:56:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21366s) And Anthropic and OpenAI should thank you for paying all those tokens and token maxing,

[05:56:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21369s) but most of your firms don't like that, at least anymore.

[05:56:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21372s) I know there were people getting bonuses from token maxing for a while.

[05:56:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21376s) Now, the next piece of this is really, once the model is actually read both these pointers,

[05:56:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21381s) it can't structurally generate numbers.

[05:56:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21385s) Meaning, unless the model actually has a mechanism of dereferencing that pointer,

[05:56:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21390s) that number or that fact cannot be generated in Kepler's output.

[05:56:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21394s) Now, we do run a post-processing step, which is the verification step.

[05:56:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21398s) And this is the piece that was kind of funny, where the last company just copied the verification piece,

[05:56:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21403s) but all the structural stuff beforehand didn't make it.

[05:56:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21406s) So what you as a consumer actually see

[05:56:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21408s) is you see something like the revenue was this amount to this amount,

[05:56:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21412s) and you physically can click on these chips

[05:56:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21414s) and see exactly how we got the information.

[05:56:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21417s) Now, in my past life, I had an opportunity.

[05:56:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21419s) I was head of business engineering at Citadel.

[05:57:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21421s) So I worked across the firm to figure out effectively

[05:57:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21424s) what do we need to produce software that helps us generate alpha.

[05:57:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21428s) You know the crazy thing with Citadel?

[05:57:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21431s) PMs are so talented and so smart,

[05:57:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21433s) but they work when they're paired

[05:57:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21435s) with deterministic tools and technology.

[05:57:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21437s) That's kind of the magical insight

[05:57:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21439s) behind fundamental investing.

[05:57:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21441s) Fundamental investors are so good at understanding data,

[05:57:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21445s) but that data has to be right.

[05:57:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21447s) And so when you build a system like Kepler,

[05:57:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21450s) what you effectively have to do

[05:57:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21451s) is look up the right information

[05:57:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21453s) and then be clever about the information you don't have.

[05:57:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21456s) So let me give you the parallel.

[05:57:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21458s) Most folks and most buy-side researchers or buy-side investors

[05:57:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21461s) try to use tooling or build DCFs on their own,

[05:57:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21465s) and then they find and search for information they don't have.

[05:57:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21469s) They schedule GLG or guidepoint calls.

[05:57:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21471s) They find experts.

[05:57:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21473s) They speak to management.

[05:57:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21474s) The same system actually applies here.

[05:57:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21477s) So what we do is we pick the path,

[05:57:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21479s) and we store the information deterministically

[05:58:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21481s) that we can actually store deterministically.

[05:58:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21484s) When we have information we can't store deterministically,

[05:58:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21486s) we run similar pre-processing steps to grab information,

[05:58:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21490s) store that information in a hybrid storage mechanism,

[05:58:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21493s) which is a combination of turbopuffer,

[05:58:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21495s) who's awesome, by the way, and a bunch of other mechanisms,

[05:58:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21498s) and then we ultimately use a combination of re-ranking

[05:58:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21502s) with literally old-school pointer dereferencing dot get methods

[05:58:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21507s) to pull things into our system.

[05:58:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21509s) Now, why this matters is a lookup cannot get it wrong.

[05:58:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21515s) Search can.

[05:58:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21516s) If you're running the same 10 searches with the same query,

[05:58:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21520s) the probabilistic nature of models means there's inherent entropy.

[05:58:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21523s) That's actually an opportunity.

[05:58:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21525s) So that's what we've changed here.

[05:58:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21528s) Now, how we do this is we break this down,

[05:58:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21531s) and this occurs across the SEC world as well.

[05:58:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21534s) We are frequently throwing candidate documents away,

[05:58:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21538s) meaning we consistently have to look at

[05:59:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21540s) not only what source is the right source for what information,

[05:59:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21543s) but also which documents should we stop looking at kind of globally.

[05:59:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21548s) So we're on ANN plus BM25 to get back a set of documents,

[05:59:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21552s) usually up to 200.

[05:59:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21554s) We do reciprocal.

[05:59:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21556s) We effectively get 60 documents back.

[05:59:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21558s) It's probably the best way to think about this.

[05:59:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21560s) Then we do cross encoders.

[05:59:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21562s) We try to figure out exactly the query and the actual document

[05:59:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21566s) chunked together and get those results that way.

[05:59:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21570s) What that allows us to do is we can search really wide.

[05:59:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21574s) You know what LLMs aren't good at doing?

[05:59:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21576s) Searching really wide.

[05:59:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21578s) What they try to do is create a plan

[05:59:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21579s) and find either the fastest or easiest

[05:59:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21582s) breadth-first search mechanism,

[05:59:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21584s) actually maybe depth-first search mechanism,

[05:59:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21586s) of actually getting the first document and moving forward.

[05:59:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21589s) So we allow this to search really wide

[05:59:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21590s) and then narrow from there.

[05:59:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21593s) The reason LLMs don't like searching wide

[05:59:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21596s) is because it's expensive.

[05:59:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21597s) It's because looking across a bunch of

[06:00:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21600s) different sources is significantly worse than getting the reward function back with your first hit and going from there.

[06:00:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21608s) Okay. So I want to talk a little about security here because it's super important. Kepler was built with a security-first mindset and security-first mentality. Now, client documents, and this is specifically targeted towards the builders here, this is the thing that no one's chatting about.

[06:00:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21625s) Everyone is talking about the security of what OAuth token is used and what user-scoped

[06:00:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21630s) mechanism of retrieving documents.

[06:00:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21632s) Which is actually pretty scary.

[06:00:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21633s) So let's talk about how things go wrong in these systems that people are trying to build

[06:00:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21637s) and how you can fix it.

[06:00:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21639s) First and foremost, you forget to filter the tenant.

[06:00:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21642s) All of a sudden Citadel can search for all of Millennium's documents.

[06:00:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21644s) You will be in a lot of trouble.

[06:00:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21646s) Second, a caller claims to be another firm.

[06:00:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21649s) Meaning I can claim that I'm impersonating this other entity and querying documents that

[06:00:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21653s) way.

[06:00:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21654s) Third, search returns information that's out of date.

[06:00:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21658s) We've had this problem kind of globally

[06:00:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21659s) and even Google search results.

[06:01:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21662s) Fourth, direct read skips the search rules,

[06:01:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21664s) meaning you're not actually querying

[06:01:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21665s) the full corpus of information,

[06:01:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21667s) or there's not an allow list.

[06:01:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21668s) You only get back the single document

[06:01:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21670s) that you did the search.

[06:01:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21671s) Five, you hit an index that isn't populated.

[06:01:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21674s) The way to actually solve all these problems,

[06:01:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21677s) and there's individual things listed here,

[06:01:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21678s) is actually scoping searches with security built in.

[06:01:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21683s) That's another critical piece that I think a lot of people miss when they're trying to

[06:01:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21687s) build these systems themselves.

[06:01:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21689s) Inevitably, people will start caring about the security associated with querying this

[06:01:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21693s) information.

[06:01:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21694s) Alpha is the most elusive thing out there, especially for folks on Wall Street.

[06:01:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21698s) Making sure you have a good mechanism of constraining the set of documents that come

[06:01:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21703s) back with each query is really important.

[06:01:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21708s) Now let's talk about verification, which is the last piece here.

[06:01:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21710s) This is probably what we're known most for, but we do the other stuff I mentioned as well.

[06:01:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21716s) How we do this is by preprocessing information and storing it deterministically,

[06:02:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21722s) we have systems like code that actually write individual pointers.

[06:02:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21727s) The model can only then read those pointers and grab information back if they exist in the database.

[06:02:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21733s) If I grab something out of a hash map that doesn't exist, I don't get anything back.

[06:02:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21737s) And so this combination is what allows us

[06:02:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21740s) to actually make sure the information that we retrieved

[06:02:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21743s) actually came from the source that it came from.

[06:02:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21745s) Now, this is a kind of hidden version

[06:02:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21748s) of what our source IDs look like,

[06:02:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21750s) but these are hashes,

[06:02:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21752s) and these are very simple things that you can do

[06:02:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21754s) in your own system and your own setup

[06:02:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21755s) to ensure that you're pulling back facts

[06:02:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21758s) that, in fact, do exist.

[06:02:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21761s) Cool.

[06:02:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21762s) So that's a little bit about how we do verification.

[06:02:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21764s) Now, I want to chat a little bit about

[06:02:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21766s) what this enables kind of globally.

[06:02:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21770s) This enables us to verify facts

[06:02:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21773s) that come out of our grounded corpus of data.

[06:02:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21776s) Now, you as a builder, or you as a startup creator,

[06:02:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21779s) or you as someone who is trying to figure out

[06:03:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21781s) how do I trust my outputs,

[06:03:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21783s) really just need to do a couple of things.

[06:03:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21785s) You need to show people that as you're creating these agents

[06:03:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21788s) that are theoretically looking for alpha

[06:03:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21791s) or whatever they'd be looking for,

[06:03:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21792s) you're using information that is actually relevant and concrete

[06:03:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21795s) and, most importantly, accurate

[06:03:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21798s) to the set of people that you're trying to service.

[06:03:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21800s) So we end up with a contract

[06:03:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21803s) where citations are either verified,

[06:03:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21806s) corrected at search time,

[06:03:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21808s) or flat-out refused.

[06:03:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21810s) This is crazy if you think about it.

[06:03:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21813s) If a model cannot de-resolve that pointer,

[06:03:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21816s) it cannot print that number out.

[06:03:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21820s) So that's kind of an insane thing.

[06:03:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21822s) It means that the whole, and not just a number,

[06:03:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21824s) let's talk about facts,

[06:03:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21825s) Like, we see these court cases all over of precedents being hallucinated.

[06:03:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21830s) Well, why can't we create a system that just has the precedents stored deterministically

[06:03:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21834s) and allows us to search through that list and not have a random court case that never existed

[06:03:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21839s) all of a sudden start existing?

[06:04:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21841s) So this mechanism and this technology actually becomes fairly globally applicable.

[06:04:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21846s) And the reason I'm on stage telling you how we do what we do

[06:04:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21849s) is because I'm terrified that Wall Street never trusts this technology

[06:04:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21854s) because of the fear that so many other companies

[06:04:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21857s) or so many build-it-at-home folks

[06:04:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21859s) have been instituting in these organizations.

[06:04:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21863s) It's gotten to the point where

[06:04:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21865s) the vast majority of finance AI tools right now

[06:04:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21868s) are being used to format PowerPoint presentations.

[06:04:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21871s) We still haven't actually built a DCF cradle-to-grave,

[06:04:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21874s) despite what most people are saying.

[06:04:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21875s) And if we had, every single company on Wall Street

[06:04:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21878s) would have adopted that overnight.

[06:04:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21880s) So this is my fear.

[06:04:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21881s) My fear comes from a very simple fact.

[06:04:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21883s) For us to actually get value out of this technology,

[06:04:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21886s) we need to be able to build technology

[06:04:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21888s) that people actually can trust and can verify.

[06:04:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21892s) That isn't just true in finance.

[06:04:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21893s) It's true in a bunch of different industries.

[06:04:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21895s) So please, take what you need to from this presentation

[06:04:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21897s) and get people using this technology.

[06:05:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21900s) Otherwise, we're all going to be out of a job.

[06:05:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21902s) Now, I want to chat through what happened when we saw this.

[06:05:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21905s) So when we went through this whole process,

[06:05:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21908s) this is actually some of our real production data here,

[06:05:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21912s) refusals are when the model

[06:05:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21914s) is not able to produce a number.

[06:05:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21918s) Meaning the model will say,

[06:05:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21919s) I couldn't dereference this pointer,

[06:05:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21921s) therefore I could not produce this number.

[06:05:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21924s) We saw that drop super drastically.

[06:05:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21927s) Now the reason that it didn't drop to zero

[06:05:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21929s) is because models are still probabilistic.

[06:05:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21932s) Meaning it can still make up numbers

[06:05:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21934s) after the fact during inference.

[06:05:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21936s) So there is another post-processing step

[06:05:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21939s) after this that we have as well.

[06:05:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21941s) But my point with this slide is just by creating a system

[06:05:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21945s) where the models have to dereference their own pointers,

[06:05:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21947s) you can actually create a very powerful, very verifiable AI platform.

[06:05:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21953s) And that to me is my ask to all of you.

[06:05:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21955s) As you go through the building process,

[06:05:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21957s) as you go through the scoping process,

[06:05:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21958s) any tool that you use,

[06:06:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21960s) ask them if you can actually verify the numbers.

[06:06:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21962s) Ask them if you can see where the sources are,

[06:06:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21965s) not in terms of some after-the-fact annotation,

[06:06:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21968s) but click on the number, whether it's a calculation,

[06:06:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21970s) whether it's an extraction or some combination of the two

[06:06:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21974s) to make sure you trust the insight.

[06:06:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21976s) And all of this ultimately is going to be the unlock.

[06:06:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21980s) So you, as a vendor or you as a builder, ask these questions.

[06:06:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21985s) Does the model actually write pointers?

[06:06:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21987s) Does it actually create a mechanism for verifying the output?

[06:06:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21990s) Is every citation source checked?

[06:06:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21992s) Is arithmetic source checked?

[06:06:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21994s) Do you have a mechanism for stopping files

[06:06:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=21997s) that reach clients or customers that it shouldn't reach?

[06:06:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22000s) And most importantly, what happens to a number that came from nowhere?

[06:06:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22004s) What happens when a fact comes from nowhere?

[06:06:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22007s) And how do I actually handle these kinds of hallucinations?

[06:06:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22011s) So this whole platform is actually to make a very simple statement, very obvious,

[06:06:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22018s) which is we can have the smartest models in the world.

[06:07:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22021s) We can hit some crazy cool piece of technology,

[06:07:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22024s) but we still have the same database problems that we've had from the beginning.

[06:07:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22030s) which are, I cannot, if I search through a row

[06:07:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22035s) and that value doesn't exist, I won't get a retrieval back.

[06:07:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22038s) We're now out of the search phase of AI.

[06:07:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22041s) We need to be building infrastructure.

[06:07:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22043s) We need to be building scaffolding and harnesses and substrates,

[06:07:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22046s) or whatever the cool VC term is these days,

[06:07:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22048s) that actually enable us to make use of this technology.

[06:07:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22051s) So this is my challenge and my ask to all of you.

[06:07:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22055s) I hope you go out there and build something compelling,

[06:07:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22057s) but I hope you build something compelling that you can trust.

[06:07:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=22060s) Thank you.