How Kepler Built Verifiable AI for Financial Services

Talk transcript

1. 0:00

   [on-hold music] Hey everyone. Thank you for being here. My name is Vinoo Ganesh, and I'm the CEO and co-founder of Kepler.

2. 0:18

   Today, I'm gonna talk to you about how we built verifiable AI for financial services.

3. 0:24

   First, a little about me. Uh, my career has been working in fairly, uh, difficult places to work in terms of numerical accuracy and verifiability. Began my career at Palantir, where I led the compute platform as well as a lot of our USG engagements.

4. 0:38

   Built and sold a, uh, alt data startup. Um, then was head of business engineering at Citadel. I have some Citadel colleagues here in the audience as well. Uh, I've advised a bunch of startups and, um, been lucky that all of them reached pretty positive outcomes.

5. 0:52

   Uh, so this whole talk is a distilled version of a case study Anthropic did on Kepler. If you scan that QR code, uh, we're the only company they've ever done a case study of, which is kind of cool.

6. 1:02

   Um, and so this is a distilled version of that. That has a lot more information about how we were able to do what we do, why, uh, it's been pretty impactful in financial services, and really where we go from here.

7. 1:15

   So my only goal with this whole talk is to convince you that AI is gonna start doing some very powerful things in terms of producing work product. So everything that I tell you is inevitable.

8. 1:28

   Whether it's Kepler or whether it's anyone else, we are already on this journey and this trajectory. Um, so we all better be ready.

9. 1:36

   So I first wanna observe every talk that we've seen in the financial services space so far has been about producing more. Like, how do I token max? How do I get AI to do more?

10. 1:47

    Very little of it is about how to actually make AI trustworthy or produce trustworthy products. And what's interesting is even terms like trust and verifiability have actually been largely abused.

11. 1:59

    Evals are not verifiable. You cannot take a non-deterministic LLM and eval your way to something deterministic. These are probability machines. And the kind of underlying reason for this is that AI has made a writing problem a reading problem.

12. 2:15

    We can produce insane amounts of content, whether that's code, whether it's marketing, whether it's, like, a DCF in record time, but we can't easily verify this. And that's because for years, the hardest part about this whole process was actually producing the work.

13. 2:33

    Edge and alpha came from people like Citadel being able to hire hundreds of analysts who could scour the internet and understand where any source of alpha could exist. So it really came from this idea of being able to consume content.

14. 2:48

    The problem is when a model reads everything, you have the most real version of alpha decay that you possibly can. There is no edge if everyone can look at Tegas and get all the same information.

15. 2:59

    So the hard part now is trusting what actually got produced by the model.

16. 3:04

    And this is kind of funny. This is not necessarily a finance problem. Every system that exists has some form of this. Software, we run CI/CD. We do unit tests.

17. 3:16

    We do integration tests. We have code reviews. When a doctor writes a prescription, a pharmacist fills that prescription. So if it says ten thousand milligrams of a medication, someone catches that.

18. 3:28

    We have a pilot and a co-pilot. We have an EMT that's a primary EMT and a secondary. In finance, we have maybe an overworked VP as a verification layer, but that concept doesn't really exist.

19. 3:40

    And now I'm gonna say something even more aggressive. Uh, the reason that people buy products like Bloomberg and FactSet is to displace culpability. When you buy a tool like that, you know that information is free.

20. 3:52

    It exists in SEC filings. But you believe that because a bunch of contractors or folks overseas vetted this data and stuck it in a central instance, at least if it's wrong, everyone on Wall Street has the same incorrect information.

21. 4:06

    And that's interesting in certain ways. It's also kind of a scary proposition. And that's because every one of these tools is read-only. You as an analyst look at Bloomberg, you look at FactSet, you consume information, and you produce the work product.

22. 4:21

    And that's where the biggest gap and biggest wall to real meaningful adoption across Wall Street has actually been. And so why does this actually matter? We can't use AI properly in this ecosystem.

23. 4:34

    There is a reason that analysts are still working till four AM, and there's a reason folks in investment banks are actually dying because of the hours they're putting in.

24. 4:42

    Because AI can produce a very confident answer, but when it comes to producing any kind of meaningful work product, we're totally lacking. Not only that, we have the SEC, we have the OCC, we have a number of these regulatory agencies who exist to make sure that you are not insider trading or you're not producing a trading decision

25. 5:02

    that can't be justified or backed by some set of primitives or some set of information that you can reliably say, "I made this decision because of these sources of information."

26. 5:13

    And so in this ecosystem, the biggest challenge is: how do I get AI to jump from producing the search technologies that it's doing right now to producing work product?

27. 5:26

    That can mean a fairness opinion. It can mean a DCF. It can mean an investment memo. It can mean looking at every SIM that your firm had five years ago and figuring out why your IRR number wasn't what it should have been or anything else.

28. 5:39

    And so the whole industry has solved this by citing things. When you search the internet with Claude or ChatGPT, it gives you a list of sources it pulled information from.

29. 5:50

    What's ironic is you can't easily curate those sources. So if you find my random Substack that says Palantir is gonna be three thousand dollars a share and you trade off of that, please go do that because it'll be very helpful for me.

30. 6:02

    But, uh, that's not a real vetted source. Seeking Alpha and some of these blogs or Reddit posts, they're data points, but they're not real vetted sources. So showing where you got the information from is only half the battle.

31. 6:16

    And that's where we- we're limited right now. The citation is effectively an after-the-fact audit. Now, a verification is a deterministic, repeatable, numerically verifiable mechanism that we can use to produce validity that a number is right.

32. 6:33

    That was a lot of buzzwords. A verification just means I can prove deterministically this number is right. So when I extract a revenue number from a 10-K, I can deterministically prove that that is the correct number from that 10-K.

33. 6:47

    So these are two sides of kind of the same game. One is showing where you got the sources from, and one is verifying that the information that you pulled out is actually correct.

34. 6:57

    This becomes challenging. It becomes challenging because verification is not a outcome. It lives in the path or the set of steps that are required for you to produce information that is v- valid for your individual firm.

35. 7:15

    Finance is one of the rare industries where two people can be-- have the same information and one can be long a stock and one can be short that stock with the exact same data.

36. 7:24

    And so, the idea of verification is not actually ground truth. It is verifying that you got an output that respects the nouns and verbs or the rules of your organization.

37. 7:35

    If a desk at Citadel-- like, a TMT desk at Citadel believes that a particular stock is going to go up and to the right, another TMT desk at Citadel may have the exact opposite belief, and they may have the same verification mechanisms that produce vastly different outcomes.

38. 7:50

    And so, this really comes down to something simple. What are the sources that we trust? What are the transformations that we apply on those sources to produce information that we care about?

39. 8:00

    And how do we make sure that's codified in a way that makes logical sense? And so, the whole point of this is simple.

40. 8:07

    This came from a ten K is not the val- the validation as a whole. Your job now is to figure out, as an individual, how do I use AI in a verifiable way?

41. 8:18

    And here's the honest answer: you don't. AI is great at doing non-deterministic tasks. It can solve problems in a way that's novel. It can figure out exactly how to do EBITDA adjustments, but it can't be the one responsible for doing the mathematical adjustments because it's a probability machine.

42. 8:37

    It is great at next token prediction. So, the second contention of this whole talk is that you cannot use AI to produce verifiable work product in finance without augmenting it with a deterministic substrate, which effectively means if you're a portfolio manager at Citadel, you have access to a number of deterministic tools that you use to make your

43. 8:59

    trading decisions. We need to model AI like that PM. It needs to exist in grounding. It needs to exist with a certain risk threshold and verifiability. So, here's how we did it.

44. 9:12

    Here's what Anthropic was excited about. We have three tenets that we use to ensure that numerical accuracy is a tenet of the Kepler platform, and that allows us to produce pretty powerful, pretty verifiably reliable information.

45. 9:26

    The first is atomic provenance, and I'll talk through all of these. The second is scope determinism, and the third is derivation chains, uh, which we'll all talk about. But the core premise is this: there are certain things humans should never do.

46. 9:41

    And I will tell you right now, I don't believe a human should sit there and look at a PDF ten K, ten Q, eight K or earnings call transcript and, on one screen, take that number and put it into an Excel model on the other screen.

47. 9:54

    Humans were not built for that. That's not-- Like, there's a reason we don't have databases just written in our heads. And so, let's break this down for how you can actually use AI to produce work product.

48. 10:06

    Let's talk about provenance. Provenance as a whole just means writing down exactly where you got the information. Now, uh, have a lot of respect for everyone else who got here on stage, but we did a mo-- we trained a model that was really good at extracting information.

49. 10:20

    It outperformed foundation models. It was ninety-four percent. Great. It's in the article. Who here would trade off of something that's ninety-four percent accurate?

50. 10:29

    Right. So fine-tuning your way on a probabilistic solution still is-- it's really cool and, like, TechCrunch will be really excited about it, but, like, no one else really cares about it.

51. 10:39

    And that's because a wrong number is still wrong if you're in that unfortunate six percent. So, with atomic provenance, what we do is the model writes effectively a reference to the number.

52. 10:51

    It cannot write the number or manipulate the number in any way. It doesn't even understand what that number is. We have tools that are really good at understanding numbers.

53. 10:59

    They're databases. They are systems that can codify information and read them and write them with appropriate fidelity. So that's the first piece of how we do things. Any time a model makes a decision, it makes the decision to figure out exactly where it got the number from and hand off to something that can write that number.

54. 11:19

    We then run it through a deterministic check, where any kind of a wrong number, if we can't verify it independently, we strip it out. That number will never make it to someone if it doesn't follow the deterministic check, the Provenance ledger, and most importantly, uh, the whole cycle kind of repeating at least a couple of times.

55. 11:39

    So this is not me saying, "Have ten models, and each indivi-- like, have OpenAI check Chat-- check, uh, Anthropic, and have Anthropic check xAI." These are not probabilistic systems evaluating each other's work.

56. 11:50

    There's a core canonical process of extracting a number, persisting it, and making sure that process actually occurred properly.

57. 11:58

    And this is when I say atomic, traditional atomicity in, like, database land. Second, scope determinism. This is-- This was a super controversial idea, like, a year ago, and VCs were like, "This is crazy."

58. 12:10

    Now, this half of this talk has been-- this session has been about this. Um, the model is really good at reasoning and planning. Intelligence is commoditized. GLM five two shows it.

59. 12:19

    You can download it off of Hugging Face right now. You have something as powerful as Opus four eight. Now, what- The model cannot do is math. And why would it?

60. 12:29

    Why would I run one plus one through a multi-billion parameter model instead of one CPU cycle? Unless you're companies that are giving bonuses on people token maxing, which is another hilarious thing.

61. 12:40

    Um, and so what the model does is the model decides what to compute. It never does the computation itself. And so from the Kepler platform perspective, what we do is we split the deterministic pieces of the model, which are none, from the non-deterministic pieces of the model, which are all of them, and we give the model the

62. 12:59

    right tooling and technology to calculate the deterministic pieces. Now let's get really concrete. A model can read something like, "Okay, I need to understand what net margin is. I know the right pieces of information to go to to get that data, but I can't be the entity that's actually running the code behind the scenes to pull that

63. 13:17

    number out of a PDF or parsing the XBRL behind the scenes to pull that information out." That is code. Now, the deterministic pieces of the platform pull that information out and persist it outside of anything the model understands.

64. 13:30

    And with those two together, we can actually produce a numerically accurate answer to the question of, like, what was the net-- what was the net margin of this stock or this company last quarter?

65. 13:43

    This also is a lot cheaper because, again, I don't need the model to do a bunch of stuff it shouldn't be doing.

66. 13:49

    Now, the last piece here is how do we actually do reconciliation? When someone asks about a ratio, like a gross margin or anything else, that doesn't exist in a filing, therefore, I can't just go look up what the gross margin is or what the set of EBITDA adjustments were.

67. 14:06

    The other thing that's really complicated is everyone calculates these ratios and these multiples differently. Everyone does enterprise value calculations differently. Things that are considered recurring or non-recurring may be unique.

68. 14:18

    So not only do we have to codify that in the processes that are run, but we need some kind of a chain of events to figure out what went into producing an individual number and what went into producing an outcome.

69. 14:29

    This is not any different than the chain of events that an analyst does on a desk at a hedge fund to make a risk reward or a trading decision.

70. 14:37

    It's just done by the model in a way that can be replayed and rewound.

71. 14:41

    And so that leads us to something fairly simple here. Uh, we have a system that knows which data points it's allowed to produce, meaning from structured filings, from numerical data, from any other ecosystem, it knows what it's allowed to produce, and it knows what it's never allowed to produce.

72. 14:59

    So if it's pulling things out of pros, raw tables, or anything else, it doesn't do that extraction. What this allows us to do is this allows us to do things like consolidate financial statements in seconds with every number tied back to its individual source.

73. 15:16

    Which means, not picking on any companies, but the CapIQs, the DLUPAs that are all using contractors for this, we don't need to do that anymore. We can actually create a financial model in a numerically accurate way that allows you to build work product.

74. 15:31

    We can build a DCF in the format that you actually want in knowledge the model will not hallucinate that a row exists that shouldn't exist.

75. 15:38

    Now, the really crazy thing and why I say this is inevitable, is everything that I'm telling you generally-- generalizes past finance. We're picking numbers here because finance cares about numbers.

76. 15:50

    But you can imagine a world where every court case, if you're a Harvey or a Legora, runs through the same process. A preprocessing step that understands that we can extract entities like case A versus case B and store that deterministically so we don't hallucinate citations.

77. 16:07

    Or every drug discovery formulation that exists in NIH white papers such that we never miss a compound or anything else. And so the kind of, uh, interesting dimension that we're entering is we're an ecosystem right now where we're almost like pre-SSL in the e-commerce ecosystem, where, like, what's the TAM of e-commerce?

78. 16:30

    Like, trillions. But how many people were comfortable putting their credit card number on the internet before there was security? Zero. So we're in the last step. AI can now produce verifiable work product across a number of industries, meaning the RAG platforms of the past are really, really helpful and really cool in codifying workflows, but there's a reason

79. 16:50

    that the foundational labs are going after every one of these. There's a reason that Claude for Science is not a deterministic system, but still a RAG-based system. And so the piece that should be really exciting about this whole thing is there is a piece of this that no one has built yet in a variety of disciplines, in

80. 17:10

    a variety of industries. We're really good at consuming tokens. In fact, there's a club here for people that consumed a billion-plus tokens. They're walking around with gold cards. It's kind of funny actually. [chuckles]

81. 17:21

    Um, but, like, that's kind of hilarious, right? Like, in what time in history has an employee been rewarded for your company to pay another vendor for how much money you're spending?

82. 17:31

    And so token maxing, I think, is now being thought of as not the right approach here to actually solve your problems. So the natural thing will become a rush to the bottom, which is an optimization problem, which we've seen over and over again.

83. 17:46

    If you rewind time, I actually sat on this stage four years ago, maybe not this particular room, talking about, uh, like, Snowflake Summit and Databricks Summit, where the idea was all of a sudden people want an ROI on top of-- like, to understand the ROI of their Snowflake investment or their Databricks investment.

84. 18:04

    And we started doing cost optimization. We started figuring out how to make sure every dollar of capital we put into Snowflake and into Databricks went to actually producing a pipeline that people were using.

85. 18:15

    That same trend is about to start, and we're figuring out right now how do we use the right tool for the right job, and sometimes you don't need a multi-billion parameter model when one CPU cycle will just work.

86. 18:27

    So kind of wrapping this up, AI has made producing work completely, I'd say nearly free, but, like, thank you VCs, heavily subsidized. The reading problem is still very open, and you verifying a data point is not enough anymore.

87. 18:44

    The system has to be able to trap, trap-- or track its own provenance and actually ensure that the numbers that you're pulling represent your own unique company philosophies. Citations got us, like, fifty percent of the way there, but the next half of the verifiability and provability is going to be how we start using this in real, like,

88. 19:05

    valuable, verifiable work. And the interesting thing here is the work product itself is the proof. In code, we have, you know, unit tests. We have every single pull request and every commit and every code review on that pull request stored in perpetuity.

89. 19:21

    There are companies here trying to mine that information to create a representation of your on- your ontology right now on a company-specific basis. We need that same ecosystem in finance.

90. 19:32

    So let's just say this. All these problems are solved at this point. The last remaining mile is gonna be that personalization. So the second version of this in 2027 is hopefully one of us will be on stage talking about how we are now able to build verifiable ontologies that actually proxy our investment processes instead of saying, "How

91. 19:51

    do I not spend a trillion tokens to solve this individual problem?" Um, so we're also, uh, me and Susanna, we're both from Kepler. Um, we're growing pretty quickly. Obligatory, come join us if these problems are interesting to you.

92. 20:05

    Uh, and yeah, I think we're starting in finance now. We'll be in a lot of different dimensions pretty quickly. Um, so thank you so much and happy to answer any questions. [audience applauding]

93. 20:19

    Yeah, go ahead.

94. 20:20

    I have a question about provenance. So you mentioned there's a provenance ledger where that piece of number comes from. Like, how can it-

95. 20:27

    Yeah. So the question is about provenance and from the provenance ledger, where does the number actually come from? Um, at its core, the number comes from three different things.

96. 20:34

    It comes from extracted information from the filings. It comes from either a mathematical calculation that we do as a derivative production, so, like, a ratio or something else, or it comes from your internal documents or other internal pieces of information that operate in conjunction with that external data to produce that individual data point.

97. 20:52

    So any time the model effectively is responsible for telling some entity to do an IO operation, that's part of the provenance chain.

98. 21:02

    Yeah.

99. 21:02

    Can you talk a little bit about, um, what your customers are most excited about in your product?

100. 21:10

     Yeah. So the question is, um, what are our customers most excited about? It's funny. Uh, everyone wants AI. Like, the dream is the AI portfolio manager. The portfolio managers don't want the AI portfolio manager. [chuckles]

101. 21:22

     Like, they want the AI analyst. And so the thing they're most excited about is a way of rapidly producing, rapidly doing kind of the repeatable, painful tasks that their analysts are doing.

102. 21:33

     Things like, a analyst actually sits there and listens to an earnings call transcript. If we didn't have to have an analyst do that, it would be amazing. Or an analyst sits there with, like, fifteen tabs, and they're opening every 8-K and 10-K over the last, you know, whatever years to create the V zero of a financial model.

103. 21:51

     So they're most excited about getting their analyst time back, which is the honest answer.

104. 21:56

     And these are expensive analysts. Like, these-- Some of these folks make six, 700K a year to do this.

105. 22:05

     Cool. I'm happy to answer any more questions outside as-- actually, yeah, outside as well. I don't know if we're at time. Yeah. The [REDACTED:gender] in the back. [upbeat music]