AI Engineer World's Fair 2026
Why Off-the-Shelf AI Doesn't Understand Money
Read the talk
Why Financial AI Needs Outcomes, Not Just Context
Financial advice can sound convincing while ignoring the constraints that keep a business alive. Udi Menkes shows how observed actions and outcomes can ground better recommendations.
From a talk by Udi Menkes
Before you start: Familiarity with LLMs and basic business concepts such as profit, revenue, and cash flow is helpful; the causal-inference concepts are introduced as they arise.
Just buy another car
When Udi Menkes scratched his car while parking, his three-year-old daughter asked what had happened. He explained the problem. Her solution was immediate: “Just buy another one.” It solved the visible problem without considering what replacing a car would cost.
Financial advice from an LLM can have the same appealing simplicity. Menkes asks the audience who has used one for financial advice; many hands go up. When he asks who trusted the answer enough to follow it, many come down. His own experience involved choosing between a real-estate investment and a stock-market niche. He supplied his finances, relevant books, extensive context, and the latest models. The recommendation came back with convincing reasoning—until a small change in an assumption flipped it from A to B. Another small change flipped it back. The explanations remained persuasive while the judgment became difficult to trust.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fix the first property's economics before buying another
Menkes introduces an Intuit study covering thousands of businesses and roughly 100,000 situations and time frames. Its first example is a new landlord with one rental property, negative cash flow, an outstanding loan, and profit trending into the red. Asked how to improve profit, a frontier model recommends acquiring a second property: more rental income would compensate for the existing deficit.
The model already had the business's data. The missing ingredient was therefore more specific than another financial statement in the prompt. Menkes contrasts that recommendation with one grounded in similar businesses, the actions they took, and what happened afterward: raise the existing tenant's rent by 5–10% before renewal. He says that change would move this particular case into profitability. The distinction is between adding another commitment while the business is under strain and improving the economics of the commitment it already has.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A price increase can put the whole business at risk
The next business is an egg supplier. One customer accounts for 70% of its revenue, and one vendor accounts for almost all its costs. The question is again how to improve profit, but the concentration changes which action is sensible.
| Recommendation source | Proposed action | Constraint it addresses or exposes |
|---|---|---|
| Frontier model | Raise the main customer's prices 15–20% | Risks losing the dominant revenue source |
| Model grounded in outcomes | Negotiate vendor pricing down 5–10% | Targets the dominant cost source |
The second recommendation shifts attention from revenue to costs. It does not simply produce a more detailed version of generic growth advice; it selects a different action because losing that customer could damage the whole business. Both examples concern frontier models that, Menkes says, had the business context available.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Confident recommendations and the long game
Menkes calls this failure mode the fluent bluff: generic, confident financial advice learned from books, blogs, and advice columns without grounding in what happened when businesses acted. He extends that criticism broadly across financial answers. In the Intuit research he describes as forthcoming, 40% of recommendations centered on acquiring customers and another 14% on increasing product revenue. Together, those categories account for more than half the advice—a narrow repertoire for businesses facing very different constraints.
The problem becomes more visible when decisions compound over time. Menkes describes a recent Princeton study in which models received tools and business data, $1 million in starting capital, and a 500-day simulation horizon. He reports that most drove their companies bankrupt before the horizon ended, that a simple rules-based system beat almost all of them, and that even models with profitable runs could fail on reruns.
The study appears to be CEO-Bench: Can Agents Play the Long Game?, which operates a fictional subscription-software company and scores ending cash. That makes two distinctions essential: finishing below the starting balance is different from bankruptcy, and the original paper reports each model's best of three runs rather than a typical run. Later simulator changes also mean the current leaderboard is not the historical comparison described here. The narrower engineering question survives those distinctions: can an agent preserve sound business judgment across a sequence of consequential decisions, rather than produce one convincing answer?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A complete company snapshot is still not experience
Company context describes a business; experience connects situations, actions, and consequences. Even a complete QuickBooks record—general ledger, profit-and-loss statement, cash flows, and invoices—remains a collection of data points about one company. Reading those records is not the same as observing how comparable businesses responded to comparable decisions.
This distinction shapes Menkes's product mandate. He describes 15 years in AI and finance, moving from AI science and team leadership into product management, then joining Intuit about four years before the talk. As a principal product manager leading financial intelligence and advisory systems, his recurring question is what Intuit possesses that access to the latest model cannot replicate—and how to turn it into useful customer experiences.
An apparel company supplies another example. One vendor represents 80% of costs while enabling 97% of revenue. A textbook instruction to cut the biggest cost overlooks the dependency: removing that supplier could remove almost all the business it supports. An expense is not necessarily waste merely because it dominates the ledger.
The human analogy is choosing an advisor. One candidate has spent years guiding businesses through real decisions; another is exceptionally smart and has read all the textbooks but has no practical experience. The value of the first advisor lies in recognizing how an apparently sensible action can unfold under actual business constraints.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What did raising prices actually cause?
Experience is difficult to measure. Suppose a restaurant raises prices and becomes more profitable six months later. Did the price increase cause the improvement, or was the restaurant already on a successful trajectory? You cannot run the same business twice under identical conditions to observe both alternatives. Menkes describes comparing groups of similar businesses with similar propensities to raise prices, separating those that did from those that did not.
In the restaurant illustration, daily profit gains are $4,200 for businesses that raised prices and $2,800 for those that did not. The raw difference is $1,400 per day. That arithmetic is straightforward:
python
profit_gain_per_day = {
"raised_prices": 4200,
"did_not_raise_prices": 2800,
}
raw_gap = (
profit_gain_per_day["raised_prices"]
- profit_gain_per_day["did_not_raise_prices"]
)
print(f"Unadjusted difference: ${raw_gap:,}/day")
But the difference does not by itself isolate the action's effect. Businesses able to raise prices may already be more successful; part of their subsequent improvement may reflect that advantage.
Menkes gives about $1,150 per day as the illustrative adjusted effect, rather than the raw $1,400 difference. He calls the relevant quantity CATE, the conditional average treatment effect: the expected effect of an action for businesses with specified characteristics. The talk does not supply the estimator or adjustment needed to reproduce $1,150. Interpreting observational comparisons causally also requires assumptions about which factors influence both the action and the outcome; finding similar businesses alone does not establish causation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate candidates, then evaluate them against outcomes
The useful data asset is a repeated pattern across entities: a situation, an action taken, and a subsequent outcome that can be verified. Menkes does not discard frontier models. He gives them a specific role: generate hypotheses and candidate actions. A model trained using reinforcement learning then evaluates which candidates are promising and which could damage the business. Generating an option and judging its consequences are separate jobs.
The training material comes from business trajectories. Menkes describes millions of them, built from Intuit data across QuickBooks, TurboTax, Credit Karma, and Mailchimp. The construction proceeds through three kinds of information:
- Build the state. Summarize a business at a particular point in time using its general ledger, profit-and-loss statement, cash-flow statements, and invoices.
- Derive the action. Infer what the business did from its records. A ledger may reveal investment in a marketing campaign, payments, payroll, or hiring costs.
- Measure the outcome. Observe subsequent profit, revenue, and cash flow over different time frames, including combinations of those measures.
These records become state–action–outcome vectors. The RL model learns which actions lead to better outcomes in situations involving similar businesses.
A subsequent training step teaches an LLM to generate the improved advice. The accompanying slide makes the sequence explicit: generate candidates, score them using an RL model, and GRPO-train the advisory model. The financial records therefore serve a different purpose from documents retrieved into a prompt: they supply evidence for evaluating candidate actions and training the system that produces recommendations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a trained model to an inspectable recommendation
The outcome-grounded recommendations in the opening examples came from a model developed with Intuit researchers using data across millions of small and medium businesses. Menkes reports that a cheaper, mid-sized model outperformed leading frontier models in head-to-head tests. He does not provide model identities, evaluation metrics, or performance margins, so the result is a reported finding rather than a reproducible comparison. Its product implication is that proprietary outcome data can matter more than access to a larger model.
At the time of the talk, Intuit had turned this work into an AI business advisor in a customer research preview. The advisor proactively surfaces opportunities, then explains why it recommends them: which businesses resemble the customer, what those businesses did, and how that experience informs the proposed action. Users can drill into the rationale and create action plans. The interface makes the reasoning available for inspection before a recommendation becomes a business decision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ground the decision—and include the person making it
The same design question applies beyond finance: how can AI be steered toward the outcomes a domain actually values? Menkes points to anti-fraud systems, healthcare, logistics, and developer tools. He expects the advantage to accrue to organizations with strong systems of record that can derive distinctive datasets and train models against domain outcomes. Model selection remains part of the system, but the records determine which experiences it can learn from.
Grounded science is only part of a trusted advisory experience. A good advisor also understands a person's preferences and dislikes, and involves that person in the decision. An action can be well supported by comparable business outcomes without being the action a particular owner wants to take. The product must connect its evidence to the user's priorities and preserve their participation in choosing what happens next.
Coding provides a useful qualification to the claim that models lack experience: coding agents already make substantial use of verified outcomes. Menkes describes finance and other domains as comparatively unexplored. His proposed direction is to embed what worked at scale into training, rather than expect a larger model alone to close the gap.
The practical starting point is inside the existing records. After a brief invitation to enjoy the Fourth of July weekend, Menkes asks builders to look for situations repeated across entities, identify the actions those entities took, and find outcomes they can verify. That is the material from which an advisory system can begin learning what follows a decision—not merely how advice about that decision is usually phrased.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Original study of agents operating a simulated software startup, including business mechanics, evaluation conditions, and repeated-run results.
Further reading
Simulator code and instructions for running agents and analyzing their business trajectories.
A worked Python example of estimating heterogeneous treatment effects, stating causal assumptions, and computing uncertainty.
Updates since the talk
Benchmark overview, updated results, and trajectory exploration, including changes made after the conference.
Read the complete timestamped transcript
- 0:00
[on hold music] So I have a [REDACTED:age] daughter, and she's absol-absolutely adorable, and the parents here in the room know how insightful that age can be.
- 0:23
And she has a complete theory about money by now, and I'll give you an example. So a couple weeks ago, I was driving the car, I was coming in to park, and there was something in my dead end, and I scratched the car.
- 0:36
I went out, I'm like, "Oh, man, I can't believe I scratched the car." And then I hear my daughter from the back, and she was like, "Daddy, what's happened?"
- 0:44
And, you know, I'm explaining it to her. And then she says, "What's the problem? Just buy another one."
- 0:52
So anyway, um, I wanna ask you today with a raise of hand, who uses LLMs, has used LLMs for getting financial advice, a recommendation on something in the financial world?
- 1:08
Great. Almost everyone. Wait, keep your hand up if you trusted the answer and you actually followed the advice.
- 1:19
Okay. A lot of hands are going down, and that's the core problem. So I had the same thing a couple, a couple, uh, months ago. I had a big decision I was looking to take.
- 1:31
Should I invest in, you know, real estate niche or in the stock market niche on a specific area? And, you know, I do AI for finance for a living, so I went the full-blown way, context, brain, um, the books, the knowledge, all my finances combined, all the latest models, and it gave me a recommendation, "You should do
- 1:53
A with great reasoning." And then I changed just a little bit one of the assumptions, and it completely flipped. "You should do B. Never do A." And then I tweaked one more small thing, and it went all the way back to A.
- 2:10
And at that moment, I understood that the advice sounds good, it sounds sound, but I can't really trust it. And by the end of the talk today, you will understand why off-the-shelf LLMs don't understand money and what you need to do about it.
- 2:32
So I'm gonna show you a couple of real examples from a study we're doing at Intuit on thousands and thousands of businesses, around a hundred thousand situations and time frames.
- 2:44
This is an example of a small business of a new landlord that is building a rental property business. His first property, and he's down. He's a, he's in a negative cash flow.
- 2:55
There's an open loan, and the profit is basically trending into the red. And a question comes up, "How do I improve my profit?" And a frontier model gives the following response: "Go and acquire a second rental property because that'll bring more income and compensate for the deficit."
- 3:17
And that model had all of the business's data. Now, that's very risky for someone in the negative, in the red to be doing. On the other hand, a model that is grounded in real outcomes, and what I mean by that is a model that has seen similar situations of such businesses, what they did and what was the
- 3:38
outcome, actually recommended to raise prices on the existing tenant by five to ten percent,
- 3:47
and to do it before the renewal. Now, some of you are thinking that it's just a matter of context. Just give it more context. And the thing is that this advice is coming based off on real situations of similar businesses and would actually move them into profitability in this case.
- 4:10
And this is not a one-off. I can go on and on showing you a lot of examples. This is second one. This is an egg supplier where one customer is seventy percent of the revenue and one vendor is almost all of its cost.
- 4:23
Same question: How do I improve my profit? A frontier model says, "Raise your prices fifteen to twenty percent on that customer."
- 4:34
Now, you understand that is very risky because you might lose almost all your revenue. On the other hand, the same grounded model and real outcomes went actually to the cost side and recommended to negotiate, um, negotiate the vendor cost pricing for a five to ten percent reduction.
- 4:55
So it went to the cost side. It took into account the constraints. So what we saw are two examples of frontiers, I'm talking about leading models in the world today that had all the business context, actually give advice that could be very harmful for the business.
- 5:14
And that's what I call the fluent bluff.
- 5:19
The fluent bluff is a generic, fluent, and confident answer that frontier LLMs can give you around money because of what they learn on the Internet, blogs, books, advice columns, what people wrote about money, but not based on what actually happened.
- 5:42
And I would argue that almost every answer that you see related to money and finances is such. And we're soon gonna release res-- the research I talked about, but I'll give you a highlight from there.
- 5:59
On-- Across these hundred thousand businesses and time frames-
- 6:02
40% of the time, the essence of the advice that the frontier models were giving was acquire a new customer,
- 6:11
which is, you know, everyone would, would wish they could do that. And 14 more additional percent were increase basically the revenue from your product. So combined, more than half of the essence of the advice that was given by the frontier models was acquire new customers and try to increase revenue from the profit.
- 6:36
And this is not just me saying, this is a very interesting research coming-- that just came out a couple weeks ago from researchers at Princeton. What they tried to do is simulate and answer the question, can the leading models drive long-horizon business decisions?
- 6:54
And what they did is they gave the models a harness with tools and data and everything they would need to take decisions across a simulation of five hundred days.
- 7:04
Can they turn a profit? Each model got a million dollars to start with, and guess what happened? Most of the models drove the company bankrupt, and it didn't even take five hundred days.
- 7:20
And the interesting part is that they also ran a simple rules-based system, and that rules-based system out-beat almost all of the models. Even the very, very few models that were able to generate some profit, it was in a specific ins-instance, and when you rerun that, they also actually drove bankruptcy.
- 7:43
So how can it be that a simple rule-based system out-beats the frontier models today on real business decisions?
- 7:55
And here's what I want you to think about. A frontier model has read about money,
- 8:03
but a grounded model and real outcome has actually watched what happens.
- 8:10
And let me be precise with my argue here. Even if you take all of a company's data, and for example, we at Intuit have all the financial data from QuickBooks, for example, the general ledger, the P&L, the cash flows, everything, the invoices of the business, and you give it to a frontier LLM, it's still just one group
- 8:28
of data points on a company. And that's the difference between sounding right and actually being right.
- 8:38
So I'm Udi Menkes, and I've been in the AI and finance world for the past fifteen years. I started in the AI science world, leading AI and data teams, and shifted into product management.
- 8:52
Becoming in, uh, about four years ago, an AI prins- AI product manager at Intuit long before, by the way, it was cool to become an AI PM. Um, and today, I'm a principal product manager at Intuit.
- 9:03
I lead financial intelligence and advisory systems that help Intuit's customers to take better decisions and grow their business. And the question that I fixate on on a daily basis is not which model is the best now that I, I can use.
- 9:21
It's what do we fundamentally have that no model access can replicate? And how can we-- can I transform that into AI-native, transformative, and delightful experiences for our customers?
- 9:37
Now, I wanna develop some intuition for-- from three different angles on why these models bluff. So the first angle is around context is not experience. And I'll give you another real example.
- 9:50
This is an apparel company where eighty percent of the cost is coming from this one vendor. And the textbook answer is, "Go cut your biggest cost," right? Textbook book answer.
- 10:04
But the issue here is you cut this cost, that same vendor was actually enabling the generation of ninety-seven percent of that company's revenue. So cut that biggest cost,
- 10:17
and you lose almost all the revenue, and that can be great margins on zero dollars.
- 10:25
And think about it. If I give you an option to work with two, two different advisors. One advisor is a very experienced one, years of experience working with businesses, guiding them.
- 10:39
And another advisor, which is very, very smart. They know all the textbook. They're fresh. They read everything. They know AI in and out, but they don't have experience. I would bet you would always go with the experienced one.
- 10:54
And that's the same thing with the models in what I just showed you.
- 10:59
And it turns out experience is very hard to measure,
- 11:03
and I'll give you an example. So let's look at a restaurant. A restaurant, let's say, raises price, and after six months becomes much more profitable. Is it because they raised prices, or is it because they're just naturally successful?
- 11:20
And the challenge is, obviously, you can't run the business twice, right?
- 11:24
So what you do is you take two groups of similar companies, similar businesses that have the same propensity to raise prices, the same likelihood to raise prices. One group raised prices while the other didn't.
- 11:39
And then we look after some time at the results. So the group that raised prices actually gained forty-two hundred a day profit, and the group that didn't raise prices actually gained twenty-eight hundred dollars a day.
- 11:53
So the question is, what is the impact of raising prices? So a naive answer would be the difference, right? Fourteen hundred is the impact. Of raising prices. But actually, you need to account for the fact that the companies that raised prices are actually naturally more successful businesses, which is also why they could raise the prices.
- 12:14
So the real difference is more like eleven hundred and fifty dollars for this illustration. And we measure the impact of actions on the outcome through a measure called CATE, conditional average treatment error, which looks at that connection.
- 12:33
And here's what I want all the AI and finance leaders here in the room to pay attention to.
- 12:38
So find where you can see a lot of different situations across entities that you have in your data, in our case, it's businesses, what they did, and verify the outcomes and how things turned out if you can see that in the data 'cause that's the one thing that frontier off-the-shelf models do not have.
- 13:01
And don't get me wrong, frontier models are amazing, and we actually use them, and I'll, I'll show you how we use them. So we use them to generate hypothesis, candidates for actions we would suggest a business to do.
- 13:17
But then we would use a model that we trained using reinforcement learning in order to figure out which one of those is actually a right move to do versus a mistake that could drive the business down.
- 13:31
Now, how we do it, a little bit into our approach, is we look at what we call, we actually create from the data, millions of business trajectories. So we have data at Intuit across our products, QuickBooks, TurboTax, Credit Karma, Mailchimp.
- 13:48
So think about a business and the financial data. There's the general ledger, the P&L, the cash flow statements, the invoices, like I mentioned before. So we take all of that data, and we create what we call business states.
- 13:59
A state of a business is, think about a very detailed summary at a given point of time. And then we derive actions. So for example, in your general ledger, I can look and see that you have invested in a campaign, in a marketing campaign, or you paid someone.
- 14:17
So I know you're paying payroll, I know how much, um, um, your hiring costs are, and so on. So we derive all of these actions,
- 14:26
and we look at what are the outcomes in different time frames. And outcomes can be increase in profit and revenue and cash flow, in time, combination of those. So we create millions of vectors of state, action, and outcome, and then we train an RL model to be able to understand, in given situations of similar businesses, which actions
- 14:49
lead to the best outcomes. And then the third step is we actually train an LLM to be able to generate that better advice.
- 15:02
That's where opinions are going in and evidence is going out. So the model you saw in the examples at the beginning were actually a model that we developed with researchers at Intuit across millions of small and medium businesses, and we actually tested it head-to-head with all of the leading models in the world.
- 15:22
And we were able, with a mid-sized, cheaper model, to outperform the frontier models because of the grounding that I just showed you. And the interesting part, as a product person, you would think that it's all about the model size and the bigger and better model.
- 15:38
Obviously, I would have a lot better chance. But it doesn't turn out to be true. And the moat here is that it's not about the model access. It's about the data itself that you have.
- 15:53
And then we went ahead and built an experience out of it, and this is an AI business advisor that is currently in beta with research, in a research preview with our customers, where we use the LLM that I just described that we created to proactively raise opportunities for businesses at every given point of time.
- 16:12
"Here's what you should do. Here is why," grounded in who is like you, who's similar to you, what they did, and why we're actually recommending you to do it, and you can drill down into it, understand, and create action plans that will lead to your business actually growing in the right direction.
- 16:31
Now, I wanna take a step back and zoom out because this isn't just about money. We are entering the era of outcome-driven AI. And the question stops being which model is better
- 16:45
and starts becoming how can we steer AI to actually make it achieve the outcomes we want in our domains? And it doesn't matter if you're building an anti-fraud system or a healthcare system, logistics, developer tools.
- 17:02
The winners, in my opinion, are gonna be those with the best system of records, creating unique data sets out of them, and then training the models to achieve the outcomes.
- 17:17
And as the product person here, it's not just about the science. The science is very important. But a great advisor, think about the great advisors and mentors that you had in your life.
- 17:27
They understand you, right? They understand your preferences, what you like, what you don't like. So a great advisory experience needs to have two things. It needs to have great, grounded science, the best science, but also it needs to understand you, what you prefer, and it needs to even make you feel as if you were part of the
- 17:49
decision to create a trusted experience. So three things I want you to remember today. Every model has read about your domain,
- 18:02
but none has actually watched the plays and the moves and their outcomes, and that gap is the whole game. Now, in coding agents and coding models, we're seeing it very advanced, a lot of verified outcomes and creating models that actually lead to better outcomes in coding, but it's very much unexplored in the financial domain and in other
- 18:26
domains as well. And you don't close the gap with bigger models. You close the gap with experience, embedding experience into the model by looking at verified outcomes in your data, what actually worked at scale.
- 18:44
So off-the-shelf models don't understand money, but grounded in real outcomes, it does, and that's what we were able to figure out.
- 18:56
So here's the one thing I want you to do tomorrow.
- 19:00
Well, actually, you know what? Go ahead and enjoy Fourth of July weekend. But right after that,
- 19:05
look in your data where you can see situations across entities and outcomes you can verify.
- 19:15
Get deep into that data and start grounding your AI in that. Think about those angles. And that's for you to build.
- 19:24
Thank you very much. Thank you for listening to me. [audience applauding] Happy to connect, LinkedIn, Twitter, in the hallway. Thank you very much. [upbeat music]