# Stop Ordering AI Takeout: A Cookbook for Winning When You Build In-House

Jan Siml

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/DjUIecgpYAo-stop-ordering-ai-takeout-cookbook-winning) · [Video](https://www.youtube.com/watch?v=DjUIecgpYAo)

## Stop Ordering AI Takeout: A Cookbook for Winning When You Build In-House

An internal sales system shows how narrow scope, revenue instrumentation, proactive delivery and close user feedback can matter more than elaborate AI architecture.

### The takeout trap

[0:01](https://www.youtube.com/watch?v=DjUIecgpYAo&t=1s)

An AI strategy can resemble ordering takeout: choose something impressive online, pay a premium, and discover that the result falls short of the photographs. The engineering version starts with multi-agent graphs and the latest models because those are the recipes circulating on social media. An architecture designed to serve millions of customers, however, can be an expensive mismatch for one internal workflow.

Jan Siml’s team faced the build-or-buy decision in Q1 2024 and chose to build. Siml reports that two developers, with roughly ten-plus sprint weeks of effort, produced a system that resulted in several million dollars of annual recurring revenue and a group-level award. He withholds the exact financial figures. That is the business result behind the case study, rather than an independently established attribution of revenue to AI.

For this project, large evaluation efforts, multi-agent systems and reinforcement fine-tuned models threatened to increase cost and delay launch. The opening menu slide makes the temptation concrete: agentic RAG, synthetic-data evaluation, multi-agent systems and in-house models all look attractive as individual courses. The question is whether the internal job needs that entire menu.

![Four menu courses beside a plated meal, with a yellow warning showing “12 months & $$$$ required!”](https://ai.engineer/api/talk-essay-media/5a5e2f4a154fe8b8dedd7086f67fa3a71401c34d14fa8a8faff044b368e37178)

The Twitter AI Prix Fixe Menu lists agentic RAG, synthetic-data evaluation, multi-agent systems, and in-house models.

### Build when the workflow is yours

[1:49](https://www.youtube.com/watch?v=DjUIecgpYAo&t=109s)

SaaS resembles a hotel buffet: broadly useful, predictable and designed for many customers. An internal tool can instead fit the particular way a company works. Neither option wins in every situation; the advantage depends on which knowledge and capabilities are already available.

| Advantage | SaaS | In-house |
|---|---|---|
| Data access | Vendor integrations | Data already owned |
| Task knowledge | Cross-industry practices | Exact deal-closing steps |
| Output checking | Less direct access to colleagues | Users involved in review |

The internal advantage is more than possession of a dataset. Colleagues know the keystrokes and decisions that turn information into a closed deal, and they can help double-check the system’s outputs.

![Three comparison cards beneath a Swiss Army knife versus cutlery illustration contrast external SaaS with internal data, workflow knowledge, and colleague involvement.](https://ai.engineer/api/talk-essay-media/186f723062475ca08244fa81de9d07e11a79bd173327e8471413241ee1b0f96e)

SaaS and the home kitchen compared across data reach, task insight, and risk posture.

Proximity also shortens the development loop: a colleague can describe a problem, a tweak can ship the same day, and the interface can use familiar business language. Compute running on infrastructure the company already pays for can reduce incremental costs, although the talk gives no unit-cost breakdown. Siml’s rule is conditional: **buy to explore an unfamiliar workflow; build once you understand and own it.**

### Start with one painful job

[3:12](https://www.youtube.com/watch?v=DjUIecgpYAo&t=192s)

The foundation of the five lessons is depth. An internal tool does not need to expand its total addressable market; it can concentrate on one painful job. Choose a job with an identifiable **value event**: the dollar-based outcome that makes the work worthwhile. This gives the project a destination before its feature set starts growing.

The team began with sales alerts, then investigated what those alerts were supposed to accomplish. What happened after someone received an alert? What additional work was necessary? Asking users those questions let the team deepen the same use case rather than spread into unrelated features. A sufficiently focused workflow also kept the implementation simple enough to avoid agentic behavior.

### Instrument the path to revenue

[4:34](https://www.youtube.com/watch?v=DjUIecgpYAo&t=274s)

F1 and nDCG can help assess system quality, but they do not establish whether a sales workflow produces revenue. Siml compares evaluations to smoke alarms: they remain useful, but their readings are not the financial outcome. **Measure the path from the AI task to the value event**, not just the quality of an intermediate output.

That requires instrumentation across the revenue funnel, from its beginning through the final dollar-based outcome. Siml’s example of connecting an AI task to twenty dollars is an illustration of the desired attribution, not a reported transaction. In this internal workflow, users also double-check outputs, providing guardrails for experiments without requiring the team to perfect a large offline evaluation program first.

Once work is connected to revenue, prioritization becomes a discussion about an idea’s expected sales impact. That visibility creates another demand: managers want team performance reports. Automating those reports and adding leaderboards can serve several purposes:

- **Competition:** Encourage teams to improve their results.
- **Leadership involvement:** Make the workflow’s contribution visible to managers.
- **Champions:** Identify people who use the system effectively.
- **Support:** Find colleagues quietly struggling to adopt the new workflow.

The reports therefore help manage adoption as well as demonstrate results.

### Deliver predictable work before users ask

[6:10](https://www.youtube.com/watch?v=DjUIecgpYAo&t=370s)

Knowing the business workflow changes the interface requirement. If the system already knows what users are likely to need next, requiring a fresh question adds avoidable work. The team moved toward anticipating those needs and sending a daily digest containing what users needed to know that day.

| Interaction | Role |
|---|---|
| Daily digest | Deliver predictable information proactively |
| Chat | Handle unexpected and unplanned tasks |

Chat remained available, but became the fallback rather than the required starting point. This division reduces the effort of obtaining routine information while preserving flexibility for exceptions. The freed time still has to become useful action.

### Turn saved time into valuable action

[7:06](https://www.youtube.com/watch?v=DjUIecgpYAo&t=426s)

Delivering information is only part of the job. The system must guide action, because saving thirty minutes accomplishes little if those minutes disappear into low-value email. As the revenue funnel becomes better understood, it reveals which activities deserve the user’s attention. That knowledge can shape what the system recommends next, converting time saved into time spent on higher-value work.

![“Turn Time-Saved into Revenue” slide with two numbered principles beside a clock decorated with coins.](https://ai.engineer/api/talk-essay-media/c10fae806e82d784df3f464124e0276c2942fbafa68b0b10a87e95e413ae381d)

Guide action, not just share information; convert time-saved into time-well-spent.

The proactive system surfaced things users would not have thought to do. Siml reports that it had twenty points higher NPS and an order of magnitude higher engagement than the chat app. The talk does not supply the measurement period, sample size or engagement definition, so this remains a comparison from the team’s deployment rather than a general benchmark for proactive interfaces.

### Spend development effort on useful data and triggers

[7:59](https://www.youtube.com/watch?v=DjUIecgpYAo&t=479s)

With revenue coming through the system, the next constraint is where to invest limited development effort. Siml favors better data over more capable models. In a sales-alert workflow, that means improving what the system can notice and how closely it fits the work users need to complete.

Siml describes [**o3**](https://openai.com/index/introducing-o3-and-o4-mini/) as sixty times more expensive and an order of magnitude slower than [**GPT-4.1 mini**](https://openai.com/index/gpt-4-1/). The talk does not specify the workload, pricing basis, token usage or latency methodology. This should not be read as a current API price comparison: published token rates changed with the [June 2025 o3 price reduction](https://community.openai.com/t/o3-is-80-cheaper-and-introducing-o3-pro/1284925), and [billed reasoning tokens](https://help.openai.com/en/articles/4936856-understanding-and-counting-tokens) mean a task’s total cost need not match its per-token price ratio. Those distinctions do not establish the cost of this particular deployment.

The team’s best results came from adding more alert triggers and going deeper into user needs. When they switched between standard and mini model series, Siml says the changes appeared in costs and evaluation results, rather than the business outcome. For this workflow, expanding useful coverage was therefore a more productive investment than changing models. The allocation principle is to build what users need, rather than what developers are curious to try.

### Let adoption improve the next experiment

[9:03](https://www.youtube.com/watch?v=DjUIecgpYAo&t=543s)

Close user contact then becomes a repeating development loop. When users see their feedback reflected in the product, they feel heard and contribute more ideas. Those suggestions inform weekly experiments; useful experiments increase adoption; increased adoption supplies more data for prioritization and more suggestions for the next change. Siml describes this as a revenue flywheel: the system becomes more useful through repeated contact with the people doing the work.

### Apply the lessons as one operating sequence

[9:44](https://www.youtube.com/watch?v=DjUIecgpYAo&t=584s)

The closing recipe connects these choices rather than treating them as independent optimizations. Start with one painful job whose dollar value is clear. Instrument its final outcome so that development decisions follow revenue, then push the predictable insights directly to users. Guide the time that delivery saves toward the highest-value activities. Keep investing in the underlying data and user experience as users reveal what they need next. A small starting scope supplies the focus; following the money and listening to users determine how it grows.

![“The Home Kitchen Advantage — Your Recipe for Success” recap with five labeled principles and explanatory text; Data+UX > Models is emphasized.](https://ai.engineer/api/talk-essay-media/bf9326e64b9041acc303b2dbc31a23ebd26973e76b2e9819305a37238169692d)

Five lessons: nail one high-value task, chase dollars, be the chef, turn time-saved into revenue, and prioritize data and UX over models.

## Resources

- [GPT-4.1 launch announcement and pricing](https://openai.com/index/gpt-4-1/) — Original introduction to GPT-4.1, mini and nano, including launch pricing and evaluation results.
- [OpenAI o3 and o4-mini launch announcement](https://openai.com/index/introducing-o3-and-o4-mini/) — OpenAI's introduction to its reasoning models, capabilities and original API access.
- [June 2025 o3 price reduction](https://community.openai.com/t/o3-is-80-cheaper-and-introducing-o3-pro/1284925) — Dated announcement of an 80% reduction in o3 API token prices.
- [Understanding token usage and billing](https://help.openai.com/en/articles/4936856-understanding-and-counting-tokens) — Explains input, cached and reasoning tokens and how they affect total request cost.
- [Jan Siml on recommendations versus chatbots](https://www.linkedin.com/posts/jan-siml-360a3858_aiengineer-pydanticai-sanfrancisco-activity-7336615054596825090-rXhq) — The speaker's conference reflections expand on proactive recommendations and practical evaluation choices.

## Slides

### Stop Ordering AI Takeout

The speaker compares choosing an AI strategy to ordering takeout and introduces an in-house recipe that delivered millions of dollars in revenue.

[0:00](https://www.youtube.com/watch?v=DjUIecgpYAo&t=0s) [Title slide introducing a cookbook for building AI in-house, alongside a dining table.](https://assets.ai.engineer/sites/aiecode2025/sha256/856ff6aa7217be63d09817e86408f466ac6465d3d989cf92bb839aba7f227835.webp)

Stop Ordering AI Takeout

A Cookbook for Winning When You Build In-House

### Proof in the Pudding

The speaker says two developers spent roughly 10+ sprint weeks building a system that led to several million dollars in ARR and a group-level award.

[0:54](https://www.youtube.com/watch?v=DjUIecgpYAo&t=54s) [Results include multi-million annual recurring revenue, two developers, and an innovation award.](https://assets.ai.engineer/sites/aiecode2025/sha256/7eda5a1155c516cf080ca69522ce9323181efc2320748dd146a785cfe6561843.webp)

Proof in the Pudding

$$$M+
Revenue
Multi-million ARR generated

2
Developers
Small team with big impact

1
Award
RELX Innovation Honor Roll 2024

### Context Matters: External SaaS ≠ Home Kitchen App

A commercial kitchen sits beside a home kitchen as the speaker contrasts generic SaaS with the tailored needs of internal users.

[1:55](https://www.youtube.com/watch?v=DjUIecgpYAo&t=115s) [A commercial kitchen and a home kitchen illustrate the difference between external SaaS and an in-house app.](https://assets.ai.engineer/sites/aiecode2025/sha256/1e94c175d14ed03e4f9f20414b427234f9fb06dfe16763673892aac298e74a16.webp)

Context Matters: External SaaS ≠ Home Kitchen App

### Where SaaS Helps and Where Your Own Kitchen Wins (1/2)

SaaS brings cross-customer patterns and vendor integrations; an in-house app can draw on its own data and colleagues’ workflow knowledge, with colleagues checking outputs.

[2:16](https://www.youtube.com/watch?v=DjUIecgpYAo&t=136s) [External SaaS and in-house applications are compared on data reach, task insight, and risk posture.](https://assets.ai.engineer/sites/aiecode2025/sha256/a66b06a37dc8ee61b4d5971176a2d4f9030dd1d85b35936330ca2a3790e527d8.webp)

VS.

Where SaaS Helps ... and Where Your Own Kitchen* Wins (1/2)

Data Reach
External SaaS: Cross-customer patterns, vendor APIs
Home Kitchen: Good warehouse, zero export pain

Task Insight
External SaaS: Broad personas, best-practice templates
Home Kitchen: Exact workflow knowledge

Risk Posture
External SaaS: Satisfy the worst-case client
Home Kitchen: Tune risk with colleagues in the loop

[illegible]p with Users-in-the-loop

### Five Home Kitchen AI Lessons

Start with one painful end-to-end task: depth enables faster loops and cheaper operations. The other lessons connect the experience to revenue through proactive delivery, guided action, and investment in UX and integrations.

[3:19](https://www.youtube.com/watch?v=DjUIecgpYAo&t=199s) [Overview of five lessons: focus on one task, track revenue, deliver proactively, guide action, and invest in data and integrations.](https://assets.ai.engineer/sites/aiecode2025/sha256/5798fca7e214d7d7f407d9e20f4e5985003e235fce22b6d27428f19b68eecc68.webp)

Five Home Kitchen AI Lessons

1. Nail One High-Value Task, End-to-End
Focus on a single painful job-to-be-done. Depth enables faster loops and cheaper operations.

2. Chase Dollars, Not Evals
Track the revenue funnel, not just model metrics.

3. Be the Chef, Not the Waiter
Deliver value proactively via zero-click experiences.

4. Turn Time-Saved into Revenue
Guide action, not just information sharing.

5. Good Data Beats Great Models
Invest in UX and integrations, not just model upgrades.

### Nail One High-Value Task, End-to-End

Start with one painful job tied to a clear value event. The team began with sales alerts and explored what else the job required. The speaker recommends talking to users and keeping the solution simple.

[3:28](https://www.youtube.com/watch?v=DjUIecgpYAo&t=208s) [A slide recommends picking one painful job, going deep rather than wide, and talking to users; a chef chops onions on the right.](https://assets.ai.engineer/sites/aiecode2025/sha256/2b99271c020a00b82cf0f39d49400f6661352dd79ef9b53a2c8a3957fbcd7fa7.webp)

Nail One High-Value Task, End-to-End

Pick One Painful JTBD
Focus on a single job-to-be-done with clear value.

Go Deep, Not Wide
Depth enables faster feedback loops and cheaper operations.

Talk to Your Users
Better understanding replaces the need for "agentic" things.

### Chase Dollars, Not Evals

Connect each touch to revenue to prioritize by dollar impact. Run A/B tests with colleagues as guardrails; leaderboards can surface champions and colleagues struggling with new workflows.

[5:28](https://www.youtube.com/watch?v=DjUIecgpYAo&t=328s) [Five-part diagram lists tagging every touch, tracking the revenue funnel, A/B testing, gaining decision clarity, and building leaderboards.](https://assets.ai.engineer/sites/aiecode2025/sha256/7855219eca2c4456ef7f3073bdbec241cc2ea29c73e887468b91e0703e54896d.webp)

Chase Dollars, Not Evals

Tag Every Touch
Connect every data point to dollar value

Track Revenue Funnel
Offline evals ≠ revenue

A/B Test A Lot
Your colleagues provide natural guardrails

Build Leaderboards
Leaderboards gamify adoption

Gain Decision Clarity
Decisions become obvious when dollars are visible

### Be the Chef, Not the Waiter

Deliver insights proactively through Slack or email. The speaker’s team sent a daily digest and kept chat available for unexpected tasks.

[6:21](https://www.youtube.com/watch?v=DjUIecgpYAo&t=381s) [The slide contrasts proactive zero-click delivery with chat as a fallback for edge cases.](https://assets.ai.engineer/sites/aiecode2025/sha256/9ee5275d0ae5a8eb8e6a91344e255b54da5afc55f52caa00d10746813a554530.webp)

Be the Chef, Not the Waiter

Zero-Click Delivery
Push insights via Slack or email. Don't wait for users to ask.

Chat as Fallback
Keep chat UI as a secondary option for edge cases.

### Good Data Beats Great Models

Adding user alerts and addressing user needs produced the best results; switching models changed costs and evals. In one May 2025 task, o3 cost ~$0.18 versus ~$0.003 for GPT 4.1 mini.

[8:32](https://www.youtube.com/watch?v=DjUIecgpYAo&t=512s) [A slide contrasts model and technology costs with UX and data integrations, linking adoption to revenue and giving an example model-cost comparison.](https://assets.ai.engineer/sites/aiecode2025/sha256/2b2b88135899e6a6be1edc4975e85aeb39567eadd390847bcd91f58ac3eff54f.webp)

Good Data Beats Great Models

Latest models (o3, GPT-4.5) and trendy tech (Multi-agent graph)
↑ cost* ≠ ↑ revenue

UX and data integrations
↑ adoption = ↑ revenue

*Example task (May 2025): GPT 4.1 mini cost of ~$0.003 vs o3 cost of ~$0.18 →60x more expensive

### The Internal Flywheel

Feedback makes users feel heard and generates ideas for weekly experiments. Successful updates can grow adoption, producing more data for prioritization and more ideas.

[9:16](https://www.youtube.com/watch?v=DjUIecgpYAo&t=556s) [A flywheel slide highlights tight feedback loops and constant user input; later steps remain blurred.](https://assets.ai.engineer/sites/aiecode2025/sha256/f7a7e4d34d2d35ff367030e3d3a85b81a5ebd5680d6459451c2ad292f65d7305.webp)

The Internal Flywheel

1. Tight Feedback Loops
Users feel heard when their feedback is quickly incorporated.

2. Constant User Input
Active engagement drives a steady stream of improvement ideas.

3. Rapid Experimentation
[illegible]

4. Growing Adoption
[illegible]

5. Data-Informed Prioritization
[illegible]

[9:33](https://www.youtube.com/watch?v=DjUIecgpYAo&t=573s) [A five-step flywheel connects feedback loops, user input, experimentation, adoption, and data-informed prioritization.](https://assets.ai.engineer/sites/aiecode2025/sha256/c35ff79cfc64ad719c6c701bb068366b06b5e4e09dd90f5956ce3da853f0a467.webp)

The Internal Flywheel

1. Tight Feedback Loops
Users feel heard when their feedback is quickly incorporated.

2. Constant User Input
Active engagement drives a steady stream of improvement ideas.

3. Rapid Experimentation
Weekly experiments test and validate new ideas efficiently.

4. Growing Adoption
Successful updates increase user base and engagement.

5. Data-Informed Prioritization
More usage generates better data for decision making.

### The Home Kitchen Advantage — Your Recipe for Success

Start with one valuable task and measure its revenue impact. Deliver insights proactively, channel saved time into the highest-value revenue-generating activities, and invest in data and UX.

[10:17](https://www.youtube.com/watch?v=DjUIecgpYAo&t=617s) [Five recommendations: focus on one valuable task, chase dollars, proactively deliver insights, turn saved time into revenue, and prioritize data and user experience over models.](https://assets.ai.engineer/sites/aiecode2025/sha256/89fb1aede6050d3b006af47960f05bdd32e1a30337208898f2a954d54f451910.webp)

The Home Kitchen Advantage — Your Recipe for Success

Nail One High-Value Task
Focus on a single painful job-to-be-done with clear dollar value.

Chase Dollars
Track everything to final revenue outcomes, not just model metrics.

Be the Chef
Push insights proactively rather than waiting for questions.

Turn Time-Saved into Revenue
Channel freed-up time into highest-value revenue-generating activities.

Data+UX > Models
Invest in solid fundamentals over chasing trendy, fancy technology.

## Full transcript

[0:01](https://www.youtube.com/watch?v=DjUIecgpYAo&t=1s) Many teams run their AI strategy the same way they order takeout. Pick something that looks good online and pay the premium, only to discover later that it's half warm and nothing like the photos.

[0:15](https://www.youtube.com/watch?v=DjUIecgpYAo&t=15s) Why? The expectations are too high. Scroll LinkedIn or Twitter, and you will see the recommended prix fixe menu with multi-agent graphs and bleeding-edge models.

[0:27](https://www.youtube.com/watch?v=DjUIecgpYAo&t=27s) That might make sense when you're cooking for millions of customers. But inside your company, that's like paying for truffles to garnish your instant noodles.

[0:40](https://www.youtube.com/watch?v=DjUIecgpYAo&t=40s) We followed a different recipe, and it delivered millions of dollars of revenue. Today, I want to share it with you.

[0:50](https://www.youtube.com/watch?v=DjUIecgpYAo&t=50s) So first the proof, and then the recipe that got us there.

[0:55](https://www.youtube.com/watch?v=DjUIecgpYAo&t=55s) In Q1 2024, we faced the classic dilemma: build or buy? We chose to build. Two devs and roughly ten plus sprint weeks of effort later, we had a system that resulted in several million dollars of ARR and a group-level award.

[1:18](https://www.youtube.com/watch?v=DjUIecgpYAo&t=78s) I can't share the exact numbers. Legal would saute me. But imagine a number big enough that finance stops asking, "Why build?"

[1:28](https://www.youtube.com/watch?v=DjUIecgpYAo&t=88s) Let me show you why Twitter's recipe would have killed us. Giant evals, multi-agent systems, RFT models. It looks delicious, costs a fortune, and delays the launch. It's perfect for flashy SaaS demos, but an overkill for your in-house needs.

[1:49](https://www.youtube.com/watch?v=DjUIecgpYAo&t=109s) But the question really isn't what to skip. It's when does it ever make sense? Think of it this way. SaaS is a hotel buffet. It's generic but safe. Your internal users want the grandma's secret sauce, not the hotel eggs.

[2:11](https://www.youtube.com/watch?v=DjUIecgpYAo&t=131s) So let's see where that home kitchen crushes the buffet option. SaaS shines when you need vendor integrations or cross-industry best practices. Our kitchen wins when we already own the pantry.

[2:29](https://www.youtube.com/watch?v=DjUIecgpYAo&t=149s) I mean, the data. Our colleagues know the exact keystrokes needed to close the deals, and we can involve them in double-checking the outputs. And the advantages don't stop there.

[2:43](https://www.youtube.com/watch?v=DjUIecgpYAo&t=163s) Because we sit next door to our users, so a tweak ships the same days, and the UI speaks their language. And the compute runs on infra we mostly pay for already.

[2:58](https://www.youtube.com/watch?v=DjUIecgpYAo&t=178s) It all basically drops the cost to pennies.

[3:02](https://www.youtube.com/watch?v=DjUIecgpYAo&t=182s) So the rule of thumb is buy SaaS, explore the unknown, but build in-house once the workflow is yours.

[3:12](https://www.youtube.com/watch?v=DjUIecgpYAo&t=192s) Okay, so pantry's stocked, knives are out. Here are the five lessons.

[3:20](https://www.youtube.com/watch?v=DjUIecgpYAo&t=200s) They work best as a set, so we will unpack them in order, starting from the foundation. This is where in-house crushes SaaS. You can go absurdly deep on one painful job to be done without chasing total addressable market.

[3:40](https://www.youtube.com/watch?v=DjUIecgpYAo&t=220s) Pick something where you can easily pinpoint the value event, which is that dollar-based outcome that you're doing it all for. Remember that one? It's gonna be important later.

[3:54](https://www.youtube.com/watch?v=DjUIecgpYAo&t=234s) We started with a simple sales alerts use case and grew it from there.

[4:00](https://www.youtube.com/watch?v=DjUIecgpYAo&t=240s) By going deep on that one use case, it was all made much easier. We didn't stop at the alerts. What were they for? What else needed to be done?

[4:13](https://www.youtube.com/watch?v=DjUIecgpYAo&t=253s) All those were questions we had. And who knows best what's needed?

[4:20](https://www.youtube.com/watch?v=DjUIecgpYAo&t=260s) Your users. Talk to them to really nail it. If you stay focused, you can keep things very simple and avoid anything agentic.

[4:34](https://www.youtube.com/watch?v=DjUIecgpYAo&t=274s) So metrics coming next. Spoiler, offline evals never sign a contract. Nobody at the board meeting asks for your F1 score or nDCG.

[4:47](https://www.youtube.com/watch?v=DjUIecgpYAo&t=287s) They ask, "Did it move the revenues?" Don't get me wrong, evals are important, but they are smoke alarms. You need to track the actual money.

[5:00](https://www.youtube.com/watch?v=DjUIecgpYAo&t=300s) So instrument everything until you can say, "This AI task led to twenty dollars here."

[5:08](https://www.youtube.com/watch?v=DjUIecgpYAo&t=308s) Build your revenue funnel. Everything from beginning to end, to that value event that we talked about earlier.

[5:19](https://www.youtube.com/watch?v=DjUIecgpYAo&t=319s) By the way, your users are your guardrails, so you can run ambitious experiments. Don't overthink your evals.

[5:29](https://www.youtube.com/watch?v=DjUIecgpYAo&t=329s) And once you link your system to dollars, decisions and prioritization become a breeze. The conversation shifts to what's your idea and how much would you sell with it?

[5:43](https://www.youtube.com/watch?v=DjUIecgpYAo&t=343s) And here is where it gets interesting. Managers will start asking for team performance reports.

[5:52](https://www.youtube.com/watch?v=DjUIecgpYAo&t=352s) You should automate them, but also prepare the leaderboards because those can really create a healthy competition, get leadership invested, surface some champions, but also help those who might be silently struggling with the new workflows.

[6:10](https://www.youtube.com/watch?v=DjUIecgpYAo&t=370s) So now that you're tracking the dollars, don't wait for users to come to you. You need to become the chef who anticipates what the next dish should be. Because the best UI is the one you never need to use.

[6:27](https://www.youtube.com/watch?v=DjUIecgpYAo&t=387s) This is your business, so you know what the next steps should be. So why wait for users to ask for it? Just do it for them.

[6:38](https://www.youtube.com/watch?v=DjUIecgpYAo&t=398s) In our case, we built a motion to send daily digest. Here is what you need to know today.

[6:45](https://www.youtube.com/watch?v=DjUIecgpYAo&t=405s) We still had chat UI, but it was the fallback for all the unexpected and unplanned tasks that came along. So what's next? Now that we've stopped waiting the tables, now it's time to turn those freed up minutes into money on the register.

[7:06](https://www.youtube.com/watch?v=DjUIecgpYAo&t=426s) Your AI system needs to guide action, not just deliver information. Why?

[7:14](https://www.youtube.com/watch?v=DjUIecgpYAo&t=434s) Because saving thirty minutes is worthless if users just fill it with an email sludge.

[7:22](https://www.youtube.com/watch?v=DjUIecgpYAo&t=442s) So the real power here comes from you actually converting the time saved into time well spent. Because you know what the highest value activities are. And more you start building up those revenue funnels, the more you will start to understand where to divert that free time and your u-users' attention.

[7:44](https://www.youtube.com/watch?v=DjUIecgpYAo&t=464s) Our proactive system was a hit. It was surfacing things users wouldn't have thought of doing. And compared to the chat app, it had twenty points higher NPS and order of magnitude higher engagement.

[7:59](https://www.youtube.com/watch?v=DjUIecgpYAo&t=479s) So you're making some money now, but it brings us to a critical decision point. Where to invest your limited development resources? You might not like the answer. Good data consistently beats great models.

[8:14](https://www.youtube.com/watch?v=DjUIecgpYAo&t=494s) This is the secret that you won't find on Twitter.

[8:19](https://www.youtube.com/watch?v=DjUIecgpYAo&t=499s) We all love shiny things, but o3 is sixty times more expensive and order of magnitude slower compared to 4.1 mini. So the biggest impact if you put it in production will be on the cost.

[8:33](https://www.youtube.com/watch?v=DjUIecgpYAo&t=513s) We've seen the best results from simply adding more triggers to alert the users on and going deeper into what they needed. Boring, right? But it worked. When we changed the models from normal to the mini series and back, the only thing that changed were the costs and the evals.

[8:56](https://www.youtube.com/watch?v=DjUIecgpYAo&t=536s) So you need to build for what your users need, not what you want to try.

[9:03](https://www.youtube.com/watch?v=DjUIecgpYAo&t=543s) When you focus on what users truly value instead of simply chasing model benchmarks, something magical happens. A powerful flywheel begins to spin

[9:15](https://www.youtube.com/watch?v=DjUIecgpYAo&t=555s) because those tight feedback loops make users feel heard. So they start providing you with ideas for improvements. So you can run weekly experiments based on their feedback, which drives even more adoption, which generates more data for prioritization and more ideas.

[9:39](https://www.youtube.com/watch?v=DjUIecgpYAo&t=579s) The revenue flywheel starts spinning faster and faster.

[9:44](https://www.youtube.com/watch?v=DjUIecgpYAo&t=584s) So that's it. Let's recap what we've covered, so you can apply it immediately. Focus on one painful job to be done that has clear dollar value. Don't try to boil the ocean with a comprehensive solution.

[10:02](https://www.youtube.com/watch?v=DjUIecgpYAo&t=602s) Revenue impact trumps evaluation metrics. Track everything to the final dollar-based outcome and make decisions based on that. Push insights proactively rather than waiting for users' questions.

[10:19](https://www.youtube.com/watch?v=DjUIecgpYAo&t=619s) And simply being proactive isn't enough. You need to ensure that those time savings are channeled into the highest value activities you can find.

[10:30](https://www.youtube.com/watch?v=DjUIecgpYAo&t=630s) And invest in the basics. Um, it really pays off.

[10:36](https://www.youtube.com/watch?v=DjUIecgpYAo&t=636s) So in short, start small, follow the money, and let your users guide you.

[10:42](https://www.youtube.com/watch?v=DjUIecgpYAo&t=642s) Thank you.
