Stop Ordering AI Takeout: A Cookbook for Winning When You Build In-House

An AI strategy can resemble ordering takeout: choose something impressive online, pay a premium, and discover that the result falls short of the photographs. The engineering version starts with multi-agent graphs and the latest models because those are the recipes circulating on social media. An architecture designed to serve millions of customers, however, can be an expensive mismatch for one internal workflow.

Jan Siml’s team faced the build-or-buy decision in Q1 2024 and chose to build. Siml reports that two developers, with roughly ten-plus sprint weeks of effort, produced a system that resulted in several million dollars of annual recurring revenue and a group-level award. He withholds the exact financial figures. That is the business result behind the case study, rather than an independently established attribution of revenue to AI.

For this project, large evaluation efforts, multi-agent systems and reinforcement fine-tuned models threatened to increase cost and delay launch. The opening menu slide makes the temptation concrete: agentic RAG, synthetic-data evaluation, multi-agent systems and in-house models all look attractive as individual courses. The question is whether the internal job needs that entire menu.

SaaS resembles a hotel buffet: broadly useful, predictable and designed for many customers. An internal tool can instead fit the particular way a company works. Neither option wins in every situation; the advantage depends on which knowledge and capabilities are already available.

The internal advantage is more than possession of a dataset. Colleagues know the keystrokes and decisions that turn information into a closed deal, and they can help double-check the system’s outputs.

Proximity also shortens the development loop: a colleague can describe a problem, a tweak can ship the same day, and the interface can use familiar business language. Compute running on infrastructure the company already pays for can reduce incremental costs, although the talk gives no unit-cost breakdown. Siml’s rule is conditional: buy to explore an unfamiliar workflow; build once you understand and own it.

The foundation of the five lessons is depth. An internal tool does not need to expand its total addressable market; it can concentrate on one painful job. Choose a job with an identifiable value event: the dollar-based outcome that makes the work worthwhile. This gives the project a destination before its feature set starts growing.

The team began with sales alerts, then investigated what those alerts were supposed to accomplish. What happened after someone received an alert? What additional work was necessary? Asking users those questions let the team deepen the same use case rather than spread into unrelated features. A sufficiently focused workflow also kept the implementation simple enough to avoid agentic behavior.

F1 and nDCG can help assess system quality, but they do not establish whether a sales workflow produces revenue. Siml compares evaluations to smoke alarms: they remain useful, but their readings are not the financial outcome. Measure the path from the AI task to the value event, not just the quality of an intermediate output.

That requires instrumentation across the revenue funnel, from its beginning through the final dollar-based outcome. Siml’s example of connecting an AI task to twenty dollars is an illustration of the desired attribution, not a reported transaction. In this internal workflow, users also double-check outputs, providing guardrails for experiments without requiring the team to perfect a large offline evaluation program first.

Once work is connected to revenue, prioritization becomes a discussion about an idea’s expected sales impact. That visibility creates another demand: managers want team performance reports. Automating those reports and adding leaderboards can serve several purposes:

• Competition: Encourage teams to improve their results.
• Leadership involvement: Make the workflow’s contribution visible to managers.
• Champions: Identify people who use the system effectively.
• Support: Find colleagues quietly struggling to adopt the new workflow.

The reports therefore help manage adoption as well as demonstrate results.

Knowing the business workflow changes the interface requirement. If the system already knows what users are likely to need next, requiring a fresh question adds avoidable work. The team moved toward anticipating those needs and sending a daily digest containing what users needed to know that day.

Chat remained available, but became the fallback rather than the required starting point. This division reduces the effort of obtaining routine information while preserving flexibility for exceptions. The freed time still has to become useful action.

Delivering information is only part of the job. The system must guide action, because saving thirty minutes accomplishes little if those minutes disappear into low-value email. As the revenue funnel becomes better understood, it reveals which activities deserve the user’s attention. That knowledge can shape what the system recommends next, converting time saved into time spent on higher-value work.

The proactive system surfaced things users would not have thought to do. Siml reports that it had twenty points higher NPS and an order of magnitude higher engagement than the chat app. The talk does not supply the measurement period, sample size or engagement definition, so this remains a comparison from the team’s deployment rather than a general benchmark for proactive interfaces.

With revenue coming through the system, the next constraint is where to invest limited development effort. Siml favors better data over more capable models. In a sales-alert workflow, that means improving what the system can notice and how closely it fits the work users need to complete.

Siml describes o3 as sixty times more expensive and an order of magnitude slower than GPT-4.1 mini. The talk does not specify the workload, pricing basis, token usage or latency methodology. This should not be read as a current API price comparison: published token rates changed with the June 2025 o3 price reduction, and billed reasoning tokens mean a task’s total cost need not match its per-token price ratio. Those distinctions do not establish the cost of this particular deployment.

The team’s best results came from adding more alert triggers and going deeper into user needs. When they switched between standard and mini model series, Siml says the changes appeared in costs and evaluation results, rather than the business outcome. For this workflow, expanding useful coverage was therefore a more productive investment than changing models. The allocation principle is to build what users need, rather than what developers are curious to try.

Close user contact then becomes a repeating development loop. When users see their feedback reflected in the product, they feel heard and contribute more ideas. Those suggestions inform weekly experiments; useful experiments increase adoption; increased adoption supplies more data for prioritization and more suggestions for the next change. Siml describes this as a revenue flywheel: the system becomes more useful through repeated contact with the people doing the work.

The closing recipe connects these choices rather than treating them as independent optimizations. Start with one painful job whose dollar value is clear. Instrument its final outcome so that development decisions follow revenue, then push the predictable insights directly to users. Guide the time that delivery saves toward the highest-value activities. Keep investing in the underlying data and user experience as users reveal what they need next. A small starting scope supplies the focus; following the money and listening to users determine how it grows.