The ROI of AI: Why You Need Eval Frameworks

How many engineering organizations have a quantitative, precise framework for measuring the return on AI tools? In Beyang Liu’s opening audience poll, among engineering leaders, individual contributors and middle managers, only one attendee raises a hand. Others acknowledge that their evaluations still depend on how the tools feel. Liu, Sourcegraph’s CTO and co-founder at the time, is addressing the gap between experiencing a useful tool and demonstrating its business impact.

Sourcegraph began with a familiar developer problem: understanding large, complex codebases. Its first product was a code search engine, built to help developers find their way through unfamiliar software.

After roughly a decade building search, Sourcegraph served organizations ranging from startups to Fortune 500 companies and government agencies. Its coding assistant, Cody, brought that search capability into AI-assisted development. Liu describes its release as about two years earlier; the public launch chronology instead places its launch in spring 2023, before this 2024 talk.

The product analogy is Perplexity for code: retrieve and rank relevant information before asking a language model to use it. In Liu’s comparison, an assistant restricted to open editor files has only local context; Cody can draw relevant snippets from across a codebase. That retrieval capability is a substantial part of the product’s value, separate from the language model itself. Selling it to organizations such as 1Password, Palo Alto Networks and Leidos—the government contractor Liu connects to airport security machines—meant answering different questions about ROI for different buyers.

The developer’s experience starts with a task that looks straightforward: build a feature. Using slides he credits to Palo Alto Networks engineering director Gunjan Patel, Liu follows what happens next. Dependencies need installing. A UI component turns out to be unfamiliar. A framework needs learning. Each gap becomes a side quest that consumes both time and attention. In Liu’s telling, thirty minutes and two cups of coffee later, the developer has solved the immediate problem but has to reconstruct the original task. The accompanying slide renders the same joke as 28 minutes.

AI can bridge those gaps while the developer still remembers the larger goal. The benefit is not only faster completion of an isolated operation: it is preserving enough focus to finish the feature and deliver the intended experience instead of giving up. Staying in flow is a recognizable benefit; turning it into business-impact evidence is the harder problem.

Consider the difference between specialty coffee and supermarket Folgers. Both involve beans, but counting beans does not describe their quality. Liu turns that distinction into a miniature company: Bob grows the beans, representing engineering; Pat sells them, representing the sales leader or CRO; Alice counts them, representing the CFO or CEO.

Now Bob grows beans with AI, and Pat sells beans with AI. Alice still has to ask where the return is. Pat’s output is comparatively easy to quantify through sales. Bob faces a harder conversation: Alice wants to know how many more beans the company produces, while Bob wants to explain that the beans are getting better. Neither concern makes the other disappear. The tool costs money, so finance needs evidence; engineering needs that evidence to reflect quality as well as quantity. Liu reports seeing this tension repeatedly in conversations with engineering leaders.

Precise AI ROI measurement inherits the difficulty of measuring developer productivity. If an organization could precisely quantify the productivity added by an assistant, it would also need a defensible account of what developer productivity means. Liu calls this NP-hard and sketches a reduction-proof joke before pretending to end the talk and tell everyone to give up. This is a rhetorical analogy, not a complexity-theoretic result about ROI.

The practical response is to choose a tractable evaluation that answers the organization’s actual decision. The following six frameworks come from different customer conversations; they are alternatives and complements, not a checklist every customer must complete.

The classic labor-saving calculation assumes a fixed demand for output. Buy a tool, observe an increase in production capacity, and reduce staffing while continuing to supply the required number of widgets. It is an intuitive framework when the business objective is to produce the same output at lower labor cost.

Liu says Sourcegraph had not encountered this evaluation criterion among its software-development customers. He had seen it more often in functions treated as cost centers, particularly consumer-facing support. Engineering leaders generally did not describe their organizations as widget factories: the work involved improving software and user experiences, not producing interchangeable units.

An extensive backlog changes the decision. Liu poses a hypothetical productivity improvement of 20% and asks whether the organization would tackle more backlog work or cut staff. Those are strategic alternatives, not equivalent percentage calculations. His answer is that software companies overwhelmingly want to address the important work they already cannot reach. Under that objective, additional capacity has value even when no roles disappear.

A velocity experiment uses the organization’s existing planning process as its baseline:

1. Divide the organization into a tool-enabled group and a control group.
2. Give the first group access to the assistant and withhold it from the second.
3. Estimate feature and bug-fix timelines through normal planning.
4. Observe how each group executes against those estimates.

The comparison asks whether access to the tool changes delivery against planned work, rather than merely increasing activity inside the assistant.

Liu reports that Palo Alto Networks used a similar evaluation and concluded that approximate delivery timelines accelerated by 20–30% with Cody. The talk does not specify an assignment procedure, measurement period or statistical analysis, so this is a customer-reported timeline result rather than a documented randomized trial.

The main challenges are comparability, confounding and cost:

• Different work: Infrastructure, backend and frontend teams are not interchangeable experimental units.
• Events outside the tool: A key contributor leaving or a COVID outbreak can change delivery speed independently of AI access.
• Observation effort: Splitting access and sustaining the experiment long enough to gain confidence requires coordination and time.

Liu considers this a useful way to test an assistant’s effectiveness when the organization can afford that effort and account for the differences between teams.

A cheaper evaluation starts with voluntary use. If developers repeatedly choose a code search engine, the working assumption is that searching helps them accomplish something faster. Identify useful actions in the product’s telemetry, assign an estimated time saving to each, and aggregate the results. Liu proposes two minutes saved per code search as an illustrative conservative assumption, even though an exceptional search might prevent half a day or a week of wasted work.

The arithmetic separates observed action counts from assumed savings:

Choosing small values for the per-action estimates produces what Liu calls a lower-bound estimate. Its conservatism still depends on whether the chosen actions actually save that time. Vendors can expose the necessary analytics: Cody’s administrator reporting included explicit invocations, chats, questions about the codebase and inline code-generation actions.

An illustrative TypeScript calculation keeps those assumptions visible instead of hiding them inside a single ROI number:

type ActionEstimate = {
  action: string;
  count: number;
  minutesSavedPerAction: number;
};

function estimatedMinutesSaved(actions: ActionEstimate[]): number {
  return actions.reduce(
    (total, action) => total + action.count * action.minutesSavedPerAction,
    0,
  );
}

const example: ActionEstimate[] = [
  { action: "code-search", count: 30, minutesSavedPerAction: 2 },
];

console.log(estimatedMinutesSaved(example)); // 60 estimated minutes

Here, 30 searches is a teaching input and two minutes is Liu’s proposed assumption. In an evaluation, telemetry supplies the count; the team remains responsible for the time-saving estimate.

An implicit suggestion is not the same signal as an explicit request. Autocomplete can appear without a developer choosing to invoke it. Counting every displayed completion would reward a product for pushing more suggestions, including low-quality ones. For this interaction, Liu recommends acceptance-rate criteria rather than raw exposure counts.

Liu’s break-even example uses about $9 per month for Cody and values developer time at $100–$200 per hour. The $9 figure was the historical individual Cody Pro price, not an enterprise deployment quote. Those inputs imply approximately 2.7–5.4 minutes of saved time per month to cover the subscription:

This expresses the subscription cost in units of developer time; it does not mean those minutes automatically become cash savings.

The framework’s simplicity is also its limitation. It can describe routine time savings, but it misses a feature that gets finished only because the developer no longer abandons a frustrating detour. It also omits second-order effects of faster execution and the value of accelerating a major company initiative. Those omissions motivate evaluations closer to the organization’s desired outcomes.

Organizations already track indicators they believe correlate with engineering quality and business impact. The danger is selecting an easy-to-count proxy, especially lines of code generated. Liu describes seeing a familiar mistake return with AI: treating code volume as developer productivity. Sourcegraph even lost deals on generated-line counts, initially prompting the team to investigate whether Cody needed a product improvement.

The investigation led to a different explanation: Liu says a competing product triggered suggestions more aggressively. Optimizing that number could therefore change the frequency of suggestions without producing the organizational improvement the buyer wanted. Choose a KPI because it measures a desired change, not because the tool makes it easy to count.

Leidos made the desired change concrete by distinguishing interruptions from activities it wanted developers to spend more time doing:

Those categories defined the evaluation period. They express how Leidos wanted work to change; Liu does not give a measured improvement for each category.

An initiative-based evaluation maps the assistant to engineering objectives and key results. Consider a migration from COBOL to Java, or from React to Svelte. If that migration is a top-level engineering priority, the relevant question is what it would be worth to finish months or years sooner—or to make the project feasible at all. The unit of value is the initiative, not an individual completion or search.

An audience member immediately asks the difficult question: how do you measure how far the tool pulled delivery forward? Liu’s answer is that this becomes a judgment call for the engineering leader. Leaders may already have started the work or completed similar migrations, giving them experience of the difficult parts. They compare that experience with what the product can do and estimate the acceleration.

Liu also cites unnamed case studies in which work taking a year or longer was compressed into a couple of months. He supplies no project details or measurement method here. Such examples can inform a leader’s estimate, but the framework still depends on an explicit judgment about the project’s likely trajectory with and without the tool.

A survey can provide useful evidence even when it appears less rigorous than a delivery experiment. Run a pilot, let developers use the tool, optionally compare it with another assistant, and ask which they preferred. This captures the judgment of the people who must integrate the product into their daily work.

Liu contrasts unrestricted developer-happiness purchasing during the 2021 zero-interest-rate period with a constrained allocation process. First establish the budget for productivity investments; then use developer preferences to decide which tools fit inside it. Referring to a Pragmatic Engineer survey chart, Liu gives a typical allocation range of 5–25% for developer productivity and tools, without specifying the budget denominator in his explanation. The decision mechanism does not require adopting that range: ask developers what they value most, subject to the organization’s actual spending limit.

The frameworks address different decisions, from routine subscription value to the timing of a major migration. None eliminates the underlying measurement problem. Returning to the P-equals-NP joke, Liu urges skepticism toward anyone promising a precise measure of AI ROI or developer productivity. The practical requirement is clear success criteria shared by engineering, stakeholders and the vendor. Everyone should know what evidence would count as a successful evaluation.

Adoption itself can require intervention. Productivity tools often spread from developers upward, but Liu reports that top-down AI mandates sometimes helped overcome skepticism. In those cases, a CEO committed the organization to adopting Cody or coding AI and asked developers to learn how to use it, despite its imperfections. The rationale was skill development for a change expected to unfold over the following decade, rather than an assumption that every interaction would already work well.

The closing direction places coding tools on a spectrum from inline next-token completions to fully automated offline agents. Increasing automation is only one goal: tools also need greater reliability and broader usefulness across languages and development scenarios. Liu’s proposed path moves through that spectrum rather than jumping directly to full automation.

The next step in Sourcegraph’s talk-time vision was an online agent living in the editor, still responsive to human feedback and guidance. That intermediate form preserves a way for the developer to steer the work while the assistant becomes more capable. It is a proposed direction here, not a report that the future system had already been delivered.

The Mythical Man-Month supplies the second closing image: familiar fallacies about software productivity may survive the arrival of AI developers. Liu asks whether the fundamentals have changed. His framing favors one highly productive engineer over many mediocre developers who require extensive guidance, then applies the same question to automation: should the goal be a crowd of mediocre AI developers, or much greater capability for the skilled humans crafting the user experience? His hundred-agent-versus-100×-leverage contrast is a provocation about the objective, not a measured multiplier.

The final audience question asks whether developers will move away from coding toward metaprogramming, leaving code as boilerplate. Liu reports that, in one personal coding session the previous Friday, Cody wrote 80% of the code while he requested individual functions. Working at the function level rather than line by line helped him stay in flow.

The output, however, was still code. Liu predicts that natural language will not fully replace it, because code describes intended behavior precisely. Prompting can change how a developer expresses a request and how much of the implementation they type, while code remains the precise source of truth for what the software actually does.