What Technical Due Diligence Finds When the Deck Says “70% Automated” — Soukaïna Abelhad

“55.8% faster” sounds like a clear answer to whether AI improves developer productivity. Soukaïna Abelhad opens with the conditions behind that number: a controlled experiment involving 95 professional developers, building one JavaScript HTTP server from a template against 12 visible tests. Those details define what was measured. A result on that task does not, by itself, tell a buyer how much of an ongoing software workflow can run autonomously.

Abelhad brings experience building AI systems for investment professionals and leading production AI evaluation. Her concern is the distance between a measurement and the executive conclusion attached to it. “The dangerous number is not always fabricated.” A genuine result can travel beyond the conditions that make it meaningful. Her diligence framework follows three checks—arithmetic, definition and evaluation—before asking whether the surviving claim makes economic sense.

The first check needs no model expertise. In the Faros observational data Abelhad cites, teams with high AI adoption merged 98% more pull requests, pull request review time increased by 91%, and average pull request size increased by 154%. The sample covered more than 10,000 developers across 1,255 teams. Read those figures together: more work entered the system while the downstream review burden also grew. This was observational telemetry, not a controlled labor study, so it does not establish that AI caused higher headcount.

For a company under diligence, put workload volume beside fully loaded human labor over an 18-month period. The labor account must include the people needed to make the AI workflow function: operators, reviewers, trainers, and the work spent on escalation and rework. Counting only the people who initiate tasks would miss the people who finish or correct them.

The relationship matters more than the output headline. If human labor rises in step with workload, the system may help people produce more while still requiring proportionate human effort. That supports a claim of assistance; it does not establish labor removal. Passing this check still leaves another question: what, exactly, has the company counted as automated?

Uber provides the next distinction. Abelhad cites its statement that 70% of pull requests are AI attributed. That describes AI's involvement, but does not reveal how many pull requests were completed autonomously. She also notes that the reporting period and total count were not disclosed. Without those, the percentage lacks context about the population it describes.

The automation ladder separates several claims that a single percentage can blur:

• AI drafted the work. The system produced an initial artifact.
• Humans edited the work. People changed the artifact before it could proceed.
• Humans approved the work. Acceptance still required a person's decision.
• No human intervention. The task completed end to end within a named workflow boundary.

The last claim corresponds to the finance term straight-through processing. Each rung describes a different relationship between machine output and human responsibility.

The named boundary makes the rate inspectable. For code, does completion mean a draft exists, a pull request has been approved, or the change has reached production? A system can finish drafting without finishing delivery. Moving the endpoint changes the work that must succeed without intervention, so a buyer should request the straight-through completion rate for the actual workflow being purchased. “AI attributed” and “autonomous” answer different questions.

A precise completion definition is only useful if the measurement recognizes acceptable work. Abelhad cites a METR comparison involving four maintainers from three repositories reviewing 296 AI-generated pull requests across distinct issues. Reviewers did not know whether a patch came from AI or a human. After adjustment to the human baseline, the comparison found a 24-percentage-point gap between automated grading and merge decisions: the automated grader was substantially more optimistic than the maintainers responsible for accepting the changes.

Her own anonymous project example makes that disagreement tangible. Ten out of ten outputs passed automated validation. When domain experts reviewed them, only a few survived. No exact expert acceptance count is given, so the example establishes a mismatch rather than a reusable failure rate. The outputs had satisfied the automated check; expert review changed the assessment of whether they were good enough.

Evaluation maturity therefore becomes a credibility signal. The useful questions examine how the quality standard reaches the measurement:

• Ground truth: Who labeled the reference judgments, and who defined acceptable quality?
• Expert agreement: Do domain experts agree with the automated judge, and is that agreement measured?
• Judge validation: What was the automated evaluator validated against?
• Escaped failures: What bad output passed the checks anyway?

An automated pass rate becomes more informative when the buyer understands its relationship to expert acceptance and the failures it misses.

Builder.ai introduces the organizational problem: identifying a warning is different from making it a decision gate. Abelhad traces public questioning of its AI capabilities back to The Wall Street Journal in 2019. She then cites a May 2025 Financial Times report that its estimated 2024 revenue had been revised from $220 million to roughly $55 million, followed by a Chapter Seven filing on June 2.

This chronology does not establish that weak AI positioning caused the collapse. Revenue and financing issues extended beyond the AI claims. The diligence lesson is narrower: a warning must affect whether a claim is accepted, tested further or used to justify a commitment. A process that notices concerns without turning them into a gate can leave the original pitch doing the decision-making.

Even a credible automation claim must survive the economics. Start with the value of the workflow eligible for automation. Then account for actual adoption, straight-through processing success and accepted quality. Each condition narrows the value the company can realize: eligible work must actually use the system, complete without intervention and meet the acceptance standard. Subtract AI costs and the human costs of review, escalation and rework.

The resulting operating metric is cost per accepted autonomous task. Expressed as a ratio, its numerator includes AI costs plus human review, escalation and rework; its denominator counts accepted tasks completed autonomously inside the named boundary. Failed attempts still incur costs even when they contribute no accepted autonomous completion. An escalated or reworked task therefore cannot disappear from the cost account simply because it missed the success criterion.

That gives the next vendor or investment meeting a concrete sequence. Compare workload with fully loaded labor. Identify the automation rung and request a straight-through rate within a named workflow. Ask who defines quality, whether experts agree and what bad output escapes. Translate the claims that survive into unit economics, then decide whether to believe them, test them or walk away. Abelhad's closing question is the useful one to carry into the room: “what must be true for the number to matter.”