The State of AI in Software Development: Data from 400+ Orgs — Justin Reock, DX

Deployment frequency is rising in DX’s data, but that is only the opening question. Justin Reock, DX’s deputy CTO, presents research on AI’s relationship to developer experience and productivity, drawing on a study covering about 200,000 engineers. The useful tension is already visible: organizations are shipping more often, while engineers’ perceived rate of delivery has increased by only about 4.5% over a year.

DX gathers organizational data through a platform whose research lineage includes DORA metrics, the SPACE framework and the DevEx framework. Here, deployment frequency and pull-request throughput serve as proxies for how work moves. They help answer whether more changes reach production; they do not directly measure the value those changes create.

The deployment trend is steadily upward, with some tapering. North America continues to rise, while Europe pulls back in the most recent quarter discussed. Reock offers differences in working practices, spending and regulation as possible explanations for regional variation. Even where frequency improves, it leaves separate questions unanswered: how many changes fail, how many defects appear and how often teams revert a release.

Perceived speed deserves its own measurement. Reock contrasts DX’s modest increase with a small METR study that, in his account, involved 16 engineers: productivity fell by about 19% while perceived productivity rose by about 20%. He disputes that study’s methodology and describes a subsequent reconsideration of its signals. The comparison illustrates the question to ask rather than settling it: do engineers feel faster, and does observed work actually move faster? Those are different measurements.

Change failure rate adds the missing quality dimension. In the company-level comparison Reock describes, some organizations improve and others worsen substantially. An increase of two percentage points is large against the approximately 4% industry benchmark he cites: moving from 4% to 6% would mean a 50% relative increase in the failure rate. That arithmetic concerns failed changes; it should not be read as a direct count of additional defects.

Release pipelines and automated tests already produce differences between organizations without AI. Reock’s interpretation is that AI increases the amplitude of those differences. The observations do not isolate AI as the sole cause, so the practical response is to measure the organization’s own failure trend alongside its release and testing conditions, rather than assume a universal quality effect.

Two developer-reported measures expose another split. Perceived code maintainability rises by almost 4%, while change confidence falls by 6%. These usually move together: modular, understandable code tends to make developers more comfortable changing it. Here, assistants can make code easier to understand and modify while leaving developers less certain that the resulting change is safe to ship.

Average pull-request size grows from around 44 lines to 72 over roughly a year. Reock connects that growth to falling change confidence and makes PR size a metric to watch closely. His complaint carries the perspective of someone who has written code professionally since the late 1990s: implementing a use case with fewer lines used to be a mark of skill. Generating more code cheaply changes the incentive, but the review burden remains.

The clearest mechanism is his hypothetical build pipeline. An assistant generates four functions almost instantly, but a build takes 45 minutes or an hour. The developer could submit four separate PRs and face the build delay for each, or bundle the functions into one PR. Fast generation beside slow validation encourages the second choice. The observable change is a larger unit of work arriving for review, even though the original speedup happened during coding.

What changes when generation becomes fast but validation stays slow? The diagram follows the same four-function example. It shows how the unchanged build delay can encourage batching; it does not imply that every pipeline runs builds serially or that every developer makes this choice.

The cost travels downstream. More lines mean more material to review and more opportunities for bugs or vulnerabilities. Larger changes also make it harder to understand what changed and to roll it back. DX’s reported sentiment about being able to work in small, incremental changes falls by 10%. The pipeline example explains why improving generation alone can work against the small-batch delivery practice that makes releases manageable.

Junior engineers use AI the most in DX’s findings. Reock’s explanation is that newer entrants have less to unlearn and may already have used these tools during school. Usage, however, does not map directly onto the amount of time saved. Staff-plus and other senior engineers save about the same amount of time while using fewer tokens.

The comparison becomes useful at the level of a shared use case. Junior developers spend more tokens on the same kind of task, while senior engineers can more readily identify hallucinations and understand the architecture around a proposed change. Experience can therefore improve how efficiently someone directs and checks the tool. Counting interactions alone would miss that difference.

Smaller companies also lead in reported time savings. Reock attributes this to simpler release pipelines and less organizational complexity. That explanation connects back to the build example: the same coding capability has more room to affect delivery when fewer delays surround it.

Measuring developer productivity was difficult before AI. Adding token spending and tool telemetry does not resolve that problem by itself. DX’s AI Measurement Framework keeps existing developer-experience and productivity measures, then asks how AI use relates to them. Tool data identifies who uses what and where; delivery and quality measures help determine whether that use improves work.

The framework separates three questions:

• Utilization: Who uses the tools, how often and for which use cases? Daily and weekly active users describe adoption.
• Impact: Which outcomes should improve as use increases? Compare speed, quality and business value rather than treating adoption as the outcome.
• Cost: What does that use cost? Token spending belongs beside the benefits it is supposed to produce.

Organizations often begin with utilization because it is the first thing they can observe. The next step is to connect it to impact. Reock proposes separating users into cohorts and comparing PR cycle time, PR size and pushback during review. A cohort might move work faster while producing larger PRs or attracting more review corrections. Keeping those measures together makes the tradeoff visible; cohort comparisons establish associations rather than automatically proving that AI caused the difference.

AI readiness shifts attention from the model to the environment it must work in. Reock compresses the progression into assistants first, agents next, and then the realization that infrastructure was not ready. If an agent struggles to find accurate context or obtain dependable test feedback, token spending becomes a poor substitute for fixing the platform.

The readiness criteria are familiar developer-experience investments:

• Understandable context: Clear, accurate, well-structured documentation and data structures with straightforward relationships.
• Manageable code: Modular code that is practical to understand and change.
• Dependable feedback: Reliable local CI and non-flaky test suites.

These conditions help both people and agents. Reock’s wry observation is that AI may finally motivate investments teams should have made over the preceding decades. The earlier pipeline problem fits here: faster generation does not remove the need for a development environment that can validate small changes efficiently.

Agent experience adds another source of feedback. DX asks agents about problems with human steering, the context supplied and the feedback cycles they went through. Breaking those reports down by use case helps identify where collaboration consumes tokens and where it produces useful results. These are qualitative reports from the agents, so their role is to help diagnose friction alongside measured cost and outcomes.

The fifth trend widens the scope to the software and product development lifecycles. Reock estimates that even perfectly accurate, instant code generation would address only about 14–16% of the overall value stream. That is his framing of the opportunity, not a claim that current models produce perfect code. Requirements, coordination, review, release and operational work still surround generation.

The reported throughput results are much smaller than the familiar productivity multiples. In the study interval Reock describes as beginning in November 2024 and ending in February, median PR-throughput growth is 7.7%, with a 13% average. Even the top performers are in the 70% range. None reaches 2×. These are changes in a velocity proxy, not equivalent increases in business value.

Meetings, context switching, interruptions and development-environment friction can consume the time that AI saves. The four-function example makes the constraint concrete: code arrives immediately, yet validation still determines how the developer packages and delivers it. Reock invokes Eli Goldratt’s theory of constraints to make the operational point: improving a step that does not limit the flow may leave overall throughput unchanged. Find the limiting step before assuming that faster coding will fix delivery.

Morgan Stanley’s example targets the work before implementation. An agent interprets legacy code, including COBOL and Perl, and creates product requirements documents for engineers. That handoff removes a reverse-engineering step: engineers receive a description of the existing behavior instead of having to reconstruct it all themselves. Reock reports approximately 300,000 hours saved per year at Morgan Stanley.

Zapier applies an ecosystem of agents to administrative work and coordination. Agent summaries help reduce stand-ups from five times a week to two. Engineers onboard in about two weeks, compared with DX’s cited industry benchmark of usually more than a month. These interventions attack time around coding—the same time that can otherwise swallow generation savings.

Reock reports about 15% more value creation per engineer at Zapier and says the company is hiring more than at any previous point in its history. His preferred interpretation is increased capacity: if each engineer produces more value, adding engineers can improve the company’s competitive position. He calls it a “throughput story.” The examples’ savings and value figures are reported organizational outcomes; the recording does not define a shared evaluation method that would make them directly comparable.

A further review example handles about 3,000 code reviews a week, triggered by pull requests. The agent checks superficial issues and leaves its findings in PR comments. Human reviewers still perform the substantive review, but they can see what the agent already examined. Keeping the output in the existing record makes the handoff inspectable and can reduce repeated work.

Spotify’s SRE agent targets discovery during an incident. It gathers remediation steps from runbooks, combines them with incident information and context, and posts the assembled material into SRE communication channels. The immediate change is that the responder starts with relevant context instead of spending several minutes finding it. The mechanism described is preparation for human incident response, with no claim here of autonomous remediation.

These closing examples give the bottleneck argument practical shape: interpret legacy behavior before implementation, reduce coordination overhead, make review findings reusable and assemble incident context before a responder needs it. Reock closes by pointing listeners to DX’s quarterly research reports. The work to carry forward is the same measurement loop: identify where time goes, apply AI to that use case, and check whether quality, cost and delivered value improve together.