What the Top Companies Do Differently With AI Coding - Martin Harrysson & Prakhar Dixit, McKinsey

AI coding was already widespread among McKinsey’s large clients when a survey late in the previous year turned up a disappointing result. Most reported only modest performance improvements. Martin Harrysson, Prakhar Dixit and their colleague Shilpa worked with companies changing how their engineers developed software. Many had put new tools into an existing process in the hope that faster coding would speed up delivery.

A few months before the talk, McKinsey surveyed a few hundred large enterprises. A leading group, roughly 20–25% of respondents, reported improvements of about 2× across the product development life cycle. That was the result that caught Harrysson’s attention. A developer might finish coding 10–20% faster while the team still spends much of its time getting a change ready to build or ready to ship. The 2× claim concerned the process end to end. Harrysson left the common unit of measurement unspecified, so translating that figure into days saved per release would require more information.

Four patterns in the client work offered a way to understand that gap. Teams changed when people reviewed work, combined responsibilities in smaller groups and made company knowledge available to agents. The fourth change concerned everyone outside the pilot team. They, too, needed to know how to work this way.

Some clients moved from two-week sprints to workflows lasting one or two days. In Dixit’s example, agents work the night shift. People validate the output the next day and send the work through again. Setting up the overnight run means deciding what an agent can carry forward on its own and where it must stop for a person to judge the result.

That stopping point can move. Once reviewers have confidence in a particular step, they can let the agent carry its output into the next stage before asking for review. Autonomy grows one handoff at a time. Six months to a year earlier, much of this work had concentrated on building and testing. Clients were beginning to use the same approach for product definition and release.

An overnight run also needs a dependable starting point. Teams in a large company may store the same kind of information in several places and describe it differently. Dixit suggested choosing the documents a workflow actually consumes, such as a product requirements document (PRD) or a specification, and giving each a consistent template. Then whichever team defines the work, the agent starts with a recognizable description of what to build.

Dixit saw clients form pods of two to four people. An organization could keep its engineers and form more teams that way. The change was especially visible at banks and other enterprises where work had previously passed through several layers of specialists. Responsibilities began to combine in two recognizable ways, even while job titles were still unsettled.

• Definer. Product roles converged around specifying what the team should build. A PRD gives that work an output the pod and its agents can use.
• Builder. Development and quality assurance increasingly belonged to the same role. The person responsible for producing a change also had to judge whether the result was good enough.

A pod can choose a capable coding tool and still need the colleague who has been at the company for 20 years. If that person alone knows how a system works, each agent run that needs the knowledge has to wait for their time. This is the practical purpose of what Dixit called a semantic layer. Once a company records its knowledge in a form agents can use, successive runs can draw on it. Trust poses a separate engineering question. How will the team govern what agents build and check that the result is secure and stable?

A pilot team building a product from scratch has favorable conditions. Its members can work out among themselves who defines the work, which tools to use and when to intervene. A striking delivery time on that project tells the next team little about how to repeat it. Before expanding across engineering, Dixit wanted companies to settle those operating decisions and account for the cost of running the agents across many teams.

Shilpa gave an example of a choice that could otherwise fall to every pod. A client wanted an agent skill for writing a simple PRD. A search turned up roughly 1,700 downloadable skills for that one task, and the client was stuck on which to choose. One way to narrow the field would be to ask which skills produce a PRD the team’s workflow can use. Dixit’s template gives that question a concrete reference. Curating a handful of skills that fit local practice would spare each team the same search and decision.

Shilpa described reports of falling quality as code generation accelerated. The denominator matters here. A team opening more pull requests (PRs) can have more defects in total even if the likelihood of a defect in any one PR stays constant. Shilpa said the reports and data she had seen still showed worsening quality per developer and per pull request. For the people doing the work, the immediate concern was review skill. Developers and testers learning to generate code faster needed equal practice reading the result and deciding whether to accept it.

Reviewers can put that judgment to work before the agent writes code. A representative sample of a repository shows what engineers have typically written, including patterns they would prefer to retire. The team can instead curate golden examples that meet its standard and supply them alongside its policies. The examples show how an acceptable implementation looks in this codebase. The policies tell the agent which properties to preserve when the next task differs from the example.

A final quality gate can reject finished work. Giving the agent the relevant standard at each checkpoint lets that standard influence the work while there is still time to change it. This starts with the PRD. At the code checkpoint, the questions become specific. Can another engineer maintain the change? Has it introduced unnecessary complexity? Will it merge cleanly with work on the other branches?

Where does a change spend its time once building and testing get faster? Shilpa highlighted those two stages in the middle of the development cycle. As they shrink, a team can run out of well-defined work at the front of the cycle or finish changes more frequently than it can release them at the other end. Either wait limits how quickly customers see an improvement.

• Upstream, requirements need evidence. Customer logs and emails give an agent information about the problems people experience. The next step is to connect that evidence to a requirement precise enough to guide what gets built.
• Downstream, release has its own requirements. A finished implementation reaches customers only as often as the enterprise can ship it. Mapping release requirements during planning gives the team a chance to satisfy them as it goes.

The less obvious delay comes from the side. A bank building a customer-facing application had rules about its color scheme. One practical question was whether a color-blind customer could tell which button to click. In the bank’s usual way of working, someone would find the team that knew the rules and sit down with them. Even with agents at every stage of development, the build could still depend on that conversation. Mapping the requirements held by design, risk, compliance and governance teams makes it possible to give an agent the relevant rule while it is doing the work.

Organizations can count the tokens they consume, the tools they deploy and the people they retrain. Figuring out what that investment achieved remains harder. Harrysson saw a gap between widespread adoption and companies’ confidence in how they measured its impact. Clients continued to use familiar software delivery measures. A couple were also experimenting with AI to infer the intent of work and score its value. They hoped to use those scores to decide where to spend tokens and engineering time. Whether the scores predicted value for customers remained an open question.

Several clients used a simpler near-term calculation. Suppose an engineering leader estimates a job at three teams for eight weeks. Keep that estimate and cut each team to half its former size. If the three smaller pods deliver the same scope at comparable quality in eight weeks, the job still takes 24 team-weeks, but it uses half as many people. The other engineers can form three more pods to take on other work. The potential gain is extra capacity across the organization. The eight-week job itself still takes eight weeks, and counting the new pods tells the leader little until they deliver.

Harrysson closed with a broader observation from client work. He had seen gains of 2×, 3× or more on individual measures in a relatively short time. He attributed his optimism to seeing companies coordinate changes that depend on one another. A small pod can spend the morning reviewing an overnight build if a usable specification was ready the night before. Getting that build into a bank customer’s hands may still depend on whether the pod knew how the customer had to be able to distinguish the buttons.