How VS Code Went from Monthly to Weekly Releases with AI — Harald Kirschner

VS Code’s small team was shipping an installed application to more than 50 million users. Agents increased the volume of changes it could produce, while those changes still needed review, validation and a reliable route onto users’ machines. Harald Kirschner, a product manager on VS Code at Microsoft, describes how the team rebuilt that surrounding process to move from monthly to weekly releases.

One early signal was code survival: the percentage of agent-written code that actually gets committed. Generated work that a developer distrusts, rewrites or throws away does not survive into that committed result. The reported share rose from 55% to 86% as models and the agent harness—the software surrounding the model—improved. That is a useful measure of developer acceptance. Because models and the harness changed together, the increase does not isolate either contribution, and committing code is not itself a measure of production correctness.

Success created more work downstream. AI helped people file better issues, while also increasing automated, low-quality reports. More pull requests arrived from both the team and the community. The community contribution story was encouraging: the number of community PRs that actually got merged was increasing too. The system had to absorb more useful participation alongside more noise.

Weekly releases replaced a cadence that had lasted ten years, since VS Code 1.0. The old monthly rhythm even shaped the release names: the January release shipped in February because the name referred to the development iteration. Changing that familiar calendar was part of matching delivery to the team’s new pace.

The work unfolded across three constraints. Faster code generation puts pressure on review and verification. Reliable delivery then matters especially for an installed application, where recovering from a bad binary is costly. Once the team can ship quickly with confidence, product judgment becomes the next question: whether the proposed change is worth shipping at all. Improving the whole system means giving each of those decisions better feedback.

An agent entering a repository needs orientation and local knowledge. Investments in developer documentation and onboarding therefore acquire another reader: the agent itself. VS Code makes two kinds of guidance reusable.

• Repository maps: A lightweight AGENTS.md tells the agent where to look and how the codebase is organized. An initial generated draft is a starting point. The document should evolve as agents work in the repository and make mistakes, so the next attempt benefits from what the team has learned.

• Expert procedures: VS Code packaged its accessibility practices into a skill. Previously, one specialist repeatedly had to be pulled into changes or supply feedback. The skill makes that expertise available to everyone, while the accessibility area owner still reviews and maintains it. The specialist’s judgment gains a reusable form and a clear maintainer.

Kirschner’s litmus test is practical: can he, as a PM, contribute effectively to the VS Code repository using an agent? Success includes the engineering team’s cost of accepting the contribution, whether it works without extensive repair, and whether it continues to work when shipped. A change that runs only on the contributor’s machine has not yet passed that test. Repository guidance earns its value when it helps more people produce changes the team can maintain.

A slow build becomes more expensive when ten or twenty agents repeatedly need its result. A human developer may use the wait to review another PR or switch tasks. An agent correcting a patch still needs the build or CI result before it can make an informed next attempt. When many agents reach the same slow stage, the delay compounds across their work.

Moving to TypeScript Go produced a reported 10× improvement in VS Code’s builds. The payoff sits inside the repeated edit-and-check process: an automated PR gets feedback sooner, and an agent can respond to a failed check sooner. Build speed becomes part of the agent’s ability to correct its work, with the benefit repeated across many active changes.

Broader participation also made it easier to put a working idea in front of the team. At the beginning of the year, Kirschner opened a large, scrappy PR for a tool that asks users questions. The useful thing to inspect was the proposed interaction inside VS Code: teammates could try the experience instead of having to imagine it from a description.

Within a few weeks, that rough proposal had become a polished, designed experience. The initial implementation made the idea clear enough for others to participate. It supplied a concrete starting point for discussion, design and refinement; the rest of the team’s contribution turned that starting point into a better feature.

An agent can confidently describe a UI as finished while the rendered result is visibly misaligned. That failure makes access to the application itself valuable. Code and a success message leave a gap between what the agent intended and what a user will encounter.

The first investment was an automated component browser that runs when VS Code changes. In the example, a PR adds a back button to the customization screen. The review question is whether that requested addition also changes anything else. The process captures screenshots of the components and identifies differences, giving reviewers a way to inspect the visible effects of the change.

Those comparisons can expose ripple effects, such as another component moving or an unrelated icon disappearing. They also reduce the effort required to understand a PR. Developers had previously been asked to attach a screenshot or video so reviewers could assess the work without running everything from scratch. The component browser automates that visual record.

The second investment lets the agent exercise a particular scenario. VS Code’s HTML interface runs in Electron, so Playwright can drive it. A /launch skill opens VS Code and clicks through a specified flow, collecting logs along the way. That gives the agent observations for diagnosing a problem and checking behavior before and after a fix. The developer can start the scenario, work on something else and return to the result without manually repeating every click.

The same approach depends on suitable automation for the target platform. For his own iOS work, Kirschner describes MCP tooling used from VS Code to create an app, open it, click through it and take screenshots. He also points to Android tooling. The useful capability is direct access to the running product, so the agent can compare its implementation with observed behavior.

Automatic GitHub Copilot code review initially failed to convince the team. As it improved, the team made it mandatory for every new PR. The ordering matters: humans wait until the AI review’s comments have been addressed and resolved before beginning their own review. This moves an initial round of feedback ahead of scarce reviewer attention while retaining human review.

Review effort also creates a cost decision. Kirschner points toward forthcoming controls for low, medium and high effort, with the choice depending on the risk in a repository. More review work consumes more resources; a useful control would let the team match that expenditure to the consequences of missing a problem.

Issues provide a different kind of signal. AI can help people from different language backgrounds write clearer reports, but the growing volume still needs attention. Work that engineers previously handled manually now includes several automated jobs:

• Filtering: Remove spam from the incoming reports.

• Enrichment and translation: Make reports more usable before an engineer investigates them.

• Assignment: Send issues to the people responsible for the relevant areas.

The agent will make triage mistakes, so the team also built a way to correct them. A Chrome extension lets people fix duplicate-issue mistakes, and those corrections can feed back into the agent’s triage work. Automation gets the volume under control; the correction mechanism preserves a route for human judgment to improve subsequent handling.

Telemetry supplies another stream of problems. VS Code already had its own data, tooling and machine-learning-based sorting for error reports. The described system starts with roughly 51 billion raw telemetry events per day, filters down to errors with complete stacks, then groups and buckets them through fingerprinting. Grouping repeated reports turns a large number of events into recognizable problems that can be investigated together.

In the example described, that reduction produces ten issues assigned to area owners, with PRs automatically created to attempt repairs. An error overview also exposes the stacks, their hit counts and the number of affected users. Those two counts describe different aspects of impact: how often an error occurs and how widely it reaches.

The concrete repair example connects an error report to a particular interface mismatch. The issue includes an initial diagnosis; the agent investigates the relevant code change, links it to symbol finding and identifies onCancellationRequested as missing from the remote procedure call (RPC) protocol. It then opens a PR proposing a fix. The question for review becomes whether that patch addresses the cause identified in the issue. A reviewer has both a diagnosis and a proposed change to inspect.

Following the handoffs makes human responsibility visible. Filing an issue assigns it to an area owner and also gives a repair agent work to pursue. A mix of deterministic systems and agents handles triage, issue creation and the proposed patch; human oversight still covers approval of the PR before the fix lands. Assignment and repair are separate branches of the same issue, while approval remains a later decision about the resulting change.

The old release process finished testing and then opened the floodgates: the release went to 100% of users. The team replaced that approach with staged rollouts as agent-assisted development increased the pace of change. Kirschner readily concedes that this was a practice they probably should have adopted earlier.

During a staged rollout, error logs and incoming issues supply feedback as the release reaches users. The team gains an opportunity to observe problems before exposure expands further. That matters for an installed application: recovering a bad release on users’ machines is expensive. Faster shipping therefore needs both checks before release and observation during distribution.

Shipping a reliable change still leaves the question of whether the product is improving. VSC-Bench gives VS Code’s agentic product an evaluation suite that can grow with the problems developers encounter. A GitHub issue requests a new scenario, and an agent uses a template to bootstrap it. Scenarios from issue reports and customer conversations can become repeatable evaluations with less setup work. The team can then improve behavior against those cases, check the results offline and follow with online experiments.

A tiny file-writing task shows why understanding the evaluation harness matters. The companion study, What 50,000 Runs of a 5-Line Eval Taught Us, specifies the say_hello smoke test: the same empty workspace, the same tools and a fixed request. Across six months, it accumulated 50,974 runs over 30 models. The task and its two checks are small enough to inspect together:

promptSteps:
  - text: Add HELLO to HELLO.txt.
assertions:
  - check: file_exists("HELLO.txt")
  - check: file_contains("HELLO.txt", "HELLO")

Passing those checks leaves the route to the result largely invisible. The study separates two kinds of behavior:

• Direct file creation: The harness already supplies the empty workspace state. A single create_file call can create HELLO.txt with the requested content.

• Additional work: Other runs plan first, list or search the empty workspace, choose a more elaborate editing tool, or emit extensive narration. Those actions can still end with the right file, while consuming more time and tokens on this one-step request.

The token comparison asks how much effort each model spends to produce the same five-character result. Kirschner highlights a roughly 70× difference. The detailed study uses about 50 output tokens, including tool-call structure, as a realistic floor for a minimal direct run; its highest-overhead model averages 3,676 output tokens. Model size was a poor guide in this study: the smallest model had the highest output-token overhead. A correct file concealed a large difference in the work required to produce it.

That result concerns a short task with an unambiguous answer. Planning and exploration can prevent expensive mistakes during longer work, so the companion study cautions against tuning the harness around this test alone. Its practical lesson is to record the tool sequence as well as success and token counts. The sequence explains where extra work came from, while a broader evaluation suite checks whether a change improves more demanding scenarios too.

The earlier ask questions PR showed how a working experience could start a design conversation. Much of Kirschner’s prototyping now serves that purpose without being intended for a production PR. A prototype makes an idea concrete enough for teammates to discuss how the experience should behave and what they would change.

Those conversations happen daily. The team looks at an idea, considers how it could work, and returns the next day with an updated prototype. Daily product exploration runs alongside weekly releases, giving the team more chances to refine the experience between shipments. Smaller squads and work streams, with continuing ownership of narrower areas, help carry those iterations forward.

The next investment depends on what is now blocking progress. A successful improvement increases the work reaching the next constraint, so the team has to keep looking for it. Agent design includes the way feedback reaches the agent: how the result is checked, how mistakes are corrected and how quickly another attempt becomes possible. Improving that path is what lets the next increase in code generation become useful work.