Beyond Code Coverage: Functionality Testing with Playwright

What does a surge in code tell us about developer productivity? Marlene Mhangami, introducing herself as a senior developer advocate working in Core AI at Microsoft and GitHub, opens with GitHub’s activity figures. She cites approximately one billion commits in the 2025 Octoverse reporting period as GitHub’s busiest year yet. That reporting period runs from September 2024 through August 2025, rather than following the calendar year.

For 2026, Mhangami cites a recent post by GitHub COO Kyle Daigle reporting approximately 275 million commits per week, distinguishing it from official statistics that had not yet been released. She annualizes that pace to approximately 14 billion commits, roughly 14 times the preceding report’s total. This is a projection from a weekly rate, not an observed annual total.

Some of that growth comes from agents. In Mhangami’s account, Claude and GitHub Copilot leave coauthor signatures, while Codex does not; wording can also provide clues about AI involvement. The underlying attribution data had not been released. Even if the volume is clear, the central question remains: does producing more code mean developers are accomplishing more useful work?

Mhangami turns to a previous AI Engineer presentation describing Stanford research involving 120,000 developers. Its central distinction is how developers use AI: clean codebases can amplify its gains, while unchecked generation can amplify codebase entropy. The condition of the system receiving the new code matters alongside the speed of producing it.

In the company case Mhangami recounts, PR output increased, but declining code quality and additional rework left only about a 1% increase in effective output. That figure describes the cited company case; the talk does not supply its measurement definition or observation period. More changes entering review were not translating into a comparable increase in useful output.

The practical response is to make codebase quality part of the workflow: test coverage, type coverage, documentation and modularity. Mhangami recommends standardizing these practices across teams, while acknowledging the competing preference for simply shipping quickly. Keeping the codebase understandable is part of realizing AI’s value, not a separate cleanup project after generation.

Test-driven development offers an existing structure for doing this. Mhangami points to Simon Willison’s Red/green TDD as an example of its use with coding agents. The cycle separates three jobs that are easy to blur together: deciding what should happen, making it happen and improving how it happens.

1. Red: Start with an incoming feature request and write a test that fails because the feature does not exist.
2. Green: Implement enough code to make that test pass. This phase prioritizes speed; Mhangami’s historical example is copying useful code from Stack Overflow.
3. Refactor: Revisit the passing implementation and improve its quality, structure and adherence to the project’s practices.

The green phase establishes working behavior. The refactor phase gives code quality its own explicit place in the process.

TDD’s usefulness depends on what the tests protect. Rails creator DHH’s 2014 essay TDD is dead. Long live testing. criticized test-first dogma and excessive emphasis on isolated unit tests. The relevant concern here is that a suite can accumulate code coverage without adequately testing the system. This is a critique of testing priorities, not a reason to abandon automated tests or require only system tests.

Consider an order calculation with a discount. If a test directly calls an internal method named calculate, renaming that method can break the test even when the customer still receives the correct final price. The test has become coupled to an implementation choice.

A stable boundary lets the implementation change while the test continues checking the same promise. For a deeper treatment, Mhangami recommends Ian Cooper’s TDD: Where Did It All Go Wrong?.

AI introduces another failure mode: self-affirming tests. An agent can generate tests that agree with the implementation it has produced, rather than independently checking the intended behavior. Coverage can look healthy and every unit test can pass while the system still does the wrong thing. The feature requirement must supply the expectation; the current implementation cannot be its only source.

Playwright is Microsoft’s open source framework for automating browser interactions and testing an application end to end. Mhangami names Python, TypeScript and C# among its supported languages. Instead of reaching into an internal calculation method, a browser test can navigate to a page and interact with the controls a customer uses.

The introductory example opens the toys page with page.goto, finds the search field by its placeholder and fills it with Furby. A TypeScript test for that interaction can also assert the visible result. With baseURL configured for the toy application, the pattern is:

import { test, expect } from '@playwright/test';

test('search finds Furby', async ({ page }) => {
  await page.goto('/toys');
  await page.getByPlaceholder('Search').fill('Furby');
  await expect(page.getByText('Furby', { exact: true })).toBeVisible();
});

The navigation and input drive the browser; the assertion states the expected outcome. Playwright can run with the browser visible in headed mode or without displaying it in headless mode.

With agents doing more of the typing, the proposed allocation of effort changes. The agent generates failing behavioral tests, then generates code to make them pass. Developers spend more of their attention reviewing and refactoring that code. Mhangami presents this as a way to address the complaint that TDD slows teams down, rather than as a measured speedup: accelerate red and green, then give refactoring the largest share of developer attention.

Mhangami identifies three integration choices: the Playwright MCP server, a CLI tool, and Playwright Test Agents. MCP and the CLI provide ways for a coding agent to use Playwright; the specialized agents divide testing work into explicit responsibilities.

In the demonstration’s description, initialization installs three agent instruction files, referred to as agent.md files. Their purpose is to give each role specialized guidance rather than asking one undifferentiated prompt to handle every stage.

The live scenario places the developer at Tailspin Toys. An email from the search product management team requests a search bar and filtering controls:

• Simple searches: Text search for straightforward queries.
• Complex searches: Azure AI Search.
• Product filters: A sidebar for category and price filtering.

These are customer-facing capabilities, so they provide the starting point for the test plan.

Mhangami works in GitHub Copilot CLI and first asks the agent to bring the email’s requirements into the terminal. She uses Microsoft Work IQ, described in the demonstration as a skill connecting the agent to Microsoft 365 applications such as Outlook and PowerPoint. The important handoff is from workplace context to an explicit list of features the coding agent can act on.

A feature request triggers a test; adding a class method does not have to. This changes the unit of planning. Rather than producing a new test whenever the implementation acquires another method, the agent starts from the behavior requested by the product team.

The second prompt asks Copilot to develop the features using red–green TDD, beginning with failing Playwright tests for each feature. For this example, Mhangami also instructs it not to commit changes. Copilot begins by examining the repository; the Playwright MCP server is already installed in the CLI environment. It needs to understand the existing application before writing tests that exercise it.

Generation takes time, so Mhangami switches to another tab for the browser demonstration. This transition matters: the next run uses prepared work, rather than showing the newly submitted request complete immediately.

Earlier that day, the agent had generated failing tests and then completed the green phase by implementing code to pass them. Mhangami now asks it to run the search and filter checks. Using the installed Playwright MCP integration, the agent locates the test file and starts interacting with the application.

The browser opens the correct page and enters search inputs. Furby is found, followed by Simon. Automated clicks then exercise the category filters, with Mhangami’s hands off the keyboard. Finally, the browser checks toys within a specified price range. The visible sequence makes the tested behavior concrete: search for products, select categories and narrow results by price.

Mhangami reports that all the demonstrated tests pass. That establishes the green point in this walkthrough; it is not the end of the development process. The next step is to refactor the generated implementation while retaining those behavioral checks. That refactoring is recommended, but not demonstrated in the session.

Mhangami recommends attaching test screenshots to pull requests so reviewers can see the application alongside the code changes. Screenshot capture must be configured or explicitly requested: Playwright Test does not capture every test automatically by default. The recording options let a project choose when to retain screenshots.

The visible browser is useful for the walkthrough, but routine runs can use headless mode. Before asking an agent to fix tests or change code, Mhangami recommends committing the current work so there is a recoverable checkpoint; an agent may not reliably remember the previous state. This general advice differs from the explicit no-commit instruction used for the demonstration. She also recommends one test per feature, keeping the tests tied to the capabilities being delivered.

For follow-up, Mhangami points viewers to the slides and Playwright documentation. She also acknowledges that she forgot to include the demonstration’s GitHub repository link.

The first audience question exposes a limit of the simple storefront example. The questioner already uses Playwright with Storybook to establish application state and asks how to handle more complex applications and admin panels. Mhangami suggests Playwright Agents because their specialized instruction files may help with state handling. She also suggests testing APIs directly where an API is available. These are proposed approaches; the answer does not demonstrate a fixture or state-setup strategy for a complex application.

A second question asks whether the tests can cover different display sizes. Mhangami confirms that mobile and desktop sizes can be checked, without showing viewport configuration.

The final question distinguishes a mobile-sized website from a native application: what about a Mac app or an iPhone app? Mhangami’s answer scopes the workflow presented here to browser-based testing. The mobile-versus-desktop answer concerns the browser experience, not a demonstration of native application automation.