How Building with AI Can Double the Throughput of Your Engineering Team

Intercom’s opening chart poses a concrete question: how does an established SaaS company return to growth while its peers slow down? Brian Scanlan presents a U-shaped recovery in Intercom’s revenue growth against declining growth among publicly traded SaaS companies. The company had pivoted toward AI the weekend ChatGPT appeared. By this point, the privately held Irish American business had roughly 1,400 employees across six cities, with R&D led from Dublin and engineering concentrated in Europe, increasingly supplemented by forward-deployed engineers.

Scanlan follows the chart with a joke about shutting Intercom down live onstage, recalling a previous live deployment. The substantive reinvention is its customer-support agent, Fin, rather than an autocomplete field attached to an existing product. He points to New York Times coverage of Intercom’s transformation and reports more than 8,000 Fin customers, describing its average resolution rates as industry-leading.

Scanlan describes Fin revenue as approaching 100 million, without specifying a currency or reporting period in the talk. Fin’s original announcement coincided with GPT-4’s release; that announcement described internal testing and a forthcoming limited beta, so the timing should not be read as general availability that day. Intercom had already been building AI features since approximately 2018. Modern language models expanded what it could do with support questions, and Fin’s customers included Anthropic, Snowflake, Linear, Glean and LaunchDarkly.

The next step was a specialized model. Scanlan says Intercom’s own model serves all Fin English text conversations and is cheaper, faster and better at its customer-support task than frontier models such as Sonnet. He reports approximately two million resolutions per week and says direct access to the model suite is available. These are company-reported deployment and performance claims; the talk does not provide the evaluation conditions behind the model comparison.

Inside engineering, the transformation was less immediate. Scanlan introduces himself as a senior principal engineer with 12 years at Intercom, working in the platform group. Its remit includes uptime, performance, security, cost management, observability, internal developer productivity and the company’s largely Ruby on Rails monoliths. Fast, iterative shipping was already central to its conception of product quality, captured in Shipping is your company’s heartbeat—an essay Scanlan remembers Honeycomb turning into stickers.

That made AI-assisted development an obvious investment. Engineers used GitHub Copilot, adopted Cursor and explored Augment. Yet by the middle of the preceding year, the results were disappointing: some tasks were marginally better or more enjoyable, but the organization had not fundamentally changed how it built software. The conviction that models and agent harnesses would transform knowledge work remained stronger than the results of simply distributing tools.

The response was 2X: double engineering throughput within a year without doubling the team. Intercom chose code changes per R&D person as its primary productivity measure, alongside developer surveys and tools such as DX. This made the ambition organizational: better individual experiences with an assistant should eventually produce a visible increase in delivery across R&D.

The measure was deliberately imperfect. Counting changes cannot capture every dimension of useful engineering work, and making a measure a target changes behavior. But those limitations were not a reason to avoid expecting a substantial throughput increase. The initiative, announced around the preceding June, became both a named project and a dedicated team.

The program also coincided with a major improvement in coding models and harnesses. Scanlan recalls a principal engineer’s astonishment around the Christmas break as capabilities changed, and explicitly credits that shift with contributing substantially to 2X’s success. The eventual outcome therefore reflects a changing technical baseline as well as Intercom’s organizational work.

Leadership made the expectation explicit. Intercom updated job descriptions so that adopting AI was part of meeting expectations for engineers, designers and product managers. The message had to recur across forums; a single announcement would not change established habits.

The supporting work combined reinforcement, practice and staffing:

• Visible examples: Automated Slack announcements surfaced skill updates, while teams celebrated useful work and shared techniques.
• Time to learn: Hackathons and AI immersion days gave people opportunities to change their working methods.
• Dedicated support: A growing, full-time 2X team helped hundreds of R&D staff adopt the tools.

For a medium or large organization, Scanlan recommends assigning strong engineers to this work full time. Mandating AI use without providing that support leaves each employee to solve the same adoption problems independently.

Intercom standardized on Claude Code, replacing an approach in which people chose among several editors and agents. Scanlan compares the decision to concentrating on one cloud provider: spreading investment across platforms can prevent a team from accumulating deep operational knowledge and reusable infrastructure. Specific, consequential needs can justify exceptions, but continually reconsidering the preferred model should not consume the effort needed to make a platform work.

The ambition was to make Claude capable of acting like a senior Intercom engineer on technical work. Anything an engineer could do on a laptop should be accessible to the agent. That included access to real systems, governed by the company’s existing permissions, controls and audits—not unrestricted permission to alter databases. Those controls provided the basis for granting agents useful capabilities.

Access alone would leave the agent missing the knowledge that makes an experienced employee effective. Intercom had to teach it Rails conventions, architecture, React patterns, testing standards and security rules accumulated over years of development. The operating loop was straightforward:

1. Use the agent for technical work.
2. Observe where it takes the wrong path or misses a convention.
3. Update shared guidance so the correction benefits subsequent work.
4. Capture reusable knowledge in skills and use hooks to enforce required behavior.

A correction becomes more valuable when it improves the shared platform, rather than only rescuing one session.

Delivering that shared platform was itself operational work. Intercom pushed internal Claude plugins directly to employee laptops, bypassing Claude Code’s update mechanisms. Scanlan compares debugging installations across hundreds of laptops to managing Python installations: a capable agent still depends on reliable local setup and distribution.

The scope includes debugging, testing and planning as well as writing code. Engineers initially drive the agent closely, then aim to intervene less as the environment and guidance improve. The purpose remains delivering working products to customers. Scanlan believes the existing building blocks already support moving substantial amounts of the software lifecycle to an agent-first approach, even without further improvements in models or harnesses.

Written principles help hundreds of people interpret that ambition consistently. Giving an agent access to production can feel uncomfortable, but the larger shift is in the engineer’s role. Scanlan recalls working as a Unix sysadmin: visiting data centers, racking servers, running cables and configuring networks. Cloud infrastructure moved that work toward SRE and automation, with greater impact and higher pay. He sees the current transition as another move up the stack, compressed into a much faster industry-wide change.

Intercom’s technical conservatism carries over from its Rails monoliths: use a small number of tools extremely well. That favors investment in reusable knowledge over a proliferation of custom multi-agent orchestrators and opinionated workflows. Specific tools will change; a clear description of how to do work correctly at Intercom is likely to remain useful.

The preferred unit of investment is a small, durable, testable skill. Historical code changes, incidents and completed work supply examples against which to backtest it. Instead of judging a skill only by whether its instructions sound reasonable, the team can examine whether it handles known work at a high standard. Finding the right internal knowledge remains a challenge, so discoverability matters alongside correctness.

Continuous improvement includes making skills capable of updating themselves. Meanwhile, the platform should remain able to adopt capabilities as Anthropic ships them, rather than becoming trapped behind custom infrastructure. Intercom may eventually change providers; the durable investment is its accumulated working knowledge, not a commitment to rebuild every feature of the agent harness.

Explicitly asking an agent to run a particular skill is still useful, and Scanlan does it himself. But the desired interface is a problem description: let the agent infer the intent and choose the relevant capabilities. This removes the requirement that the person asking for help already know every available internal procedure.

A security incident made the distinction tangible. Someone had accidentally published Snowflake table metadata to a public GitHub repository. Scanlan opened Claude Code, asked it to join the incident’s Slack channel and investigate, and discovered that an internal skill already captured the company’s data-breach policies, assessment criteria and response procedure. He had not known the skill existed.

Claude downloaded the files, analyzed the exposure, concluded that it was innocuous and supplied next steps. Scanlan reports that the investigation took about two minutes, compared with his estimate of 20 minutes to locate the policy and do the analysis manually. The useful mechanism was the combination of accessible evidence, discoverable policy and a well-written skill; the timing is his account of this particular incident.

Adoption remained uneven even inside Intercom. A maturity rubric, which Scanlan compares with Steve Yegge’s discussion of engineering maturity, helps people see a path beyond occasional assistance. The progression is to use Claude Code broadly, automate recurring work, capture that work in skills, become proficient at writing skills and then improve the skills themselves.

The later stages move the focus beyond the individual session. Engineers write skills for others, improve them using evaluations and session data, and optimize the environment in which agents operate. That can mean changing software architecture, documentation or working methods to make effective behavior easier for today’s agents. The advanced stages on the slide describe contribution to a shared capability, rather than simply more frequent tool use.

Intercom made the single-platform decision in December and began rolling it out in January. Scanlan reports that Intercom doubled PR throughput in less than a year. He presents a sharp inflection after standardization, although that sequence does not isolate the effects of platform choice from the capability improvements and organizational changes already described. A separate Claude Code PR dashboard figure is described only as being in the nineties, too imprecisely to establish an exact percentage here.

Code review then became the bottleneck. Scanlan reports a 17.6% automatic-code-approval rate in the dashboard shown during the talk. The measurement window and denominator are not specified. Getting there required more than asking Claude whether a change looked acceptable: Intercom backtested approvers against historical work, had humans label outputs, and used those results to establish confidence. It also shaped pull requests toward safe, simple changes that could be approved automatically.

Scanlan says Intercom worked with its auditors on SOC 2, ISO 27001 and HIPAA requirements. His account is that human approval is not inherently required for every change, but the organization must understand the approval process precisely and maintain appropriate audit controls. This is a description of Intercom’s approach, not a blanket compliance determination for other systems.

The approval system uses a tested suite of agents, including Codex for code review. That is an explicit exception to platform consolidation: a focused multi-model review process can justify additional tools. Scanlan expresses confidence that these controls prevent the automated process from degrading the environment or adding risk.

Scanlan goes further, suggesting that well-defined agents can remove risks that human reviewers miss. Assessing agent behavior requires evidence, however, and he flags the earlier skill-invocation figures as unreliable. Intercom uses two distinct data paths:

Scanlan says the internally available usage telemetry contains no private information. That statement concerns the basic telemetry; the separately collected session transcripts are a richer source of behavioral evidence.

The session archive closes the loop between written instructions and observed behavior. It can help the team see whether a skill is effective, rather than merely popular. Scanlan says the loop is already useful, while acknowledging that Intercom could extract more value from the data.

Defect trends provide another view of the outcome. Scanlan says defects had been increasing until recently, but were now being closed faster than ever. Some teams deliberately pursued backlog zero and worked through large inventories of defects. Other reductions appeared to follow naturally from making the work faster. The observation combines planned cleanup with an improvement in day-to-day repair capacity.

Scanlan also reports improving code quality according to metrics used by a Stanford research group to which Intercom supplied its code. He does not identify the group or explain its methodology in the talk, so the claim supplies a directional quality signal rather than a reproducible quality benchmark.

The shared plugin system had hundreds of contributors and thousands of lines of code. Base plugins handled session transcripts, session syncing and safety hooks. On top of that infrastructure, specialized skills captured particular kinds of engineering work.

One example was flaky-spec repair. Intercom had hundreds of thousands of tests, and frequent shipping had encouraged engineers to push past intermittent failures. Scanlan did not create the repair skill by sitting down and attempting to enumerate every possible cause of a flaky test. He gave the agent a goal, guided it through actual repairs and used the repeated work to develop the procedure.

The resulting skill organized accumulated knowledge into lookup tables, shortcuts and progressively disclosed detail. A compact Markdown outline illustrates that organization: keep the initial workflow easy to find, then expose the relevant repair knowledge when the investigation needs it.

# Repair a flaky spec

## Goal
Find and repair the cause of an intermittent test failure.

## Investigation
- Inspect the failing spec and the available failure evidence.
- Consult the relevant entries in the repair lookup table.
- Develop and validate a repair against the observed failure.

## Repair lookup table
Record recurring symptoms, their established causes, and
links to the detailed repair guidance.

## Improve this skill
After a validated repair, update the relevant guidance
with what the investigation established.

The important asset is the repair knowledge accumulated through use. Scanlan says the skill produces fixes that would impress him if they came from senior Rails engineers.

More throughput also exposed infrastructure limits: Scanlan says CI became overwhelmed and had to be fixed, without detailing the remediation. Meanwhile, Claude Code spread beyond software engineering. That raised a broader organizational question: if the same agent platform supports engineering, design and product management, should the boundaries between those roles change?

Intercom was experimenting with single-person product teams. Scanlan describes shipping code that lets other people’s agents sign up for Intercom, while using internal skills to perform product-management work himself. The closing slide places these experiments alongside broader Claude Code adoption, replacing runbooks, remote agents and automatic approvals. The scope has expanded from producing code faster to changing who can carry a product idea through to delivery.

Scanlan expects these practices to become widespread soon. He closes by directing readers to his personal site and describing ways to interact with Fin through its messenger and a CLI. The substantive ending is the role experiment: once organizational knowledge and technical capabilities are available through a shared agent platform, an individual can attempt work that previously required coordinating several specialties.