Give Your On-Call Its Own AI Teammate - Swaapnika Motapothula, Chinmay Gandhi & Vlad Tsvang, Meta

A change can pass code review and still wake someone at 2 a.m. At Meta, Swaapnika Motapothula, Chinmay Gandhi and Vlad Tsvang build the tools engineers use when that happens. They saw AI accelerate coding while the work of operating the resulting software still fell to the engineer carrying the pager. Their team runs an AI teammate on real rotations around the clock. A simplified night with 12 alerts illustrates the job they gave it.

In the first version, you sort through checkout, payments and disk alerts, pick checkout, and ask an assistant why its 99th-percentile latency is high. It pulls the metrics and finds a deployment at 1:49 a.m. You ask what changed. The assistant finds that the change lowered a timing setting and checks the runbook, which recommends a rollback. By 2:39 you know the likely cause. You chose each question, and the rollback and follow-up work are still yours.

What would be waiting for you if the investigation started before you woke up? Given the same 12 alerts, the teammate ranks checkout first, groups payments with it because the two share a cause, and handles the other ten separately. It reads the dashboards, inspects the 1:49 configuration change and finds two similar incidents. You get a hypothesis, a quick way to check it and a fix ready for review. In this version of the night, you verify the finding and go back to sleep.

Replay the checkout page from the perspective of the existing tools. The alert script saw P99 latency exceed two seconds. The triage bot saw three tickets reporting slow checkout. The deployment bot knew about the 1:49 configuration push. The incident history contained two precedents. Each tool had enough context to do its own job. Connecting a user report to a latency spike, a specific change and a possible remedy was a separate job, and it belonged to whoever answered the page.

The team chose one teammate per rotation. It can investigate across those signals, keep a team memory across shifts, and act under its own identity. Underneath, the agent is a model that calls tools, reads what happened and uses the results to choose its next step. Giving that loop enough reach to investigate checkout raises a practical question. Who decides whether the proposed remedy should touch production?

A new on-call engineer starts by shadowing someone experienced. The newcomer investigates incidents and suggests fixes. Later the pair can divide the work of writing and approving a change. The team gives the agent a similar progression.

• Level one — investigate. The agent searches in layers, drawing on team memory and the runbook. It stops at the first answer it considers confident and shares its root-cause analysis. A human applies the fix.
• Level two — draft a fix and follow it through review. The agent writes the change, explains the cause and supplies a test plan. It answers reviewer comments and addresses continuous integration (CI) failures. A change needs human approval to reach production. Most drafts never land.
• Level three — clear selected routine work. The agent can close a noisy alert on its own. High-criticality work stays with the engineer, including the decision to apply a production mitigation.

Promotion applies to a particular action. What would a mistake cost? How readily would tests, CI or review catch it? How has the agent performed on that same action? A history of correctly closing noisy alerts earns trust for closing noisy alerts. Rolling checkout back carries different consequences. At the first two levels, the agent can do extensive homework while leaving the engineer with just as many items to grade. The third level changes how many items reach the engineer.

The example board groups work by who needs to act next. Of 23 incoming items, the agent has handled 18, including noisy alerts, duplicates and routine fixes that went through normal review and CI. Two are still in progress with the agent. The remaining three form the engineer’s decision queue. Each item says what it needs and offers a button to take that action. A piece of unfinished work can stay visible on the board without becoming another task for the engineer to drive.

Checkout still belongs in the human queue. The investigation can be ready while the critical decision is pending. The board records the work, and the teammate also participates in chat and in the incident itself. The engineer can ask a question or act on a finding wherever they encounter it.

At 9 each morning, the teammate posts a digest. In the 23-item example, the 18 handled items get a collective summary. The two in progress get a description of what the agent is doing, and each of the three decisions gets a one-click next step. An engineer can approve a quota increase from a phone. A decision that deserves a closer look can continue in the thread.

What would justify a checkout rollback? In the incident-thread example, the engineer asks for evidence. Errors began at 8:41, two minutes after a configuration push shortened the payment client timeout. Two previous incidents followed the same pattern. The engineer asks whether the problem is confined to the EU and gets a regional breakdown. With the timing, precedents and scope available in one conversation, the engineer approves the drafted rollback there. Errors begin to subside.

The rotation sees one teammate. Behind it, three automations run under different conditions. A new signal can start an investigation at any hour. Recovery has its own schedule, and a handoff happens even while some investigations are still open.

• On a new signal, triage and route it. The system classifies the signal and checks its criticality. An alert goes to an incident investigator, a task to the team’s own agent, a test failure to a coding agent, and a lookup to memory and code search.
• On a schedule, check the automation itself. A health check looks for stuck or failed runs, attempts recovery and repairs broken schedules. If recovery fails, it pages a human. Even an investigation that produces no answer has a way to get attention.
• At handoff, tell the incoming engineer what needs a decision. Every shift change and every morning at 9 trigger a summary in team chat, with prepared fixes and an account of work already handled. At handoff, the teammate tags the incoming engineer. An urgent issue can trigger its own chat notification with findings and a draft fix.

A team that already trusts an agent with its runbooks and knowledge base can use it to handle tasks alongside the incident investigator and coding agent. Work can arrive through alerts, tickets, incidents, team posts or chat. The playbook, skills and memory give the generic agent local expertise. Every run is logged, so another engineer can inspect the steps it took and resume the conversation partway through an investigation.

One rotation shares one board. Whoever takes the pager can question the teammate, correct it or pick up its work. At 2 a.m. on Monday, Sam tells it to remember a piece of advice. If checkout starts timing out after a configuration push, roll back the configuration first. Three days later, Jordan sees the same alert and asks for a fix. The teammate retrieves Sam’s advice, tells Jordan where it came from and has the rollback drafted.

The rotation owns the teammate’s identity and decides what access to grant it. Remembering Sam’s advice lets it prepare a change for Jordan. Applying a production mitigation still takes a human decision, and that decision can itself become part of the next shift’s memory.

Advice can age. The teammate retrieves entries by meaning, uses relevant ones in its work, then scores them on whether they helped. It can promote a useful entry or retire one automatically. A correction has two effects on different timescales. If an engineer says the answer was wrong, the work item reopens immediately. The correction also enters memory, where a later investigation can retrieve it.

Taking the system beyond its original team depended on keeping that correction path open and giving each rotation control of its expertise. Teams supply their own playbooks and skills while sharing the underlying implementation. Identity, configuration and memory are set up per team. Adding a rotation becomes a setup task, and teams can bring an existing workflow with them without maintaining a fork.

The change engineers can describe in practice is where their work starts. They have a hypothesis with linked evidence to examine and a decision to make. Teams running the agent autonomously report early signs of a lighter load. How much on-call time they get back remains open. The planned study will measure engineers’ time directly, since usage counts alone cannot answer that question.

Code changes still go through human review and CI, and most drafts never land. That makes approval a useful outcome to learn from. The team grades every proposed mitigation and feeds the outcome back, aiming to produce more fixes that engineers will accept. It also wants teams to be able to turn a runbook into a skill from chat without writing code. The next autonomy decision concerns medium-criticality work. To earn it, the teammate will need a record that justifies taking those particular actions.