From answering the page to sharing the on-call rotation

1:58:38and Vlad. We're excited to share our

1:58:40journey as well as our learnings around

1:58:43building an AI on call teammate at Meta

1:58:46that now runs real on call rotations

1:58:48every day and around the clock.

1:58:51We are part of the engineering team that

1:58:53builds the tooling platform and

1:58:55infrastructure for incident response on

1:58:58call experience and mitigations at meta.

1:59:01That space is evolving rapidly with the

1:59:04increase in AI capabilities.

1:59:07AI changed how fast we write code. About

1:59:103/4 of new code is now written with AI.

1:59:13And more AI generated changes reach

1:59:16production every single day. And some of

1:59:19it can cause production failures just

1:59:22like any other code change can.

1:59:25But on call did not speed up with it.

1:59:28Highquality code can pass review and

1:59:31still break in production. and when it

1:59:33does, someone still gets paged at 2 a.m.

1:59:36The bottleneck moved from writing code

1:59:38to operating it in production. And on

1:59:42call is where that operational load

1:59:44lands. So we asked if AI is writing the

1:59:48code, can AI also help operate it?

1:59:53And we think it can. Now let's quickly

1:59:57run through one of those on call pages

1:59:59with a simplified example.

2:00:02It's 2:00 a.m. and 12 alerts are firing.

2:00:05Same night, it can go two different

2:00:07ways. On the left is how most of us do

2:00:09this today. And notice we already have

2:00:12good AI. First, you sort the alerts by

2:00:15yourself. Checkout, payments, disk,

2:00:18which one to pick first. Let's say you

2:00:19pick checkout. Then you ask your

2:00:22assistant why P99 latency is high.

2:00:26It's genuinely useful. It pulls the

2:00:27metrics and finds the deployment at 149.

2:00:31And when you ask, it reads what changed

2:00:34and checks the runbook.

2:00:36It figured out that the time config was

2:00:38changed to a lower value. And based on

2:00:41the runbook, it suggests to roll back,

2:00:44which is all very helpful. However, if

2:00:47you look at the shape of it, it waited

2:00:49to be asked.

2:00:51You picked every next step. You were the

2:00:54glue between the questions and once it

2:00:58answered, it went dormant.

2:01:00At 239, you know the cause, but the roll

2:01:05back, the changes, and the follow-ups

2:01:07are still yours.

2:01:10Now, on the right, the same 12 alerts,

2:01:14but an AI teammate started before you

2:01:17woke up. It had already ranked them.

2:01:21Checkouts first, payments merged in as

2:01:24the same cause, and the other 10 handled

2:01:26separately.

2:01:28It read the dashboards, difficed

2:01:33two past incidents.

2:01:35You get a hypothesis, a 30-cond way to

2:01:39verify it, and a fix waiting for your

2:01:42review. At 2011, you verified it and

2:01:46you're back to sleep. That gap is what

2:01:49this talk is about.

2:01:53So why were you the glue?

2:01:56Every team was bolting automation onto

2:01:58on call one silo at a time. An alert

2:02:03script for alerts, a triage bot for

2:02:05tickets, a deploy bot for deploys, a fix

2:02:08it agent for incidents.

2:02:10Each one solves its slice, keeps its own

2:02:13context, and forgets it. Now replaying

2:02:16your 2 AM page, the alert script saw P99

2:02:20latency above 2 seconds. The triage bot

2:02:23saw three tickets saying checkout is

2:02:25slow. The deploy bot knew about a config

2:02:28change at 149 and the incident history

2:02:30had two similar past incidents. Every

2:02:33clue existed,

2:02:35but nothing connected them. So the on

2:02:39call became the glue exactly

2:02:42as we saw on the left side of the

2:02:44previous slide.

2:02:46Our bet was one AI teammate per rotation

2:02:50instead.

2:02:52One loop across every signal, one team

2:02:55memory that learns every shift, its own

2:02:58identity so it acts as itself and a

2:03:01human gate so you approve instead of

2:03:03operate. But giving one agent all that

2:03:07context and reach raises the stakes.

2:03:09Which is why autonomy has to be earned.

2:03:15Under the hood, the teammate or the AI

2:03:18teammate is the simplest agent. There is

2:03:21model using tools in a loop acting on an

2:03:24environment and reading the feedback.

2:03:27Everything else in this talk is what we

2:03:29[music] build around that loop to make

2:03:32it safe to leave it running. And for

2:03:35that I will hand over to Swapnikai and

2:03:37Vlad next to get into the actual details

2:03:40of how it works.

2:03:43To achieve this we started with an

2:03:45autonomy ladder. Think about how you

2:03:47onboard a new engineer to on call. First

2:03:50the shadow. The investigators suggest

2:03:53but they don't touch production. Then

2:03:56the pair the right to fix and another

2:03:59engineer approves it. With time they

2:04:01gain knowledge. We give our agent

2:04:04exactly the same ladder. At level one,

2:04:07it investigates and tells you what it

2:04:09found. It changes nothing. At level two,

2:04:12it acts with approval. It drafts the

2:04:14fix, but nothing reaches production

2:04:16until a human approves. At level three,

2:04:19it clears lower criticality work on its

2:04:22own and only interrupts you for what

2:04:24matters. High criticality work stays

2:04:27with humans.

2:04:29So what earns the next run? We use a

2:04:32simple test. How costly is a mistake and

2:04:35how easy is it to catch? So an action

2:04:38climbs only when mistakes are cheap.

2:04:41They're caught by tests, CI or review.

2:04:45and the agent already has a track record

2:04:47on that exact action. That's why the

2:04:50agent can close a noisy alert on its own

2:04:53while a production mitigation still

2:04:55waits for a human.

2:04:58At level one, the investigation is

2:05:00layered and stops at the first confident

2:05:02answer. The agent investigated the issue

2:05:05using the team's memory and runbook. It

2:05:08also shared the root cause analysis, but

2:05:10human had to apply the fix.

2:05:13At level two, it writes the fix with the

2:05:17root cause and a test plan. It answers

2:05:20reviewer's comments and fixes CI, but

2:05:23nothing reaches production until a human

2:05:25approves.

2:05:26Most drafts don't land, and that's fine.

2:05:30Review is the filter. At both levels,

2:05:33the agent does the homework, but you

2:05:36still had to grade every single paper.

2:05:39The help got much better, but the tasks

2:05:42requiring human input didn't get

2:05:44smaller. Level three shrinks the pile.

2:05:50Look at this example. 23 items came in,

2:05:53but only three need you. On the right,

2:05:57the agent handled the long tail, noisy

2:05:59alerts, duplicates, routine fixes that

2:06:02went through normal review and CI. Two

2:06:05more are still being worked. You don't

2:06:08have to drive each one to completion. In

2:06:11the middle is your much smaller decision

2:06:13queue. Each item makes clear what it

2:06:17needs from you with a run this button

2:06:19that has a call to action. And remember

2:06:22our 2 a.m. check out page that still

2:06:25stays human owned. The agent brings the

2:06:27investigation. The human owns the

2:06:30critical decision. It's the new high

2:06:32ladder again. Trusted to handle routine

2:06:35work, ready with the homework when it

2:06:37needs help. Not just less work per item,

2:06:40but fewer items waiting on you.

2:06:45That board is where the work lives. But

2:06:47nobody stares at a board all the time,

2:06:50right? So the agent is also available

2:06:52across many surfaces like chat and

2:06:55native incident surfaces. At 9 every

2:06:58morning, it posts a digest. 23 came in,

2:07:02three need you, 18 handled. Each thing

2:07:06that needs you come with a one-click

2:07:08next step. So approving the quota bump

2:07:12takes one tap from your phone. The 18 it

2:07:15handled are summarized, not listed. And

2:07:18the two that are still in progress tell

2:07:20you what it's doing about them. And when

2:07:23you want to dig in, you just reply in

2:07:25the thread. As you see on the right

2:07:28here, you open the checkout incident and

2:07:30ask for the evidence. It already has it.

2:07:34Errors start at 8:41, 2 minutes after a

2:07:37config push cut the payment client

2:07:39timeout. The same pattern as two past

2:07:41incidents. You ask a follow-up, is it

2:07:44only EU? And it answers with a

2:07:46breakdown. It drafts the roll back. You

2:07:49approve it right there and errors start

2:07:52recovering. You never opened a dashboard

2:07:54and you never had to explain the context

2:07:57because it was already there. That's

2:07:59what a teammate feels like. A morning

2:08:01hand off a colleague you can ask.

2:08:06Now under the hood of this AI teammate,

2:08:09it's AI automations. The first

2:08:12investigates on every new signal auto

2:08:15triage classifies it, checks its

2:08:18criticality and kicks off the right

2:08:20agent. An alert goes to the incident

2:08:23investigator, a task to the team's own

2:08:25agent, a test failure to a coding agent,

2:08:29a look up to memory and code search.

2:08:32The second keeps the system healthy.

2:08:35Unattended automation fails quietly. So

2:08:39on a schedule, it looks for stuck or

2:08:41failed runs, retrace and repairs broken

2:08:44schedules, and it only pages a human if

2:08:47it can't fix it itself.

2:08:50Nothing stalls silently.

2:08:53The third hands off at every shift

2:08:56change and every morning at 9, it

2:08:58summarizes the shift. What needs you

2:09:00with the fix ready and what it handled

2:09:03on its own posted to the team chat. Not

2:09:06only in the morning, but every urgent

2:09:08issue that needs user attention gets

2:09:10notified in chat with findings and draft

2:09:13fixes. It also tags the next on call in

2:09:15the chat during handoff. The entire

2:09:18cycle still continues

2:09:23and because it belongs to the team, it

2:09:26runs the agents and the skills the team

2:09:28already trusts. That's the slot in the

2:09:30middle. Bring your own agent, add your

2:09:33own skills. If a team already has an

2:09:36agent that has the team's runbooks and

2:09:38the knowledge base, the automations

2:09:40leverage it. Alongside that, it has our

2:09:43own incident investigator and a deep

2:09:45dive coding session.

2:09:48Work comes in from alerts, tickets,

2:09:51incidents, team posts, and chat. The

2:09:55agent itself is generic. What makes it

2:09:57good at an on call rotation is the

2:09:59team's agents skills, memory, and

2:10:02playbook. And every run is logged, so

2:10:05anyone on the team can inspect the

2:10:08trajectory or resume the conversation.

2:10:11Now, I'm handing off to my teammate Vlad

2:10:13to talk more on the shared agent.

2:10:19An uncle agent can't belong to just a

2:10:21single person because the uncle itself

2:10:23doesn't. So it belongs to the team.

2:10:26There's one teammate per rotation on one

2:10:29board. Wherever's on call can ask it to

2:10:32correct it or pick up its work mid

2:10:33investigation. It carries the team's

2:10:36uncle expertise, the playbook, lord

2:10:38profiles and past decisions from one

2:10:40shift to the next one the way a good

2:10:42handoff should. Here's what that looks

2:10:45like. Monday at 2 am, Sam is on call and

2:10:48tells it remember if checkout time is

2:10:51out after a config push, roll back the

2:10:53config first. It saves it to the memory.

2:10:57Three days later, Jordan is on call hits

2:11:00the exact same alert and just asks for a

2:11:03fix. The agent answers with some fix,

2:11:06says where it came from, and has the

2:11:08roll back drafted. Teach it once, the

2:11:11next phone call gets it.

2:11:14And it access the team its own identity

2:11:17owned by the rotation with only the

2:11:19access the team is comfortable granting

2:11:21it.

2:11:24And it learns from every uncle shift.

2:11:26Handoffs, decisions, and corrections all

2:11:29flow into memory. So ship number 30 is

2:11:32better than the ship number one. The

2:11:34same way a new uncle gets better every

2:11:36week. But memory erodess over time. So

2:11:40ours has to earn its place. It retrieves

2:11:43semantically uses what it finds, scores

2:11:46where it actually helped and promotes or

2:11:49retires entries automatically.

2:11:52When someone says that was wrong, the

2:11:54item reopens and the correction becomes

2:11:56memory.

2:11:59Three things made this work beyond our

2:12:01team.

2:12:02First, a loop that never stops

2:12:05listening. Every piece of feedback or

2:12:07course correction reopens the item and

2:12:09ultimately gets absorbed into memory.

2:12:12Feedback is an input to the system, not

2:12:15a story read once a quarter. Second,

2:12:18customize instead of forking. Teams

2:12:22bring their own playbooks and skill

2:12:24workflows, and it's still the same

2:12:26teammate underneath.

2:12:28Third, what we call one rotation to end.

2:12:32Identity, config, and memory are per

2:12:34team. So, add a rotation becomes a

2:12:37setup, not a new project.

2:12:39The agent is generic while the expertise

2:12:41is owned by the team.

2:12:45So what has changed for the people on

2:12:47call? Well, first their job shifted from

2:12:50driving deep investigations to making

2:12:52key decisions. They start from a

2:12:54hypothesis with linked evidence, not a

2:12:56blank task or dashboard.

2:12:59Second, the load is getting lighter.

2:13:02Early signs of on call load reduction

2:13:04are promising for teams that run it

2:13:06automatic autonomously and we want to

2:13:08measure it rigorously before we put a

2:13:10number on it.

2:13:12And the third every change that agent

2:13:14writes still meets a human. Fixes land

2:13:18only through normal review and

2:13:20continuous integration. And most drafts

2:13:22don't land. That's the point. Review is

2:13:25the filter.

2:13:27Uncle stops being attacks and becomes a

2:13:29review cube. Here's

2:13:32what we're focusing on next. First,

2:13:35mitigations that get approved, not just

2:13:37drafted. We great every mitigation that

2:13:40agent proposes and feed the outcome back

2:13:42so it earns its next level instead of

2:13:44being handed it. Second, skills are

2:13:47written from the chat so a team can turn

2:13:50its runbooks into skills without writing

2:13:52any code. And third, measuring the time

2:13:55we give back properly with a rigorous

2:13:57study of uncall time saved rather than

2:13:59usage counts.

2:14:01And the next rung on the ladder is uh

2:14:04earning medium criticality autonomy with

2:14:07evidence. On that, thank you so much for

2:14:10listening to us and hopefully that was

2:14:12helpful. Bye.