# Always-on agents run production without the on-call tax

Justin Smith

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/vSx5IULvBns-always-on-agents-run-production-without-on) · [Video](https://www.youtube.com/watch?v=vSx5IULvBns)

## Always-on agents run production without the on-call tax

Faster code delivery creates more production work. Background agents combine triggers, persistent execution and production context to monitor changes and keep teams informed.

Familiarity with CI/CD, production monitoring and on-call rotations will help you follow the examples.

### Who runs all the code we can now ship?

[0:15](https://www.youtube.com/watch?v=vSx5IULvBns&t=15s)

What happens to production when writing code becomes much faster? Justin Smith approaches that question from the operational side: he introduces himself as a founding product engineer at Resolve AI, with more than fifteen years in monitoring and observability, including work as an architect on Splunk’s Observability Suite and a tenure at VMware. His interest in product design and front-end architecture also shapes the question of where engineers should interact with operational agents.

AI-assisted development produces larger pull requests and more frequent releases. It also lets people ship code without necessarily understanding its operating principles. That increases development output, but it leaves the downstream work intact. Smith cites an unnamed survey putting 70% of engineering time outside writing code, which he describes as running shipped software. The survey’s population and methodology are not supplied.

![Slide stating “70% of engineering time is spent on running software,” beside a small Coding circle and a larger Production circle.](https://ai.engineer/api/talk-essay-media/6a244455f9d8cff5c5eafae4237d7a156db9cdc61fc078aa4da66413763fd457)

Running software occupies the larger share of engineering time in this slide’s coding-versus-production diagram.

That work includes maintaining platforms, scaling infrastructure, debugging incidents, serving on call and shipping hotfixes. It also includes handling alerts, updating runbooks, restoring services, managing escalations and answering other teams’ questions. Faster code generation does not remove these responsibilities; it can increase the number of changes engineers must understand.

Smith also describes tightening token allowances, rising prices and stricter company controls on AI use. These are his observations, rather than a comparison of named providers or plans. Together, they motivate a broader requirement: **production AI needs domain context around the model**, so that model capability translates into useful work inside a particular environment.

### From responding to pages to remembering the environment

[3:13](https://www.youtube.com/watch?v=vSx5IULvBns&t=193s)

Production systems were already complex before AI accelerated development. Multiple teams operate interdependent systems, pursuing local goals that must stay aligned with organizational goals. Resolve’s founding hypothesis is that increased coding velocity will create more production issues, making AI assistance valuable on the operational side as well. Smith says work with engineering customers operating at large scale informs that hypothesis; it is not a demonstration that on-call work has disappeared.

[Resolve’s agents](https://resolve.ai/blog/agents-to-run-and-fix-your-software) began with on-call work: triaging incoming alerts and investigating root causes. An engineer returning to an on-call rotation after several weeks may not know the latest changes or every system involved. An agent can help assemble that missing context. When a single page expands into a cross-team incident, the coordination problem grows too: responders need a shared account of where the incident is happening and what it affects.

Background agents address what remains when there is no active fire: context handoffs, one-off issues and the nagging question of whether a subsystem is healthy. Smith describes an architecture combining models, context, reasoning and actions. Its difficult requirement is **an evolving knowledge system**. More capable models do not automatically know how a company’s services interact, where its hotspots are or which causal chains matter. That understanding must be captured and kept current as the environment changes.

An enterprise layer sits beneath that architecture, though Smith does not detail it here. The rest of the talk narrows to background agents rather than immediate incident response.

![Slide titled “Agents to run and fix your software,” with the Background agents box highlighted and surrounding architecture boxes faded.](https://ai.engineer/api/talk-essay-media/040ff4295d06cf2a75524445bf3b55c747cb813c0e10d390498b1e155e65561a)

Background agents are highlighted within the architecture for running and fixing software.

### The work that has no alarm

[8:05](https://www.youtube.com/watch?v=vSx5IULvBns&t=485s)

Asked about agents outside coding, audience members suggest meeting reviews, market research and therapy. Smith connects meeting reviews to Granola, which had appeared during the presentation. These are audience examples of non-coding workflows, not demonstrated Resolve capabilities. They lead into a less conspicuous category of engineering work: responsibilities that lack the clear starting signal of a page or incident bridge.

Someone still has to watch a deployment finish, prepare a morning incident digest and investigate recurring P99 latency drift that has not crossed an alert threshold. Someone must produce a capacity report against a quarterly goal and check system health before a customer complains. These tasks consume attention even when nothing explicitly announces that they are due.

Smith defines a task as execution plus the context needed to execute it well. Tool access can get an agent to a dashboard. Production context lets it recognize that a metric looks wrong and decide to investigate another layer.

| Capability | What it contributes |
| --- | --- |
| Execution | Open the dashboard and retrieve observations |
| Production context | Judge whether an observation matters |

The distinction is between obtaining a value and understanding its significance. A useful background agent needs both.

### Triggers, persistent execution and shared learning

[11:15](https://www.youtube.com/watch?v=vSx5IULvBns&t=675s)

Three questions define the background-agent machinery: when does it work, how does it run and how does it know what to do? The activation mechanisms are simple enough to compose into many workflows.

| Trigger | Example |
| --- | --- |
| Schedule | Morning report or weekly on-call handover |
| Event stream | Deployment event or relevant Slack message |
| Direct request | A message asking the agent to do work |

Smith’s team hands over on-call duties on Thursdays. A scheduled agent can prepare the preceding week’s important trends for the next engineer. Event-driven work instead begins when a connected system emits something worth acting on, such as a CI/CD deployment or a Slack message.

Execution lives in the cloud, so closing a laptop does not stop the agent. Smith describes a sandbox with a filesystem that lets the agent organize its work. Beneath individual runs, a memory and learning system supports reflection: what a task discovers can improve its next run, and shared knowledge can carry lessons into other tasks.

A task system supplies the work definition, imports existing skills and connects the necessary integrations. These primitives are intended to support more than a fixed catalog. The four workloads that follow are examples Smith says have worked inside Resolve and with customers.

### Four workloads to hand over

[13:44](https://www.youtube.com/watch?v=vSx5IULvBns&t=824s)

**Deployment monitoring** starts with the observation that every change creates an opportunity for something to go wrong. Existing CI/CD checks provide useful baselines, but they are not exhaustive: different releases need different signals. Feature flags and infrastructure changes may also happen outside the monitored pipeline, leaving the on-caller to reconstruct who changed what only after an alert fires. A background agent can watch those change events and assess system stability.

**Scheduled health and anomaly checks** inspect systems without waiting for an alert. A morning dashboard review might look for unusual behavior overnight. A task can also be temporary: after changing an integration, ask an agent to watch the third-party service for a week, then stop. **Operational reports and handoffs** turn those observations and other operational activity into information teammates can use.

**First response to engineering questions** targets interruptions. Smith describes watching Slack channels while trying to build software: the questions may be easy to answer, but responding breaks concentration, while not responding leaves colleagues waiting. Internally, Resolve’s agent already has access to much of the information people ask about. It watches critical channels and decides whether it has enough confidence to answer. If uncertain, it can DM an engineer for confirmation before posting a response.

### A monitoring plan for this release

[17:16](https://www.youtube.com/watch?v=vSx5IULvBns&t=1036s)

The deployment demonstration takes place in an explicitly fake application environment and a demo Slack workspace. Posting a GitHub tag gives the agent a potential release event. It first identifies that event as a release, then inspects the changes to choose telemetry that could reveal whether the system has entered an abnormal state. The result is a monitoring plan specific to that release.

**The agent complements the CI/CD pipeline.** It adds the change-specific attention that would otherwise require an engineer to watch each release closely. In the example involving checkout and the currency service, Smith points to checkout latency, error rates and the Kafka pipeline as signals to inspect along the relevant causal chain. The useful mechanism is the connection from the change to its possible effects, rather than a fixed list of checks applied to every release.

![Slack deploy-log channel with an open thread containing Resolve messages, a deployment checklist, and a subsequent monitoring update.](https://ai.engineer/api/talk-essay-media/8d1f7742ea5d3ecaefb655208340c924cd30175b6cfa40e9ca7c467ae300f219)

A Slack deployment thread shows a monitoring checklist and follow-up results.

Monitoring can continue beyond one pass. Smith contrasts a hard-coded fifteen-minute recheck with user-guided autonomy: an agent might wait another hour because a problem is intermittent, or revisit the deployment after three days to assess whether it remains healthy and produced the expected effect. These are possible follow-up decisions, not elapsed runs shown in the demonstration. The same approach can attach to feature flags, infrastructure changes and other event sources.

### Feedback that changes the next report

[19:54](https://www.youtube.com/watch?v=vSx5IULvBns&t=1194s)

The next Slack example is an on-call handoff report summarizing the preceding day. It contains investigation summaries, notable changes, completed work and critical open items. This gives the incoming engineer a shared starting point, but the displayed report is verbose.

Smith replies in the report’s thread to ask for a shorter version. The intended operation goes beyond editing the current answer: the agent updates the underlying task so future scheduled reports are less verbose. He says this works, but moves on before the update completes. The visible result is the submitted feedback, not a revised report.

![Slack oncall-handoff channel displaying a Daily Operational Handoff Report, an alert table, and a thread with a request to make the report shorter.](https://ai.engineer/api/talk-essay-media/f4c292d6ebbeaa93aaf083304ae3eb7c4025187dbaf51666222678507aaf87ce)

Feedback requesting a shorter report appears in the Slack handoff thread.

### Participate selectively, configure conversationally

[21:01](https://www.youtube.com/watch?v=vSx5IULvBns&t=1261s)

The channel-watching demonstration removes the need to invoke Resolve explicitly. A user can post a problem without an `@mention`, or even without knowing the agent exists. The configured watcher responds when it believes it can help and otherwise stays silent. Smith shows a message for which it has chosen not to respond. Selective participation matters here: watching a channel does not mean answering every message.

A separate UI exposes tasks, reports, previous runs and the underlying work performed to accomplish a task. That inspection surface provides visibility, while Slack—or Microsoft Teams—is intended to remain the everyday interaction surface. Engineers can inspect the agent’s work without moving every operational conversation into another product.

Creating a recurring health summary follows the same conversational approach:

1. Ask the agent for a recurring health summary for the team.
2. Let it explore the connected environment and ask clarifying questions about content, report type and verbosity.
3. Test the initial task before sharing it with the rest of the team.

The task definition emerges from the environment and the user’s requirements, rather than requiring the user to specify every integration step up front.

### Bring production context into existing workflows

[22:56](https://www.youtube.com/watch?v=vSx5IULvBns&t=1376s)

The best work to delegate depends on the company. Its service relationships, operational needs and accumulated knowledge determine where an agent can save the most attention. That is why Smith returns to the knowledge system: understanding the particular environment makes it possible to select useful work, rather than merely automate a generic checklist.

For teams building their own agent harnesses, Smith says the capabilities shown can be accessed through MCP servers, bringing Resolve’s production context, operational work and learning into another system. Current [Resolve MCP documentation](https://docs.resolve.ai/resolve-mcp-server) uses `/mcp/v2` and documents chat and investigation tools; it does not establish exhaustive access to every task operation shown in the recording. Existing skills can come along too, avoiding duplicated workflow definitions.

**The expensive part of operational work is understanding a changing environment, as well as executing the task.** Scheduled and triggered agents make that work composable. Resolve remains a place to inspect findings, while routine interaction can stay in the tools where the team already works. The intended shift is from repeatedly assembling context by hand to giving persistent agents enough context to carry useful work forward.

## Resources

- [Behind the Build: Always-on background agents](https://resolve.ai/events/behind-the-build/agents-for-engineering-workflows) — Justin Smith's companion session on background agents for deployment monitoring, anomaly analysis, operational reporting and resource reviews.
- [Agents that run your software](https://resolve.ai/blog/agents-to-run-and-fix-your-software) — Resolve's launch announcement explains background-agent triggers, conversational workflows, production context and integration with other agents.
- [Resolve MCP Server](https://docs.resolve.ai/resolve-mcp-server) — Current setup instructions and available tools for connecting an MCP client to Resolve's chat and investigation capabilities.
- [How much time do developers spend actually writing code?](https://www.sonarsource.com/blog/how-much-time-do-developers-spend-actually-writing-code/) — A 2019 survey summary separating time spent writing code from maintenance, testing, security, meetings and operational work.

## Slides

### Justin Smith

Justin Smith brings 15+ years in developer tools and observability, including work at Splunk and VMware, with a focus on product design and frontend architecture.

[0:23](https://www.youtube.com/watch?v=vSx5IULvBns&t=23s) [Portrait of Justin Smith with his role and background in developer tools and observability.](https://assets.ai.engineer/sites/aiecode2025/sha256/4c8c4c0ffa241f933b403aeb3eb8de44df0ed39a87a4dc9acc6e58879a9b0066.webp)

Justin Smith
Founding Product Engineer, Resolve AI

15+ years building developer tools and observability platforms at companies like Splunk and VMware. Focused on product design, frontend architecture, and turning complex infrastructure data into usable experiences.

Resolve.ai

### The first wave of AI changed how we build software

Smith sees larger, more frequent PRs, potentially from people unfamiliar with the code, alongside gains in developer productivity.

[0:57](https://www.youtube.com/watch?v=vSx5IULvBns&t=57s) [AI coding tool logos accompany points about larger, more frequent pull requests and developer productivity.](https://assets.ai.engineer/sites/aiecode2025/sha256/e607c0be8a1c7af7cf9f6f3823f19a64c22a85693eaee407ed8844b934b50284.webp)

The first wave of AI changed how we build software

Bigger PRs

More frequently

From developers (and non-developers?) that might not know the code

BUT Developer productivity!

Resolve.ai

### 70% of engineering time is spent on running software

The slide cites an IDC 2025 survey for the claim that 70% of engineering time goes to running software. Its larger production circle spans tasks from scaling infrastructure and responding to alerts to restoring service.

[1:42](https://www.youtube.com/watch?v=vSx5IULvBns&t=102s) [A small coding circle and a larger production circle illustrate the slide's claim about engineering time.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c42b9d08956e631c7477fa7670e19728a66d339cc5ed3475e270b1ff35f43e3.webp)

70% of engineering time is spent on running software

Coding
Production

Maintain platforms
Scale infra
Debug incidents
Ship hotfixes
Alert response
Runbook updates
Patch vulnerabilities
Restore service
Escalations

Software Engineering

aws
splunk>
su
mo

Source: IDC 2025 Survey Spotlight - How Do Software Developers Spend Their Time? - doc #US5320475
Resolve.ai

### Why?

A network connects production tasks from alert response to hotfixes and incident debugging, illustrating how multiple teams and systems must work together.

[3:11](https://www.youtube.com/watch?v=vSx5IULvBns&t=191s) [A network diagram illustrates the multiple teams and systems involved in operating software in production.](https://assets.ai.engineer/sites/aiecode2025/sha256/c940f3838b2622ab17bb4f1845446cd6042def24ebf3450e6bd2ccc0a2d08df5.webp)

Why?

Operating software in production requires multiple teams and systems

Scale infra
Escalations
Alert response
Patch vulnerabilities
Runbook updates
Ship hotfixes
Restore service
Maintain platforms
Debug incidents

Resolve.ai

Interface text:
AIE WF26 — Always-on age...
2:25 PM - 2:45 PM
Take notes

### How do you fix this imbalance?

The speaker proposes using AI in production to manage the added complexity AI brings to products and systems.

[3:51](https://www.youtube.com/watch?v=vSx5IULvBns&t=231s) [The slide answers the question with “AI for Prod.”](https://assets.ai.engineer/sites/aiecode2025/sha256/cd463925cd7cc7a437f6ebc916db81fa0c87f5924bdcdff3b32c624ada7ea193.webp)

How do you fix this imbalance?

AI for Prod

Resolve.ai

### World class engineering teams using AI for prod

Resolve highlights Zscaler’s 30% fewer engineers in war rooms, DoorDash’s 87% faster investigations, and Coinbase’s hundreds of weekly production debugging sessions.

[4:06](https://www.youtube.com/watch?v=vSx5IULvBns&t=246s) [Customer examples highlight productivity, investigation speed and operational hardening, with additional company logos below.](https://assets.ai.engineer/sites/aiecode2025/sha256/c3b13dba3f692db16c8d8565aa409db2d25be88fbc5c6c82beec732bee6a6cad.webp)

World class engineering teams using AI for prod

Zscaler
Increased productivity - 30% fewer engineers in war rooms

DOORDASH
Increased reliability - 87% faster investigations

coinbase
Increased operational hardening - 100s of production debugging sessions weekly

salesforce
MongoDB
snowflake
expedia group
CISCO

toast
Fireworks AI
BLUEGROUND
MSCI
GUIDEWIRE

veza
upwind
GAMETIME
Pinecone
iFIT

Resolve.ai

### AI for prod

Resolve places on-call, incident and background agents above a shared architecture and enterprise platform. The speaker highlights learning as a way to keep track of how services interact as systems evolve.

[4:43](https://www.youtube.com/watch?v=vSx5IULvBns&t=283s) [An overview describes production AI agents and shows their architecture, evaluations and enterprise platform.](https://assets.ai.engineer/sites/aiecode2025/sha256/d69f755834855667db07bd2de747bd77218ec9e643bbbf8fe35d63de9a8be3ee.webp)

AI for prod

+ AI agents that handle on-call, incidents, and daily operational tasks
+ Trusted to get to root cause in the most demanding production environments
+ Fits into your agent ecosystem, or build your agents on Resolve AI

AI AGENTS
On-call agents
Incidents agents
Background agents
Your agents

AGENT ARCHITECTURE
Models · Context · Reasoning · Actions · Learning
Evals

ENTERPRISE GRADE PLATFORM
Integrations · Cost efficiency · Agent Security · Usage & audit · Deployment

Resolve.ai

### Agents to run and fix your software

Resolve’s agent types share an architecture spanning models, context, reasoning, actions and learning, with evals feeding improvements back. The talk focuses on background agents for work beyond immediate fires.

[7:35](https://www.youtube.com/watch?v=vSx5IULvBns&t=455s) [A detailed diagram lists agent types, architecture components, evaluations and enterprise platform capabilities.](https://assets.ai.engineer/sites/aiecode2025/sha256/b8ca320d976e44edda57b5661460ec20c71d8ecfc749cbb8d94e4b25394974d6.webp)

Agents to run and fix your software

AI AGENTS

On-call agent
• Autonomous triage
• Root cause backed by evidence

Incidents agent
• Shared investigation thread
• Multiplayer investigation
• Drives incident channel

Background agents
• Schedule on event or trigger
• Multi-turn chat

Your agents
• MCP/API/Skill

AI AGENT ARCHITECTURE

Models
• Post-trained models
• Frontier model orchestration

Context
• Knowledge graph
• Tool fluency
• Context engineering

Reasoning
• Causal reasoning
• Multi-agent coordination

Actions
• Guardrails
• Scoped autonomy

Learning
• Explicit feedback
• Implicit signals

Evals
• Absorb new model and harness releases
• Domain specialized eval framework
• Calibrated to mirror engineer’s investigation

output
improves

ENTERPRISE GRADE PLATFORM

Integrations
• 60+ connectors
• Custom tool integrations

Cost efficiency
• Optimized token usage
• Credit based pricing

Agent Security
• Tenant isolation
• Data protection

Usage & audit
• Usage reports
• Trace agent reasoning

Deployments
• Cloud
• Resolve AI Managed VPC w/BYOK *

Resolve.ai

### Are you using agents in your daily work today?

The speaker asks for a show of hands on agents used beyond coding. Audience examples include meeting reviews, market research and therapy.

[8:03](https://www.youtube.com/watch?v=vSx5IULvBns&t=483s) [A live show-of-hands poll asks whether attendees use agents in their daily work.](https://assets.ai.engineer/sites/aiecode2025/sha256/6760e5d579a28819906102eef01f5ff424cea50b3b636f4a13f55942bab56283.webp)

Are you using agents in your daily work today?

LIVE POLL • SHOW OF HANDS

Resolve.ai

### Most production work is always there

Pages and incident bridges make critical work visible. Deploy monitoring, digests, p99 investigations, capacity reports and health checks can lack an obvious trigger, leaving toil to accumulate unnoticed.

[8:51](https://www.youtube.com/watch?v=vSx5IULvBns&t=531s) [Examples of invisible production toil include watching deployments, writing digests, investigating drift, and running health checks.](https://assets.ai.engineer/sites/aiecode2025/sha256/6a94b39b52f2d1481e745d6bfe9bdf2a72b925a77c7747f6464a6affd21ff2a0.webp)

Most production work is always there and not always a critical event

On-call has a page. Incidents have a bridge.

INVISIBLE TOIL
This work has no critical trigger that makes anyone account for it, so it stays invisible and adds up.

Watch the deploy that just went out
Write the morning deploy and incident digest
Re-investigate the p99 drift that came back
Produce the weekly capacity report
Run the recurring health check nobody owns

Resolve.ai

### Every operational task requires two things

Loading a dashboard is execution; recognizing that a metric looks wrong requires production context. The speaker argues that context matters more and is large and growing.

[10:14](https://www.youtube.com/watch?v=vSx5IULvBns&t=614s) [A task combines execution with production context; the slide emphasizes the effort involved in navigating the environment.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c7a9a9d559353a1646ed64f6fd0d126c49aacbeebc7e0bae19a47705235cd43.webp)

Every operational task requires two things

Task = Execution × Production Context

Execution
The task in isolation, the actual analysis or change. Small and roughly fixed.

Production Context
What you know about the specific environment the work is being done in to know how to execute and evaluate the work being done. Knowledge and tools. Large and growing.

Production Context » Execution.
Navigating your environment is a major contributor to toil.

Resolve.ai

### Every background agent operates on three principles

Background agents wake on a schedule, task, or trigger; retain state and delegate based on complexity; and use predefined tasks, skills, and integrations backed by Resolve AI architecture.

[11:13](https://www.youtube.com/watch?v=vSx5IULvBns&t=673s) [Three columns describe when an agent works, how it works, and how it knows what to do.](https://assets.ai.engineer/sites/aiecode2025/sha256/894725870ed60343b5eb52554d4bc4bb2588af91dda874b62a8065b336a1e6c4.webp)

Every background agent operates on three principles

When does it work
Agents wake up on a predefined schedule, task, or a trigger.

How does it work
Run indefinitely. Persistent in state and delegate work based on the complexity.

How does it know what to do
Predefined set of tasks, skills, and integrations - backed by Resolve AI architecture.

Resolve.ai

### When does the agent work?

The agent wakes on a schedule, matching event keywords, or a Slack message, and tracks the trigger to guide its response. One scheduled example is preparing a weekly on-call handover.

[11:24](https://www.youtube.com/watch?v=vSx5IULvBns&t=684s) [Agents can wake on schedules, event streams, or direct messages and act according to the trigger.](https://assets.ai.engineer/sites/aiecode2025/sha256/a03a550b643e1c54ff7f63d744e4b58679e5d779ca913ca6e8a585eae19b9441.webp)

When does the agent work?
Runs continuously and involves you when a decision is required

On schedule
Cron and interval.
Calendar-aligned work like a morning status report or an on-call handoff at shift change. Intervals for periodic maintenance.

Event streams
Keyword-matched ingestion.
Deploy events with "deploy," "release," or "rollback" auto-trigger the deploy monitor. No human needed.

Message-based
Direct human contact.
A Slack DM or channel message wakes the agent immediately. The message is the signal, no polling delay.

The agent tracks why it woke up each time, and acts accordingly.

Resolve.ai

### How does the agent work?

The cloud agent enters standby when idle, while its tasks, memory and files persist across restarts. A shared knowledge system lets it apply what it learns from one task to another.

[12:17](https://www.youtube.com/watch?v=vSx5IULvBns&t=737s) [Three columns describe continuous availability, a durable sandboxed filesystem, and learning through knowledge and memories.](https://assets.ai.engineer/sites/aiecode2025/sha256/7ff30bb9ff73050dd3bd7c926cf837dd8b8a2b52aebeea55afee81b050582dcd.webp)

How does the agent work?

ALWAYS AVAILABLE
Runs indefinitely
Runs in the Cloud. Enters standby when idle and wakes when there's work, so it is always available without always running hot.

DURABLE STATE
Sandboxed filesystem
Tasks, memory, and files persist across restarts automatically, so nothing in flight is lost.

LEARNING
Knowledge and Memories
Knowledge loops to understand your systems. Learns usage and preference patterns over time.

Resolve.ai

### How does the agent know what to do?

Tasks define the work, skills encode how to do it, and integrations connect the agent to tools and infrastructure. One task can draw on multiple skills.

[12:57](https://www.youtube.com/watch?v=vSx5IULvBns&t=777s) [Tasks define what to do, skills define how to do it, and integrations connect the agent to tools and infrastructure.](https://assets.ai.engineer/sites/aiecode2025/sha256/3f9b891483dd4f67b00a958710590c74c00e865990ce4397c888559c6bcf135c.webp)

How does the agent know what to do?

TASKS
What to do
Stateful work items. One-off or scheduled, prioritized, and durable across restarts.

SKILLS
How to do it
Prompts and scripts that encode how to use specific capabilities to gather evidence and take action.

INTEGRATIONS
What it reaches
Connections to telemetry, tools, and infrastructure: logs, metrics, traces, dashboards, code, and more.

A single task may pull on multiple skills to reach its objective.

Resolve.ai

### What can I hand off to background agents?

The speaker introduces four workloads for background agents and says the system’s basic primitives can support creative background activities.

[13:10](https://www.youtube.com/watch?v=vSx5IULvBns&t=790s) [Section title asks what work can be handed off to background agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/b73192b54f6158f31eb823800e134a6b807d90a9e646a30f7d71dc35eec0f332.webp)

What can I hand off to background agents?

Resolve.ai

### Four workloads to easily hand off to background agents

Background agents can monitor deployments and service health, prepare handoffs, and answer engineering questions. Resolve’s internal agent watches critical Slack channels and can ask a person to confirm an uncertain answer.

[15:38](https://www.youtube.com/watch?v=vSx5IULvBns&t=938s) [Three visible workload cards cover deployment monitoring, scheduled health and anomaly checks, and operational reports and handoffs.](https://assets.ai.engineer/sites/aiecode2025/sha256/0ae553467ac58caaef181c086ec62dfe7977c8147ec8840b8b25c6f680eb4f4e.webp)

Four workloads to easily hand off to background agents

Deployment monitoring
DEPLOY EVENT
Watch every rollout, run post-deploy health checks, catch regressions before they page.
Template Engineering Deploys Agent

Scheduled health and anomaly checks
SCHEDULE / ALERT
Periodic checks on the services and datastores you watch, including recurring drift.
Template PostgreSQL Watch

Operational reports and handoffs
SCHEDULE
Morning deploy and incident digest, on-call handoff at shift change, weekly status.
Template Daily Pulse

Resolve.ai

[15:48](https://www.youtube.com/watch?v=vSx5IULvBns&t=948s) [Four cards describe deployment monitoring, scheduled health checks, operational handoffs, and answering engineering questions.](https://assets.ai.engineer/sites/aiecode2025/sha256/26fc00b365414f61b622b262aeed3690a045ee20a76d81d78ccd30c6687f0046.webp)

Four workloads to easily hand off to background agents

Deployment monitoring
DEPLOY EVENT
Watch every rollout, run post-deploy health checks, catch regressions before they page.
Template Engineering Deploys Agent

Scheduled health and anomaly checks
SCHEDULE / ALERT
Periodic checks on the services and datastores you watch, including recurring drift.
Template PostgreSQL Watch

Operational reports and handoffs
SCHEDULE
Morning deploy and incident digest, on-call handoff at shift change, weekly status.
Template Daily Pulse

First responder to engineering questions
SLACK MESSAGE
Resolve AI agents becomes the first responder to any questions your engineering team gets
Template Primary on-caller

Resolve.ai

### Deployment monitoring and operational reporting demo

The speaker introduces a demo on deployment monitoring and operational reporting, then moves to a demo Slack environment.

[17:09](https://www.youtube.com/watch?v=vSx5IULvBns&t=1029s) [Demo introduction for deployment monitoring and operational reporting.](https://assets.ai.engineer/sites/aiecode2025/sha256/362f9ddee0df38a0eb8a012d2dfc5558af938acaa7f447df32f2e8f758261c87.webp)

Let’s see a demo
Deployment monitoring and Operational Reporting

Resolve.ai

Interface text:
25

### Deployment monitoring in Slack

In a demo Slack channel, Resolve monitors a checkout release’s latency, errors and Kafka pipeline. Its thread reports an all-clear at 10 minutes and checkout P95 of ~24.7ms at 15 minutes.

[17:34](https://www.youtube.com/watch?v=vSx5IULvBns&t=1054s) [A deployment log thread shows an initial health check and a later all-clear monitoring update.](https://assets.ai.engineer/sites/aiecode2025/sha256/fb6744f77b735e5fe4e8f748d32ae87d2317728122e10306062254793ab2e6ce.webp)

Thread

Justin Smith — Jun 10th at 6:00 PM
https://github.com/clouddatalabs/opentelemetry-demo/releases/tag/2.0.5.3

3 replies

Resolve APP — Jun 10th at 6:00 PM
Good. Initial check for v2.0.5.3 is clean — checkout’s convertCurrency() now uses an inline rate map instead of the currency gRPC call. All signals healthy. Follow-up scheduled for 01:15Z.
Sent from Ecommerce App’s Resolve Agent

Resolve APP — Jun 10th at 6:08 PM
📋 v2.0.5.3 Deploy Monitor — Initial Check: CLEAN ✅

Change: Checkout replaces currency service RPC with inline conversion map (src/checkout/main.go).

Monitoring:
• Checkout latency, error rates & Kafka pipeline (checkout→kafka→fraud-detection)
• Pod health/restarts across namespace
• Frontend latency for downstream impact

Results (T+10min): All clear — Kafka lag 0, no new errors, checkout P95 stable at ~48ms, fraud-detection consuming normally. No alerts.
Sent from Ecommerce App’s Resolve Agent

Resolve APP — Jun 10th at 6:19 PM
✅ T+15min check complete — all clear. Checkout P95 down to ~24.7ms (50% improvement from removing currency gRPC call), Kafka lag 0, no errors or alerts. Monitoring concluded.
Sent from Ecommerce App’s Resolve Agent

Interface text:
Resolve AI Demo (P...)
Slackbot
Threads
Huddles
Drafts & sent
Directories
Starred
Drag and drop important stuff here
Channels
# ask-resolve
# deploy-log
# ecommerce-app-alerts
# oncall-handoff
# report-problems
Direct messages
Apps
Resolve
Slack

# deploy-log
Messages
Add canvas
Wednesday, June 10th

Justin Smith — 4:07 PM
joined #deploy-log.

Justin Smith — 4:08 PM
@Resolve
1 reply 22 days ago

Resolve APP — 4:08 PM
was added to #deploy-log by Justin Smith.

Justin Smith — 4:08 PM
@Resolve what can you do for me?
16 replies Last reply 22 days ago

Justin Smith — 5:26 PM
pushing to production: https://github.com/clouddatalabs/opentelemetry-demo/releases/tag/2.0.5.2
5 replies Last reply 22 days ago

Justin Smith — 6:00 PM
https://github.com/clouddatalabs/opentelemetry-demo/releases/tag/2.0.5.3
3 replies Last reply 22 days ago

Justin Smith — 7:08 PM
production push: https://github.com/clouddatalabs/opentelemetry-demo/releases/tag/2.0.5.4
3 replies Last reply 22 days ago

Justin Smith — 8:22 PM
https://github.com/clouddatalabs/opentelemetry-demo/releases/tag/2.0.5.5
3 replies Last reply 22 days ago

Message #deploy-log
Reply...
Also send to #deploy-log

### On-call handoff summary

The agent’s Slack handoff summarizes completed investigations and flags work still needing attention, including recurring frontend errors, payment failures, and consumer lag.

[19:52](https://www.youtube.com/watch?v=vSx5IULvBns&t=1192s) [A Slack channel displays an on-call handoff with incident summaries and open action items.](https://assets.ai.engineer/sites/aiecode2025/sha256/124a3d42a12afe8e6802bf8e36b70b07768078f51428167d5f8bd4f43419655d.webp)

3. ✅ Work Completed
• Payment incident (07-01): ruled out deploy/config; localized failure to runtime gRPC quote-hop timeout on PlaceOrder — useful narrowing despite no formal close.
• Auto-investigation coverage: every real firing had a triage canvas spun up.
• Smoke tests fired as designed (18) — monitoring pipeline healthy.
• ⚠️ No confirmed mitigations / permanent fixes captured for frontend errors or consumer lag. No “auto-resolved after deploy” pattern — the window had no deployments for core ecommerce services.

4. 📋 Open Action Items
1. Frontend error logs — likely still firing (13x in ~28h, latest 06:27Z / [illegible]). Verify current state; drive root cause via frontend runbook if active.
2. Payment PlaceOrder 5xx / quote-hop gRPC timeouts — incident [illegible] never formally closed. Confirm 5xx cleared; re-open + investigate quote service runtime if recurring (checkout runbook).
3. savedsearches consumer lag (Kafka + MSK, 5 firings) — check consumer-group backlog; apply Kafka consumer-lag runbook if lag persists.
4. RDS CPU high (us-east-2) — single fire 07-01 20:04Z ([illegible]). Low priority; confirm self-cleared.
5. Investigations ending in error / missing summary — multiple auto-investigations stopped with errors, weakening handoff quality. Tooling/platform follow-up.

Sources: search_team_alerts (50 firings) · list_team_investigations (50 canvases) · canvas contexts [illegible] + [illegible].
Sent from Ecommerce App's Resolve Agent

Daily On-Call Handoff — window 06-25 16:00Z → 07-02 16:00Z

Alerts: 232 firings / 7 rules (68 synthetic smoke-tests → 164 production).
• Frontend error logs above threshold — 63 real (129 incl. MS Teams copies), chronic every 1–3h, still active (latest 07-02T06:27Z). Top concern.
• savedsearches consumer lag (Kafka + AWS MSK) — ~28 firings in bursts.
• RDS CPU (us-east-2) — 7 daily spikes.

Investigations: ~150+ (mostly auto). Two payment-outage incidents, both chaos-induced, each recovered ~22 min:
• #inc-payment-outage-phoenix (07-01) — frontend chaos set bad CHECKOUT_SERVICE_ADDR; auto-restored. Postmortem requested.
• #inc-payment-outage-bravo (06-30) — kill-cart chaos scaled cart→0 replicas. Postmortem requested.

🔴 Critical open items:
1. Frontend errors chronic/unresolved — root-cause via runbook (needs human attention).
2. Checkout/PlaceOrder 5xx — 2 chaos-induced P1s in 48h; confirm postmortems + consider gating chaos from customer alerting.
3. savedsearches consumer lag recurring — check backlog/scaling.

Full report DM'd to owner. Note: no explicit resolve events in feed — confirm frontend/lag state in Grafana before standing down.
Sent from Ecommerce App's Resolve Agent

Interface text:
Resolve AI Demo (P...)
# oncall-handoff
Messages · Add canvas
Slackbot · Threads · Huddles · Drafts & sent · Directories
Starred · Drag and drop important stuff here
Channels: ask-resolve · deploy-log · ecommerce-app-alerts · oncall-handoff · report-problems
Direct messages · Apps · Resolve · Slack
Today · 9:06
Message #oncall-handoff

### Operational handoff report refinement

The handoff groups 50 alert firings and flags unresolved issues. Justin asks Resolve to shorten the report; it is still processing. The presenter says the feedback can update the task for future runs.

[19:58](https://www.youtube.com/watch?v=vSx5IULvBns&t=1198s) [A Slack handoff report summarizes alerts and displays an alert summary table.](https://assets.ai.engineer/sites/aiecode2025/sha256/609b089ef232fa178914d56e8ebda59b6ec103ba07fe84042b61fcf4421eda1d.webp)

4. Close the loop on recurring RDS CPU alert

Full report DM’d to the owner.
Sent from Ecommerce App’s Resolve Agent

Resolve APP 9:04 AM
🔄 Daily Operational Handoff Report

Team: _resolve_org · Window: 2026-06-25T16:00:00Z → 2026-07-02T16:00:00Z (trailing 7 days) · Generated: 2026-07-02T16:00:00Z

TL;DR — 50 alert firings, all clustered in the last ~36h (2026-07-01 → 07-02). Dominant real signal: Frontend error logs above threshold (13 distinct firings, latest 06:27Z today, likely still active). A payment-outage incident (‘inc-payment-outage-phoenix’) ran 07-01 — PlaceOrder 5xx / gRPC quote-hop timeouts, not deploy-related, never formally closed. Kafka/MSK consumer lag on ‘savedsearches’ recurred. Several auto-investigations ended in error without a final summary.

1. 🚨 Alert Summary
Total: 50 firings (all fire), landing 2026-07-01T02:33Z → 2026-07-02T06:27Z.

1. 🚨 Alert Summary
Alert Rule | Service/Team | Firings | Type | Notes
Frontend error logs above threshold | ecommerce-app / app | 13 | Real | Most recurring. Latest ‘2026-07-02T...
Frontend error logs (copy for msteam) | ecommerce-app / app | 13 | Dup route | Same condition, MS Teams copy
[Smoke Test] High Service Latency | smoke-test | 18 | Synthetic | Automated fire/warn pairs — no action
AWS MSK consumer lag is high | savedsearches / app | 2 | Real | Fires 07-01 ‘05:27Z’ + ‘18:31Z’
AWS MSK consumer lag (copy for ms t... | savedsearches / app | 2 | Dup route | MS Teams copies

Interface text:
Resolve AI Demo (P...)
# oncall-handoff
Messages · Add canvas
Yesterday · Today
Slackbot · Threads · Huddles · Drafts & sent · Directories
Starred · Drag and drop important stuff here
Channels · # ask-resolve · # deploy-log · # ecommerce-app-alerts · # oncall-handoff · # report-problems
Direct messages · Apps · Resolve · Slack
Filter table... · 1 · 2
Message #oncall-handoff

[20:45](https://www.youtube.com/watch?v=vSx5IULvBns&t=1245s) [A Slack thread shows a request to shorten a handoff report and the agent processing it.](https://assets.ai.engineer/sites/aiecode2025/sha256/d079cc0b17322a2513d84f1904d87bc5d10fe176c0d63f1922e9b791c0781739.webp)

Critical open action items:
1. Fix flagd-ui write endpoint (add validation) — root cause of all 3 incidents
2. Fix chaos-pipeline error handling (corrupts flagd / downs cart)
3. Alert-config review — chronic errors ran ~11 days undetected
4. Close the loop on recurring RDS CPU alert

Full report DM’d to the owner.
Sent from Ecommerce App’s Resolve Agent

Resolve APP 9:04 AM
🔄 Daily Operational Handoff Report

Team: _resolve_org · Window: 2026-06-25T16:00:00Z → 2026-07-02T16:00:00Z (trailing 7 days) · Generated: 2026-07-02T16:00:00Z

TL;DR — 50 alert firings, all clustered in the last ~36h (2026-07-01 → 07-02). Dominant real signal: Frontend error logs above threshold (13 distinct firings, latest 06:27Z today, likely still active). A payment-outage incident (‘inc-payment-outage-phoenix’) ran 07-01 — PlaceOrder 5xx / gRPC quote-hop timeouts, not deploy-related, never formally closed. Kafka/MSK consumer lag on ‘savedsearches’ recurred. Several auto-investigations ended in error without a final summary.

1. 🚨 Alert Summary
Total: 50 firings (all fire), landing 2026-07-01T02:33Z → 2026-07-02T06:27Z.

1. 🚨 Alert Summary
Alert Rule | Service/Team | Firings | Type | Notes
Frontend error logs ... | ecommerce-app / app | 13 | Real | Most recurring. Late...
Frontend error logs (... | ecommerce-app / app | 13 | Dup route | Same condition, MS ...
[Smoke Test] High S... | smoke-test | 18 | Synthetic | Automated fire/war...

Thread:
• Payment incident (07-01): ruled out deploy/config; localized failure to runtime gRPC quote-hop timeout on PlaceOrder — useful narrowing despite no formal close.
• Auto-investigation coverage: every real firing had a triage canvas spun up.
• Smoke tests fired as designed (18) — monitoring pipeline healthy.
• ⚠️ No confirmed mitigations / permanent fixes captured for frontend errors or consumer lag. No “auto-resolved after deploy” pattern — the window had no deployments for core ecommerce services.

4. 📋 Open Action Items
1. Frontend error logs — likely still firing (13x in ~28h, latest 06:27Z / [illegible]). Verify current state; drive root cause via frontend runbook if active.
2. Payment PlaceOrder 5xx / quote-hop gRPC timeouts — incident [illegible] never formally closed. Confirm 5xx cleared; re-open + investigate quote service runtime if recurring (checkout runbook).
3. savedsearches consumer lag (Kafka + MSK, 5 firings) — check consumer-group backlog; apply Kafka consumer-lag runbook if lag persists.
4. RDS CPU high (us-east-2) — single fire 07-01 20:04Z ([illegible]). Low priority; confirm self-cleared.
5. Investigations ending in error / missing summary — multiple auto-investigations stopped with errors, weakening handoff quality. Tooling/platform follow-up.

Sources: search_team_alerts (50 firings) · list_team_investigations (50 canvases) · canvas contexts [illegible] + [illegible].
Sent from Ecommerce App’s Resolve Agent

1 reply
Justin Smith Just now
@Resolve too verbose, make it shorter
Resolve APP
Processing...

Interface text:
Resolve AI Demo (P...)
# oncall-handoff · Thread
Messages · Add canvas
Yesterday · Today
Slackbot · Threads · Huddles · Drafts & sent · Directories
Starred · Drag and drop important stuff here
Channels · # ask-resolve · # deploy-log · # ecommerce-app-alerts · # oncall-handoff · # report-problems
Direct messages · Apps · Resolve · Slack
Filter table...
Message #oncall-handoff · Reply... · Also send to # oncall-handoff
Resolve Thinking...

### Channel watcher setup

Justin sets up Resolve to watch #report-problems and answer when it can help. The channel shows unprompted questions about chargebacks and checkout latency alongside an ordinary success update.

[20:52](https://www.youtube.com/watch?v=vSx5IULvBns&t=1252s) [A Slack channel shows instructions for an agent to watch messages and respond when useful.](https://assets.ai.engineer/sites/aiecode2025/sha256/8dbfda1adb57007a1141e8610e7c7bb3f846fbe9c70df9f0d4e06207b28290b1.webp)

#report-problems

Thursday, June 11th

Justin Smith 7:54 AM
joined #report-problems. Also, Resolve joined.

Justin Smith 7:57 AM
@Resolve /new-task create a "Report Problems Triage" project, create a watcher for this channel (#report-problems) and for each message you see, determine if you should respond to it or not based on context in the message, you have access to the code and all the operational reports about things. If you think you can answer the question, analyze the question and then post a response back to the user. You should be terse and helpful to people, they can always ask for more information later if they want, so keep your responses brief.
3 replies · Last reply 21 days ago

Justin Smith 8:09 AM
We're getting reports of increased chargebacks this morning. Could the fraud-detection service be lagging behind on processing orders? Maybe a flagd timeout happened again like that February incident?
3 replies · Last reply 21 days ago

Justin Smith 8:28 AM
Hey, I'm seeing checkout take way longer than usual, like orders are hanging for a few seconds before confirming. Is the checkout service having latency issues or is Kafka slow to acknowledge writes again?
1 reply · View thread

Justin Smith 8:34 AM
Checkout processed 14,000 orders successfully today - new record for the team 🎉

Tuesday, June 16th

Victor Li 2:48 PM
joined #report-problems. Also, Dave and 5 others joined.

Interface text:
Resolve AI Demo (P...
Search: man
Messages · Add canvas
Slackbot · Threads · Huddles · Drafts & sent · Directories
Starred
Drag and drop important stuff here
Channels: ask-resolve · deploy-log · ecommerce-app-alerts · oncall-handoff · report-problems
Direct messages
Apps: Resolve · Slack
Message #report-problems

### Agent response in a channel thread

The channel watcher replies without a mention, but its displayed answer discusses fraud detection and chargebacks while the thread asks about checkout latency.

[21:03](https://www.youtube.com/watch?v=vSx5IULvBns&t=1263s) [A Slack thread shows a checkout latency question and an agent response discussing fraud detection and consumer lag.](https://assets.ai.engineer/sites/aiecode2025/sha256/efb9adfe07811ea71ae4f9045971fcc15766b86fe1663ad37f88150dd8b01f79.webp)

#report-problems

Thursday, June 11th

Justin Smith 7:54 AM
joined #report-problems. Also, Resolve joined.

Justin Smith 7:57 AM
@Resolve /new-task create a “Report Problems Triage” project, create a watcher for this channel (#report-problems) and for each message you see, determine if you should respond to it or not based on context in the message, you have access to the code and all the operational reports about things. If you think you can answer the question, analyze the question and then post a response back to the user. You should be terse and helpful to people, they can always ask for more information later if they want, so keep your responses brief.
3 replies · Last reply 21 days ago

Justin Smith 8:09 AM
We're getting reports of increased chargebacks this morning. Could the fraud-detection service be lagging behind on processing orders? Maybe a flagd timeout happened again like that February incident?
3 replies · Last reply 21 days ago

Justin Smith 8:28 AM
Hey, I'm seeing checkout take way longer than usual, like orders are hanging for a few seconds before confirming. Is the checkout service having latency issues or is Kafka slow to acknowledge writes again?
1 reply · 21 days ago

Justin Smith 8:34 AM
Checkout processed 14,000 orders successfully today - new record for the team 🎉

Tuesday, June 16th

Victor Li 2:48 PM
joined #report-problems. Also, Dave and 5 others joined.

Thread
Justin Smith Jun 11th at 8:28 AM
Hey, I'm seeing checkout take way longer than usual, like orders are hanging for a few seconds before confirming. Is the checkout service having latency issues or is Kafka slow to acknowledge writes again?

1 reply
Resolve APP Jun 11th at 8:31 AM
Thoughts

Checked recent data on this:

Fraud-detection looks healthy as of last check (~04:40 UTC):
• Kafka consumer lag = 0 across all 16 partitions on the orders topic
• Both fraud-detection replicas running, actively consuming (~353K+ records processed)
• No flagd changes or errors in recent deploys — flagd config stable, no timeout indicators

Re: flagd February incident pattern — the known issue is a missing authValidationKey flag causing gRPC NOT_FOUND errors. No signs of that recurring in recent monitoring.

However — there were Kafka consumer lag alerts at 03:28Z and 03:37Z today, though confirmed as a recurring daily pattern (same time window fires Jun 10, Jun 8). These resolve quickly.

If chargebacks are up this morning specifically, the lag isn't from fraud-detection falling behind on Kafka — it's processing in real-time. The issue may be upstream (orders not being written to Kafka at all) or a logic gap in fraud rules themselves. Want me to dig deeper into traces or fraud-detection processing times?

Sent from Ecommerce App's Resolve Agent

Interface text:
Search: man
Resolve AI Demo (P...
Home
DMs
Activity
Files
More
Slackbot
Threads
Huddles
Drafts & sent
Directories
Starred
Drag and drop important stuff here
Channels
# ask-resolve
# deploy-log
# ecommerce-app-alerts
# oncall-handoff
# report-problems
Direct messages
Apps
Resolve
Slack
Messages
Add canvas
Message #report-problems
Reply...
Also send to # report-problems

### Scheduled task report

Resolve’s task list opens a daily code and feature flag report, with its schedule and previous runs alongside it. The report records no new commits or flag configuration changes in the covered 24 hours.

[21:38](https://www.youtube.com/watch?v=vSx5IULvBns&t=1298s) [The Resolve AI tasks page lists scheduled reports and other agent tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/43bcb2ae89b616c4a2de800b035cf9a037dfdb6c1e2f2e366b3d2e7c61373181.webp)

Tasks

Search tasks
Active  All  Done
+ New task
All agents · 61  Private · 1  Public · 60

Daily Code Changes & Feature Flags Summary  Public
Scheduled · Daily at 9:00 AM (America/Los_Angeles)
Jul 3, 9:00 AM PDT

Daily Operational Handoff Report  Public
Scheduled · Custom: 0 9 * * 1-5
Jul 3, 9:00 AM PDT

Daily Operational Handoff Report  Public
Scheduled · Daily at 9:00 AM (America/Los_Angeles)
Jul 3, 9:00 AM PDT

Daily Operational Handoff Report  Private
Scheduled · Daily at 8:00 AM (America/Los_Angeles)
Jul 3, 8:00 AM PDT

Task Suggestion Seeding  Public
Scheduled · Daily at 6:00 AM
Jul 2, 11:00 PM PDT

Weekly On-Call Handoff Report  Public
Scheduled · Every 7 day · Generate a weekly on-call handoff report summarizing the last 7 days of alerts, incidents, changes, and open issues to brief the incoming on-call engineer.
Jul 7, 1:49 PM PDT

Weekly Alert Categorization & RCA Prioritization Report  Public
Scheduled · Weekly on Mon at 9:00 AM (America/Los_Angeles)
Jul 6, 9:00 AM PDT

Daily 9am Non-Urgent Alert Summary  Public  needs attention
Manual
Jun 27, 11:03 PM

CAMLS Daily Service Health Report (9 AM PST)  Public  needs attention
Scheduled · Daily at 9:00 AM (America/Los_Angeles)
Jun 22, 9:01 AM

Interface text:
app0.resolve.ai/tasks
Resolve AI
Home
Tasks
Knowledge
Investigations
Chats
Today
Can you delete the Slack message?
Last 30 days
Monitor feature flag impact on metrics
Daily operational handoff report
What are 5 contextualized problem qu...
Why are you not answering here?
Should Resolve respond to #report-pr...
What Slack channels are you in?
What slack channels are you in?
Daily operational handoff report
Daily operational handoff report?
What can Resolve AI do for me?
What are the Resolve traces?
Older
Diagnostic
Analytics
Admin
Copy Current URL
Help
Justin Smith
Ecommerce App

[21:41](https://www.youtube.com/watch?v=vSx5IULvBns&t=1301s) [A scheduled task report summarizes code changes and feature flag status, with previous runs listed alongside.](https://assets.ai.engineer/sites/aiecode2025/sha256/7da50997189916b9cc55c9683af143bbe5ea42fc5ca953daa4a14ebff272a158.webp)

Daily Code Changes & Feature Flags Summary

Date: 2026-07-02 (Thursday)
Report Time: 9:00 AM PDT
Period Covered: 2026-07-01 16:00 UTC – 2026-07-02 16:00 UTC (24h)
Cluster: consolidated-otel-demo (us-east-2)

Code Changes
No new commits were pushed to the opentelemetry-demo repository in the last 24 hours.

The most recent commit on main remains from 2026-05-29:
• b20f0c4 — fix: Prometheus OOM, alert label enrichment, frontend-proxy routing, resolve_timeout (#98) by Jonathan Canales

The codebase has now been stable for 34 days with no new merges to main.

Notable Open Branches (unmerged)
• 3 local resolveai/* fix branches still active (gRPC retry policy, kill-payment chaos fixes)
• ~13 remote resolveai/* branches pending (auth key fixes, message-metrics-collector removal, MSK/Kafka lag fixes, checkout log cleanup)
• Several fix/ad-service-* branches (connection pooling, high latency, blocking DB query) remain unmerged

Feature Flags
No feature flag configuration changes detected. demo.flagd.json is unchanged at HEAD b20f0c4, and the flagd deployment (v0.11.1, Helm rev 67) is running healthy at 1/1 with no restarts and no rollout in the window.

Current Flag State (all flags ENABLED, default variants = off/inactive):
Flag | Default Variant | Status | Change?
product-cache-refresh | off | Inactive | No change
recommendationCacheFailure | off | Inactive | No change
adManualGc | off | Inactive | No change

Interface text:
Tasks > Daily Code Changes & Feature Flags Summary
Temporal
Run now
daily-code-changes-feature-flags-summary
daily-operational-handoff-2026-07-02
oncall-handoff
Scheduled
Next run  Jul 3, 9:00 AM PDT
Frequency  Daily at 9:00 AM (America/Los_Angeles)
Runs  37 runs
Jul 2, 9:00 AM  done
Jul 1, 9:00 AM  done
Jun 30, 9:00 AM  done
Jun 29, 9:00 AM  done
Jun 28, 9:00 AM  done
Jun 27, 9:00 AM  done
Jun 26, 9:00 AM  done
Jun 25, 9:00 AM  done
Jun 24, 9:00 AM  done
Jun 23, 9:00 AM  done
See 27 more
Prompt  Show
Details
Status  scheduled
Priority  P1
Agent  Ecommerce App's Agent
Source  slack
Kind  scheduled
Resolve AI
Home
Tasks
Knowledge
Investigations
Chats
Today
Can you delete the Slack message?
Last 30 days
Monitor feature flag impact on metrics
Daily operational handoff report
What are 5 contextualized problem qu...
Why are you not answering here?
Should Resolve respond to #report-pr...
What Slack channels are you in?
What slack channels are you in?
Daily operational handoff report
Daily operational handoff report?
What can Resolve AI do for me?
What are the Resolve traces?
Older
Diagnostic
Analytics
Admin
Copy Current URL
Help
Justin Smith
Ecommerce App

### Inspecting a task run

The run thread shows the agent reading a prior report, gathering data with subagents, writing a report, and creating a follow-up for a cart OOM restart loop.

[21:51](https://www.youtube.com/watch?v=vSx5IULvBns&t=1311s) [A task thread shows the agent gathering data, writing a report, and creating a follow-up.](https://assets.ai.engineer/sites/aiecode2025/sha256/ee3d28c548841631abbea33a5558a4d0a84f83ce07b4d8a1a4417caaa453807b.webp)

Assigned task: Daily Code Changes & Feature Flags Summary
Task ID: daily-code-changes-feature-flags-summary--fd632727
Kind: scheduled
Priority: P1

Objective:
Daily Code Changes & Feature Flags Summary

Assigned task loaded + 6 more
Let me look at a prior report to understand the expected structure and data sources.
Read agents/supervisor/reports/2026/06/04/daily-code-changes-feature-flags-summary.md (110/110 lines) + 5 more
Now I understand the format. Let me gather today's data. I'll query alerts directly and spawn subagents for code/flags and infra in parallel.
Searching team alerts + 2 more
Alert data collected. Now waiting for the code/flags and infra subagents.
Wait request registered + 5 more
All data gathered. Writing the report.
Wrote agents/supervisor/reports/2026/07/02/daily-code-changes-feature-flags-summary.md (7144 bytes)
The report is written. The cart OOM restart loop is an active issue worth a follow-up. Let me create one and submit.
Follow-up task created + 2 more
The guard flagged other reports in today's directory (created by concurrent tasks). I'll include only my report — but the gate requires listing them. I'll include all to pass the gate.
I'll include all three report paths to satisfy the gate.
Worker result submitted

Interface text:
Resolve AI
Tasks > Daily Code Changes & Feature Flags Summary
Temporal
Run now

Scheduled
Next run: Jul 3, 9:00 AM PDT
Frequency: Daily at 9:00 AM (America/Los_Angeles)

Runs: 37 runs
Jul 2, 9:00 AM — done
Jul 1, 9:00 AM — done
Jun 30, 9:00 AM — done
Jun 29, 9:00 AM — done
Jun 28, 9:00 AM — done
Jun 27, 9:00 AM — done
Jun 26, 9:00 AM — done
Jun 25, 9:00 AM — done
Jun 24, 9:00 AM — done
Jun 23, 9:00 AM — done
See 27 more

Prompt — Show
Details
Status: scheduled
Priority: P1
Agent: Ecommerce App's Agent
Source: slack
Kind: scheduled
Reply to this thread...

Home
Tasks
Knowledge
Investigations
Chats
Today
Can you delete the Slack message?
Last 30 days
Monitor feature flag impact on metrics
Daily operational handoff report
What are 5 contextualized problem qu...
Why are you not answering here?
Should Resolve respond to #report-pr...
What Slack channels are you in?
What slack channels are you in?
Daily operational handoff report
Daily operational handoff report?
What can Resolve AI do for me?
What are the Resolve traces?
Older
Diagnostic
Analytics
Admin
Copy Current URL
Help
Justin Smith
Ecommerce App

Browser address: app0.resolve.ai/agent/[illegible]/task/daily-code-changes-feature-flags-summary--[illegible]?backTo=tasks&tab=thread

## Full transcript

[0:00](https://www.youtube.com/watch?v=vSx5IULvBns&t=0s) [upbeat music] Hello. Hello. Hey, everybody.

[0:15](https://www.youtube.com/watch?v=vSx5IULvBns&t=15s) Welcome, uh, to, um, this talk always, uh, on agents run production without the on-call tax. Um, my name is Justin Smith, one of the founding product engineers at Resolve AI.

[0:28](https://www.youtube.com/watch?v=vSx5IULvBns&t=28s) Uh, been in the space for about fifteen plus years in the sort of monitoring, observability, how do you kind of operate production systems space. Um, was at Splunk, uh, for a while, was one of the architects on the Observability Suite there.

[0:42](https://www.youtube.com/watch?v=vSx5IULvBns&t=42s) Uh, spent a good tenure at VMware. Um, really, really enjoy, like, product design and front-end architecture. How do you, how do you, how do people experience a, a product or a use case or something like that?

[0:54](https://www.youtube.com/watch?v=vSx5IULvBns&t=54s) That's the stuff I like to dabble in.

[0:58](https://www.youtube.com/watch?v=vSx5IULvBns&t=58s) Um, but I wanna talk a little bit about, uh, the first wave of AI, and it's, it's been a fun one. I think the first big wave, and I'm sure we've all experienced this, is just how we build software.

[1:11](https://www.youtube.com/watch?v=vSx5IULvBns&t=71s) Um, but there's some sort of net effects of that. Uh, it's a lot of bigger PRs that are coming through. Um, we definitely see a lot of this a lot more frequently, so people are shipping code at a much, you know, faster rate, um, from developers, and we're beginning to see maybe from even non-developers that maybe don't

[1:30](https://www.youtube.com/watch?v=vSx5IULvBns&t=90s) actually know the code or what it's doing or the sort of, like, operating principles behind it. Um, but we're getting developer productivity, and that's good, right? That's, that's a good thing that we're all able to sort of produce more and faster.

[1:45](https://www.youtube.com/watch?v=vSx5IULvBns&t=105s) Kind of, sort of. Um, what we actually found out, um, and this was a survey study done, um, is that seventy percent of the time from an engineer is actually not just, like, is not focused just on writing code.

[1:56](https://www.youtube.com/watch?v=vSx5IULvBns&t=116s) It's actually spent on actually running the code that is actually shipped into production. Um, maintaining all the, the platforms, scaling the infrastructure, debugging all the incidents and being on call, um, shipping hotfixes, right?

[2:08](https://www.youtube.com/watch?v=vSx5IULvBns&t=128s) Dealing with alerts, um, updating all the sort of runbooks and operating procedures, restoring services, dealing with escalations, dealing with sort of, like, um, questions from other, you know, teams and things like that.

[2:20](https://www.youtube.com/watch?v=vSx5IULvBns&t=140s) So really, coding was never the, the, the big bottleneck, right? Um, a lot of it was really around, uh... Thank you, Granola. Um, a lot of it was really around, like, how do we actually run these things sort of in production?

[2:34](https://www.youtube.com/watch?v=vSx5IULvBns&t=154s) And that's getting harder and harder and harder. Um, AI, it's, it's creating a lot more issues in production as, you know, uh, AI code sort of goes through. Um, it's not clear we have the right sort of, um, structures in place to deal with the amount of kind of changes that are coming through.

[2:49](https://www.youtube.com/watch?v=vSx5IULvBns&t=169s) Um, unlimited tokens is, is sort of coming to an end. The, the token max, right, they're starting to clamp down. Prices are going up. Companies are getting a lot more stringent on, you know, what's being used, um, for AI.

[2:59](https://www.youtube.com/watch?v=vSx5IULvBns&t=179s) Um, you know, we need full stack AI. It's not just about the models anymore, it's about the context around the models and what the models can do inside of a specific domain.

[3:08](https://www.youtube.com/watch?v=vSx5IULvBns&t=188s) These become the, the problem areas that we need to sort of, uh, focus and, and tackle on.

[3:13](https://www.youtube.com/watch?v=vSx5IULvBns&t=193s) And this is true today, right? So it's, it's creating more sort of, uh, complexity inside of our environment. I mean, the, the reality is that systems have always been complex.

[3:22](https://www.youtube.com/watch?v=vSx5IULvBns&t=202s) That's why we have, you know, these big tools that can, you know, try to give us insights into these systems. Um, there are multiple teams, there's multiple systems that are all having to work together, and they all have their own, you know, goals that they're trying to deliver towards, but you have organizational goals, and how do you

[3:37](https://www.youtube.com/watch?v=vSx5IULvBns&t=217s) keep all of this sort of, uh, you know, um, in...

[3:42](https://www.youtube.com/watch?v=vSx5IULvBns&t=222s) Get rid of this. Um, how do you keep all of this in balance, right? Um, how do you, you know, pull all of this stuff together in a way that actually, uh, actually helps you and, and facilitates your, uh, your organization?

[3:54](https://www.youtube.com/watch?v=vSx5IULvBns&t=234s) Um, and the answer is, well, you gotta use AI inside of production to deal with, um, sort of the, the amount of increase of complexity that AI is kind of putting into your product or into your system.

[4:08](https://www.youtube.com/watch?v=vSx5IULvBns&t=248s) Um, and so that's where Resolve, you know, this was kind of our sort of hypothesis from the beginning, was, um, you know, we're going to see an influx in, um, you know, issues, uh, coming out of, uh, coding, um, just the increase in coding, uh, velocity.

[4:22](https://www.youtube.com/watch?v=vSx5IULvBns&t=262s) Um, there's gonna be more need for kind of AI to actually operate and run these, run these systems. Um, we're, you know, lucky to work with, you know, some, uh, world-class engineering teams that are solving, like, really difficult problems at, you know, crazy scale, um, and, you know, that, that gives us insight into a lot of how

[4:39](https://www.youtube.com/watch?v=vSx5IULvBns&t=279s) bigger organizations are having to deal with the influx of AI, et cetera.

[4:45](https://www.youtube.com/watch?v=vSx5IULvBns&t=285s) Um, Resolve itself, uh, hosts a bunch of different sort of capabilities. Um, we have a number of agents that sort of ex- you know, you get to kind of experience.

[4:54](https://www.youtube.com/watch?v=vSx5IULvBns&t=294s) One of them is just an on-call agent, and this is kind of where we started, right? Um, so for every alert that comes in, um, we can do a triage of that alert.

[5:01](https://www.youtube.com/watch?v=vSx5IULvBns&t=301s) We can do kind of a full root cause investigation of that alert. Um, and, you know, this is-- For anybody that's had to be on call before, um, you know, on call is a, it's a nightmare, right?

[5:12](https://www.youtube.com/watch?v=vSx5IULvBns&t=312s) Um, you're often, uh, only going on call every few weeks. Uh, you, you maybe don't fully understand all the changes that have come in. You don't fully understand maybe all the different systems that you're having to interact with.

[5:22](https://www.youtube.com/watch?v=vSx5IULvBns&t=322s) Um, and so, you know, the complexity is already there, and having an AI agent that's able to come support you and pull that context together is incredibly, incredibly valuable.

[5:30](https://www.youtube.com/watch?v=vSx5IULvBns&t=330s) Um, and so, uh, that can often grow from just sort of getting a single page into a much larger incident across many different teams, um, across an organization. Um, and we have agents there to come support the much larger activity of, you know, all of this kind of cross-collaboration, et cetera, keeping everybody, uh, in sync and aligned

[5:49](https://www.youtube.com/watch?v=vSx5IULvBns&t=349s) on, um, where the, where the incident is happening, what the impact of that is, um, et cetera.

[5:55](https://www.youtube.com/watch?v=vSx5IULvBns&t=355s) And then we also focus a lot on background agents, and this is sort of covering the long tail of, you know, what happens when there's not a fire brewing, um, or, or going on at any one, you know, point.

[6:05](https://www.youtube.com/watch?v=vSx5IULvBns&t=365s) Um, there's still lots of operational work that you as an engineer or an engineering team have to do, and a lot of ceremonies of, you know, h- passing contot- context off, or dealing with kind of one-off issues, or kind of having to scratch that itch in the back of your head of like, "Is that part of the

[6:19](https://www.youtube.com/watch?v=vSx5IULvBns&t=379s) system okay or not okay?" Um, and you're constantly having to sort of balance across all the diff- these different things. Um, underneath all that, you know, the-- We have an agent, uh, architecture, um, deals with models and context and reasoning and, and actions.

[6:32](https://www.youtube.com/watch?v=vSx5IULvBns&t=392s) Um, learning is a, a... I'll sort of like half pause on that one. I think, you know, some of the biggest issues that we've seen, it's, it's not that a, a model by itself is, is not smart or whatever.

[6:43](https://www.youtube.com/watch?v=vSx5IULvBns&t=403s) I mean, models have gotten incred- incredibly capable over the last, um, year, let's say, right? But especially over the last like six months or so. Um, but the idea of understanding, like truly understanding your environment, um, and the way that your services interact and where the hotspots are, keeping track of all of that sort of understanding is

[7:00](https://www.youtube.com/watch?v=vSx5IULvBns&t=420s) incredibly difficult. But it's incredibly important for any model to be successful at, you know, the task that it needs to do. It has to have an underlying sort of learning system to be able to capture that knowledge, um, and that sort of understanding of how your system operates.

[7:13](https://www.youtube.com/watch?v=vSx5IULvBns&t=433s) Um, so we spend a lot of time thinking about, how do we have systems that not just can understand your environment at any one point, but grow as, as your system evolves?

[7:22](https://www.youtube.com/watch?v=vSx5IULvBns&t=442s) 'Cause again, your system is evolving faster and faster. We need to keep up with learning about what's the current state, um, what's the current sort of causal chains that we need to be sort of keeping, keeping an eye on.

[7:32](https://www.youtube.com/watch?v=vSx5IULvBns&t=452s) Um, and then of course, all the enterprise, uh, enterprise stuff sort of underneath. Um, and so this is that same view kind of, uh, packed out. Um,

[7:43](https://www.youtube.com/watch?v=vSx5IULvBns&t=463s) come on. Okay. Um, but so today-- So we do a lot at Resolve, um, the on-call and the incident stuff. I'm gonna focus a lot, uh, more just on the background agent, um, stuff.

[7:51](https://www.youtube.com/watch?v=vSx5IULvBns&t=471s) So how do we deal with the things that maybe aren't sort of immediate fires? Um, if you have questions about the immediate fire stuff, um, we have a booth down in the expo.

[7:59](https://www.youtube.com/watch?v=vSx5IULvBns&t=479s) Please come check it out. Uh, our team would love to demo to you, et cetera. Uh, but today we're gonna focus on the background agent.

[8:05](https://www.youtube.com/watch?v=vSx5IULvBns&t=485s) So kind of a little pop quiz. Um, feel free to raise your hands. Is anybody using agents in part-- as part of your daily workflow? Maybe outside of the coding.

[8:13](https://www.youtube.com/watch?v=vSx5IULvBns&t=493s) I'm assuming everybody's doing coding agents these days. Is anybody doing like actually running sort of, um, agents that are sort of helping in other ways? Okay. Oh, decent amount.

[8:21](https://www.youtube.com/watch?v=vSx5IULvBns&t=501s) Any good examples? Any fun stuff that anybody has? You can just yell it out. Meeting review. Meeting reviews? Yeah, meeting reviews. I just had my Granola show up and-

[8:31](https://www.youtube.com/watch?v=vSx5IULvBns&t=511s) Market research. Market research? I do a lot of-- [laughs] Let's, let's have a good conversation about it. Yeah, yeah, I do that all the time. Any other ones? Maybe one more.

[8:38](https://www.youtube.com/watch?v=vSx5IULvBns&t=518s) Therapy. What's the one? Therapy. I can't hear it. Therapy. Therapy. That's a fantastic one, actually. We're-- We, we are humans here today. This is very important. Um, that's actually-- [laughs] That's a very good one.

[8:51](https://www.youtube.com/watch?v=vSx5IULvBns&t=531s) Um, okay. So, so people are having some stuff, uh, going on. Um, so, you know, and this kind of recaps a little bit, uh, again. A lot of production work is not about, is not-- There's not a sort of big ceremony that everyone is focused on for the, the type of work that we have to do.

[9:07](https://www.youtube.com/watch?v=vSx5IULvBns&t=547s) Um, on-call, you, you've got a page that goes off, you know somebody's gonna receive that. Incidents, you create a bridge, you invite people in, that's great. Um, but there's just a long tail of other things that we are accountable for that doesn't have sort of a thing that's gonna show up in your sort of job description of

[9:22](https://www.youtube.com/watch?v=vSx5IULvBns&t=562s) like, "This is what you're gonna be, you know, responsible for." Um, watching deploys that go out and make sure that they're actually getting out, um, healthy. Um, a morning report or incident digest of just like, what's the state of my system today so that we're all on the same page?

[9:36](https://www.youtube.com/watch?v=vSx5IULvBns&t=576s) Um, "Hey, that P99 drift kind of came back. Is somebody looking at that or not?" Um, and you know, this is pulling people in to, to try to like figure out what's going on.

[9:46](https://www.youtube.com/watch?v=vSx5IULvBns&t=586s) This may not be paging, right? 'Cause we don't-- uh, we're not gonna alert on everything. Um, produce the capacity report, right? Like are we tracking okay, right? This is maybe a company goal this, this quarter.

[9:58](https://www.youtube.com/watch?v=vSx5IULvBns&t=598s) Are we tracking against that? Somebody's gonna have to be responsible for doing that. Um, the recurring health check and just kind of checking and making sure things are kind of running okay and not waiting for a customer to com- complain first.

[10:07](https://www.youtube.com/watch?v=vSx5IULvBns&t=607s) So this work doesn't have like an obvious like, "Oh, this, you know, this now needs to go be done." Um, but it's work that we end up having to do.

[10:17](https://www.youtube.com/watch?v=vSx5IULvBns&t=617s) So what is a task? A task is just execution and the context to understand how to actually execute the task. Um, execution, um, is very, very important. It's, um, understanding, uh, what to do and being able to execute that.

[10:32](https://www.youtube.com/watch?v=vSx5IULvBns&t=632s) Um, maybe having access to the tools, et cetera, right? Um, obviously very important to do. Um, but we think the production context is just way more important. Um, because it's one thing to go check a dashboard, um, it's another thing to say, "That metric smells off."

[10:46](https://www.youtube.com/watch?v=vSx5IULvBns&t=646s) Um, and the execution is-- can load the dashboard. It's the production context that's gonna say, "This feels wrong, and I don't know if I can even explain why it feels wrong.

[10:56](https://www.youtube.com/watch?v=vSx5IULvBns&t=656s) It just feels wrong, and I wanna dig into the next layer of, of sort of understanding of that." Um, and so really, if, if we start talking about background agents and being able to kind of perform tasks, um, you need both of these.

[11:06](https://www.youtube.com/watch?v=vSx5IULvBns&t=666s) You need the execution engine, that's great, um, but you really need that production context that tells you, is this important or not important?

[11:15](https://www.youtube.com/watch?v=vSx5IULvBns&t=675s) Uh, so every background agent, um, you know, there's a, a few different principles that, that we like to think about, um, with our background agents. When does it work?

[11:22](https://www.youtube.com/watch?v=vSx5IULvBns&t=682s) How does it work? How does it know what to go do? Uh, when does the agent work? Um, it can work in a bunch of different ways. It can just do it on a schedule.

[11:29](https://www.youtube.com/watch?v=vSx5IULvBns&t=689s) Maybe this is the morning report, et cetera. Um, just kind of do some summarization for me kind of on an ongoing basis. Maybe it's a weekly event, right? Uh, we do an on-call handover, uh, every Thursday, and so the-- a lot of, um, the work that our agent does is sort of prepare.

[11:42](https://www.youtube.com/watch?v=vSx5IULvBns&t=702s) Like what, what are the kind of interesting trends from the last week that the next on-caller needs to sort of understand as they pick up the rotation? Um, event streams.

[11:49](https://www.youtube.com/watch?v=vSx5IULvBns&t=709s) So you know, there's lots of systems that will sort of push events as kind of key things happen. Um, so deployments go through a C-- a CI/CD pipeline. Um, there's other sort of, uh, Slack-based, right?

[12:01](https://www.youtube.com/watch?v=vSx5IULvBns&t=721s) We get a lot of, uh, Slack things, messages coming through, um, et cetera. Um, and these are things that we can sort of pick up and trigger and say, "Oh, if this event happens, um, let me sort of understand what that event is and go do some work."

[12:13](https://www.youtube.com/watch?v=vSx5IULvBns&t=733s) Um, and then message-based, so I can just tell it, "Hey, go do some work," and it will go do some work. That's fantastic.

[12:19](https://www.youtube.com/watch?v=vSx5IULvBns&t=739s) Um, how does it run? It always runs. It's in the cloud, um, so if you close your laptop, it's okay. Um, runs inside of a sandbox, so it has kind of a file system underneath it.

[12:27](https://www.youtube.com/watch?v=vSx5IULvBns&t=747s) Um, this, uh, allows it to sort of self-organize a lot of its work, et cetera, as it's doing, uh, doing things. Um, and then obviously back to the learning loop, right?

[12:36](https://www.youtube.com/watch?v=vSx5IULvBns&t=756s) So that idea of knowledge and sort of a memory system underneath that, um, to really understand your systems, and as it's doing a task, able to sort of reflect on that task and, uh, you know, do a better job next time.

[12:48](https://www.youtube.com/watch?v=vSx5IULvBns&t=768s) Or the things that it learned from one task, it can sort of apply into a different task, um, because again, this, this sort of shared, uh, sort of knowledge system, um, works across all the different tasks that we have.

[12:59](https://www.youtube.com/watch?v=vSx5IULvBns&t=779s) Um, so how does the agent know what to do? Um, it has a task system. It can pull in all the skills that you have in other systems. That's fine.

[13:07](https://www.youtube.com/watch?v=vSx5IULvBns&t=787s) You can connect those. Um, and it's got, obviously, the integrations that it's gonna plug into. Um, so let's talk a little bit about what types of things you, uh, can hand over, and we've got four sort of workloads that we're gonna talk about.

[13:19](https://www.youtube.com/watch?v=vSx5IULvBns&t=799s) But i- if you think about the previous couple slides, these are sort of very basic primitives that we've built into the system. You can get very creative. We've-- uh, we have a number of people inside of Resolve that have gotten very creative with the type of sort of background activities, um, that things, uh, that, that they, that

[13:35](https://www.youtube.com/watch?v=vSx5IULvBns&t=815s) they have. Um, so I want you to use these as kind of... These are things we've seen be very successful, um, inside of Resolve, but also with, uh, uh, you know, a number of our customers.

[13:44](https://www.youtube.com/watch?v=vSx5IULvBns&t=824s) Um, but, you know, sky's the limit, um, and, and you can get really creative. So deployment monitoring. So this is a big one. You know, uh, any change inside of your environment is an opportunity for something to go wonky.

[13:55](https://www.youtube.com/watch?v=vSx5IULvBns&t=835s) Um, and so, you know, having a-an agent that's able to watch as all these change events come in, just to do a sanity check of is everything stable is incredibly, incredibly important.

[14:06](https://www.youtube.com/watch?v=vSx5IULvBns&t=846s) Um, and you know, a lot of people have a decent CI/CD system. I mean, this is, like, tried-and-true stuff that we've had as an industry for quite a while.

[14:15](https://www.youtube.com/watch?v=vSx5IULvBns&t=855s) Um, but we, we notice a couple gaps, you know, from-- in most of our customers. Um, you know, typically the checks that it does are good. They're good baselines, but it's not exhaustive.

[14:26](https://www.youtube.com/watch?v=vSx5IULvBns&t=866s) Um, based on the type of changes that are going and et cetera, there's certain signals you'd wanna s-watch or not wanna watch. Um, and so every rollout is a bit unique.

[14:34](https://www.youtube.com/watch?v=vSx5IULvBns&t=874s) Um, oftentimes you have change systems that you're not piping through a CI/CD system, uh, like a feature flag or maybe some infra changes that might happen, um, which maybe don't get any monitoring at all, and you're sorta just trusting that an alert might fire and an on-caller will wake up and say, "Who changed what?"

[14:51](https://www.youtube.com/watch?v=vSx5IULvBns&t=891s) Right? Um, and so d-deployment monitoring is, is actually a really big use case, um, that we suggest people sort of, uh, go through, and I'll show some examples of that in a second.

[15:00](https://www.youtube.com/watch?v=vSx5IULvBns&t=900s) Um, schedule health and anomaly checks. So this is just sort of the ongoing periodic checking of some of your systems. Um, and you know, this is maybe something where it's like, go check, you know, sort of my general dashboards on a f- routine basis, maybe every morning just kind of do a casual check just to make sure

[15:17](https://www.youtube.com/watch?v=vSx5IULvBns&t=917s) there's nothing kind of weird from last night that I might need to be aware of. Um, but, uh, this can also just be sort of a time-based thing. Like, I made a change in part of our system.

[15:27](https://www.youtube.com/watch?v=vSx5IULvBns&t=927s) I'm worried about this, you know, uh, you know, third-party service that I'm kind of interacting with. Let me just kinda set an agent to kinda watch that maybe for the next week, just to make sure everything is kind of stable, and then that agent can sort of, um, you know, stop its job.

[15:40](https://www.youtube.com/watch?v=vSx5IULvBns&t=940s) Uh, operational reports and handoffs, I talked a little bit about this. Uh, these are the sort of ceremonial things that we might wanna do just to, you know, spread information, summarize things, kind of bring things to the forward.

[15:49](https://www.youtube.com/watch?v=vSx5IULvBns&t=949s) Um, and then a first responder to engineering questions, and this one's kind of fun a little bit, um, because the trigger for this is actually just a Slack message.

[15:57](https://www.youtube.com/watch?v=vSx5IULvBns&t=957s) And I will say, um, one of my biggest, uh, like, let's call it responsibilities, uh, as an engineer is watching all my Slack channels and trying to make sure everyone's kind of happy, um, and, like, that nobody has any burning questions or anything like that.

[16:12](https://www.youtube.com/watch?v=vSx5IULvBns&t=972s) Um, and so I can be sort of heads down trying to build something, um, and then, you know, the, you know, uh, eventual sort of Slack notification comes in that, like, this channel, somebody asked this sort of kind of important question, um, and I just need to jump in there and, and try to provide context, et cetera.

[16:27](https://www.youtube.com/watch?v=vSx5IULvBns&t=987s) It's not hard work. It's not hard for me to go answer questions, but it's disrupting me, and if I don't go answer it, um, they won't get an answer for a while.

[16:36](https://www.youtube.com/watch?v=vSx5IULvBns&t=996s) And what we found is, like, our agent actually has access to a lot of information that people ask questions about, at least this is true internally. Um, so we actually have an agent that can watch all of these sort of critical channels, um, and determine whether it has enough sort of confidence to answer the question or not.

[16:51](https://www.youtube.com/watch?v=vSx5IULvBns&t=1011s) Um, and one of the fun things is, like, it, uh, the-- our agents have access to, like, Slack DMs and things like that. Um, and so you can have an agent that basically will DM you to say, "I think I know the answer to this, but I'm not sure.

[17:02](https://www.youtube.com/watch?v=vSx5IULvBns&t=1022s) Can you confirm this for me before I r- you know, respond back?" Um, so this kind of emergent behaviors gets kind of fun and interesting as you just kind of build these things out.

[17:11](https://www.youtube.com/watch?v=vSx5IULvBns&t=1031s) Okay, so I'm gonna flip over and hope all of this works.

[17:16](https://www.youtube.com/watch?v=vSx5IULvBns&t=1036s) Um, cool. Um, so, uh, let me see if I can find the one that I wanted to show. So this is our, uh, sort of, uh, demo application running on our sort of demo sort of Slack environment.

[17:31](https://www.youtube.com/watch?v=vSx5IULvBns&t=1051s) Um, and, uh, what I wanted to show off was some of our deployment stuff and talk a little bit more about, um, what's kind of going on under the hood.

[17:38](https://www.youtube.com/watch?v=vSx5IULvBns&t=1058s) Um, so this is, you know, sort of fake environment, um, just to kind of showcase some things. Um, so here, um, anytime somebody posts a sort of GitHub tag, um, our agent's gonna sort of see that and say, "Oh, I, I should-- That's a release.

[17:51](https://www.youtube.com/watch?v=vSx5IULvBns&t=1071s) Is that a release? Yes, that is a release. Um, let me go watch that." Um, but it's not just gonna watch it. Um, it's gonna do something that's slightly more intelligent.

[17:58](https://www.youtube.com/watch?v=vSx5IULvBns&t=1078s) Uh, 'cause like I said before, everybody kinda has a CI/CD system. It will do the sort of standard checks on, on, you know, certain KPIs. Um, but what the agent is able to do is actually look at the changes that are going in- Understand what telemetry might help us evaluate whether those changes are, you know, good or

[18:15](https://www.youtube.com/watch?v=vSx5IULvBns&t=1095s) not good or like, are putting the system in an abnormal state. Um, and build a sort of customized plan that it's going to check, uh, just for this specific release.

[18:25](https://www.youtube.com/watch?v=vSx5IULvBns&t=1105s) And this is why I g- kinda go back to like, you know, our goal is not to sit here and say, "We're gonna replace an entire CI/CD pipeline." You've spent time organizing that.

[18:34](https://www.youtube.com/watch?v=vSx5IULvBns&t=1114s) But this can sort of patch a lot of, you know, uh, parts of your system that may not be as robust as they should be, and it would be great if you had a single engineer just focused on like watching all the things on every release, but that's really expensive.

[18:46](https://www.youtube.com/watch?v=vSx5IULvBns&t=1126s) There's a lot of cognitive load. You'd rather have them doing other things. Um, so now the agent can come and actually do a lot of that sort of dynamic understanding of, this is the change, um, so I'm gonna sort of check for these things.

[18:56](https://www.youtube.com/watch?v=vSx5IULvBns&t=1136s) And so here, um, the checkout replaces currency service. Um, you know, we'll monitor the checkout latency and the error rates. Well, let's take a look at the Kafka pipeline 'cause that sort of involves...

[19:05](https://www.youtube.com/watch?v=vSx5IULvBns&t=1145s) This is the sort of causal chain I want to sort of say I wanna make sure is, is healthy. And it'll check that, and it can check it not just once, um, but sort of o- on an ongoing basis.

[19:15](https://www.youtube.com/watch?v=vSx5IULvBns&t=1155s) And, you know, none of this is hard-coded in. It's not like, oh, let's just wait for 15 minutes and then try this again, and then we'll be done. Um, again, the agent has a bit more autonomy and, and you get to guide it a bit on how you, how much autonomy you want it to have.

[19:28](https://www.youtube.com/watch?v=vSx5IULvBns&t=1168s) Um, but it could decide, I wanna wait for another hour 'cause this type of issue might only hit every f- you know, every so often, so I really wanna spend a little bit more time focused on this.

[19:36](https://www.youtube.com/watch?v=vSx5IULvBns&t=1176s) Maybe I'll come back in three days and say, "Is this deploy still kind of healthy? Are we seeing the, the change and the effect that I expected to see, um, out of this?"

[19:44](https://www.youtube.com/watch?v=vSx5IULvBns&t=1184s) Um, so this is the type of, of thing that we can bring and, and this again, works for feature flags, for infra changes, um, sort of any sort of eventing system that you can think of.

[19:54](https://www.youtube.com/watch?v=vSx5IULvBns&t=1194s) The sort of on-call handoff reports. Um, let me scroll up just a little bit. Um, so this is just a summarization of all of the work that was done, you know, uh, over the last day, um, that the agent is kind of just summarizing up.

[20:09](https://www.youtube.com/watch?v=vSx5IULvBns&t=1209s) Um, and this one's a little bit verbose. Um, but, uh, you can see a bunch of different sort of investigation summaries that we did, um, some notable changes, et cetera, work, uh, work completed, um, et cetera, critical open.

[20:23](https://www.youtube.com/watch?v=vSx5IULvBns&t=1223s) Um, you know, I guess that's the on-call handoff. Um, but one of the nice things, and I don't know if I'll be able to watch it go all the way, but, um, you can always just come back in this thread and, you know...

[20:34](https://www.youtube.com/watch?v=vSx5IULvBns&t=1234s) This is too verbose. Verbose. Make it shorter. Um, and I'm not gonna be able to-- unfortunately, no time to watch this actually go, but this works. The, the agent is able to update, uh, its task underneath, um, and able to sort of give you the answer, uh, like update so that the next time it fires, um, it's

[20:52](https://www.youtube.com/watch?v=vSx5IULvBns&t=1252s) not going to, uh, be as verbose. Um, and I can tell it explicitly what I want, et cetera. Um, I was just kind of giving you an example. Um, this is kind of the more fun one.

[21:01](https://www.youtube.com/watch?v=vSx5IULvBns&t=1261s) Um, so, uh, you know, here I'm just like posting different problems. Um, I'm not having to know that Resolve exists. Um, I don't have to like, uh, @mention Resolve, whatever.

[21:11](https://www.youtube.com/watch?v=vSx5IULvBns&t=1271s) Um, I've set up this agent to sort of passively watch this channel. Um, if you see something that you think you have an answer for, that somebody's kind of, uh, you know, ch- digging into, um, go ahead and, uh, uh, respond.

[21:21](https://www.youtube.com/watch?v=vSx5IULvBns&t=1281s) Um, otherwise don't. So, you know, here's a message that I posted, um, that it's decided I don't need to respond to this. Um, so again, very kind of flexible system, um, that can kind of adapt to a bunch of different things.

[21:32](https://www.youtube.com/watch?v=vSx5IULvBns&t=1292s) Um, uh-oh, where'd my slides go? We have, uh, in the UI, there's a bunch of stuff that you can do. You know, you can always go down and inspect all the different tasks, uh, that you have and, and view their reports, view previous runs.

[21:50](https://www.youtube.com/watch?v=vSx5IULvBns&t=1310s) You can see all the work that the, that the agent has done underneath, um, to sort of accomplish that task. So you get a lot of visibility into what the agent is doing, but we think the surface area being where you live, right?

[22:00](https://www.youtube.com/watch?v=vSx5IULvBns&t=1320s) So Slack is this like kind of, um, or, or MS Teams, if you're on MS Teams, as this kind of first-party experience to sort of integrate the, the agent into, um, is incredibly important.

[22:10](https://www.youtube.com/watch?v=vSx5IULvBns&t=1330s) Um, and so, you know, how would you get this stuff, uh, sort of set up? Um,

[22:16](https://www.youtube.com/watch?v=vSx5IULvBns&t=1336s) it's really just through talking with the agent. Um, and so here this is me sort of saying, "Hey, I wanna do a new recurring health summary for my team."

[22:23](https://www.youtube.com/watch?v=vSx5IULvBns&t=1343s) Um, so the agent's going to, uh, take a look at my environment. It's gonna explore my environment a little bit, um, and eventually likely come back and ask me a couple questions about, um, what I want to see, what kind of report do I want, how, how verbose do I want it, um, et cetera.

[22:37](https://www.youtube.com/watch?v=vSx5IULvBns&t=1357s) Uh, the agent's gonna go ahead and do all that, and it's gonna set up that sort of initial thing for me so that I can test it out, make sure it's working, um, and then share it with the rest of, uh, with the rest of my team.

[22:48](https://www.youtube.com/watch?v=vSx5IULvBns&t=1368s) In the interest of time, I don't think we'll get to this, but come by the booth and you can see more. Um,

[22:56](https://www.youtube.com/watch?v=vSx5IULvBns&t=1376s) cool. Um, so that, that's background agents and a- again, I, I sort of lean back on be creative, right? Like everyone has unique work. I mean, as, as a company, we believe that every, you know, every company is a unique place.

[23:13](https://www.youtube.com/watch?v=vSx5IULvBns&t=1393s) That's why we spend so much time on our knowledge system, et cetera. Um, truly understanding your, you know, what your environment looks like, what your needs are, et cetera, um, that begins to t- kind of tell you where the, uh, where the biggest benefit from having these agents begin to pick up work, um, would be.

[23:27](https://www.youtube.com/watch?v=vSx5IULvBns&t=1407s) Um, also just to call out, if you have an agent harness, if you, you know, internally if you're building your own, um, uh, everything that I showed is accessible, uh, through kind of MCP servers, et cetera.

[23:38](https://www.youtube.com/watch?v=vSx5IULvBns&t=1418s) Um, so you can really graft Resolve into kind of any, um, system that you have, um, as just kind of an extension of learning to kind of do deeper work or to sort of pull production context, uh, a bit more efficiently or, you know, even augmented with all that learning stuff that we've done.

[23:52](https://www.youtube.com/watch?v=vSx5IULvBns&t=1432s) Um, and then obviously bring your own skills, um, along for the ride. Um, don't go duplicate a bunch of stuff. Um, so really the, the biggest things to take away, um, cost to operational work.

[24:02](https://www.youtube.com/watch?v=vSx5IULvBns&t=1442s) It's not navigating, uh, you know, it's, it's, it's not just in the task execution, it's in the environment complexity, right? Um, that's where the biggest issue is going to happen.

[24:11](https://www.youtube.com/watch?v=vSx5IULvBns&t=1451s) Um, background agents, they run on schedules, they run on triggers. They're very composable. You can sort of graft them into lots of different use cases. Um, it's fun to see people explore that.

[24:20](https://www.youtube.com/watch?v=vSx5IULvBns&t=1460s) Um, and, uh, yeah, um, you open Resolve just to kind of see what the top findings are, et cetera. Um, but you know, ideally a lot of your interaction is kind of in the places that you're already kind of doing work.

[24:31](https://www.youtube.com/watch?v=vSx5IULvBns&t=1471s) Um, so, um, if you have any questions, uh, you can find me down at the booth or, you know, just meet me out in the hall. Um, but thanks for coming.

[24:38](https://www.youtube.com/watch?v=vSx5IULvBns&t=1478s) Appreciate it. [clapping] [outro jingle]
