# The Devops Engineer Who Never Sleeps

Diamond Bishop, Datadog

AI Engineer Summit 2025

[Talk page](https://ai.engineer/talks/VZzUhELgYk4-devops-engineer-who-never-sleeps) · [Video](https://www.youtube.com/watch?v=VZzUhELgYk4)

## The DevOps Engineer Who Never Sleeps

Datadog’s agent workflows show how incident investigation, code repair, evaluation and observability fit together—and where human verification still matters.

Familiarity with alerts, runbooks, logs, metrics, traces and pull requests will help you follow the workflows.

### From observing a system to operating it

[0:43](https://www.youtube.com/watch?v=VZzUhELgYk4&t=43s)

What changes when an observability platform can investigate the problems it exposes? Diamond Bishop approaches that question through Bits AI, Datadog’s assistant for DevOps work. His background spans Cortana at Microsoft, Alexa at Amazon, PyTorch at Meta and a startup building a DevOps assistant. The underlying job at Datadog connects two activities: understanding what is happening in cloud applications and taking action to make those systems safer and easier to operate.

![Slide showing data sources feeding the Datadog platform and products, with arrows to Dev, IT Ops, Security, Support, and Business users.](https://ai.engineer/api/talk-essay-media/38aaaaeb17b42a2d9ce255203e8e1388da1258bd49f15b17d7b93040787d86fa)

Datadog’s observability and security platform connects data sources, products, and users.

AI already supports parts of that job: proactive alerting, root-cause analysis, impact analysis and change tracking. Bishop dates Datadog’s broader AI work to around 2015; that is a recollection of the program’s history, not a common release date for every capability. Stronger foundation models, emerging reasoning and multimodal capabilities, and rising expectations from products such as Cursor create an opportunity to move further up the stack.

**The next product layer is an agent that uses the platform on the customer’s behalf.** Bishop compares the potential shift to the microprocessor and SaaS eras, while acknowledging that its direction remains uncertain. Turning that opportunity into a product requires three kinds of work together: developing agents, evaluating their behavior and building observability for the workflows they execute.

### An investigation starts with the alert

[3:21](https://www.youtube.com/watch?v=VZzUhELgYk4&t=201s)

At the time of the talk, Datadog had two agents in private beta. The **AI Software Engineer** examines errors and recommends generated code changes. The **AI On-Call Engineer** investigates operational problems, aiming to reduce the occasions when an engineer must wake up and start from scratch. These are the historical product names used in the demonstration.

The concrete problem is the 2 a.m. alert. Normally, someone wakes up, opens a runbook and begins figuring out what happened. The On-Call Engineer starts proactively when the alert occurs, following the same broad investigative sequence:

1. **Orient to the incident.** Read the alert context and relevant runbooks to establish the problem.
2. **Gather service knowledge.** Collect the context needed to interpret the alert.
3. **Investigate iteratively.** Query logs, metrics and traces, using what turns up to guide the next step.

The goal is to have useful investigative work underway before a human reaches a computer.

![The 2am Alert slide pairs a workflow—build problem statement, execute runbook, gather service knowledge, and investigation—with a Datadog alert screenshot and line chart.](https://ai.engineer/api/talk-essay-media/240e4469f365415b22cd3e6e2fd09ee4ee2d615c1682fdf667c32c9f2b0774d8)

The AI Oncall Engineer workflow begins with an alert and surfaces investigation results to a user.

That work can support both automatic investigations and an engineer seeking a specific explanation. The agent gathers findings and summaries about why an alert fired or why a trace shows an error. Reducing overnight paging is the intended benefit; starting an investigation does not itself establish that the incident has been resolved.

### Make the investigation inspectable

[4:46](https://www.youtube.com/watch?v=VZzUhELgYk4&t=286s)

An agent working independently still needs a place where people can inspect its work. Datadog’s collaboration page exposes why a hypothesis was generated and what evidence the agent found. An engineer can follow the investigation and decide whether to agree with its conclusions. **Trust depends on being able to verify the work**, rather than merely receiving a confident summary.

The page also shows which runbook steps the agent actually took. A human can ask follow-up questions about why it chose a particular action, much as they would supervise a junior engineer. That analogy describes the review relationship: the agent does work, makes it inspectable and remains open to questions.

### From hypotheses to remediation and a postmortem

[5:30](https://www.youtube.com/watch?v=VZzUhELgYk4&t=330s)

Underneath that interface is a hypothesis-testing loop. The agent proposes explanations, reasons about how to test them and uses tools to query logs, metrics and other evidence. Bishop describes this as tool use in the [Toolformer](https://arxiv.org/abs/2302.04761) sense; the reference is an analogy for using tools, not a claim that Datadog implements the paper’s training method. Query results help the agent validate or invalidate each candidate explanation.

When the evidence supports a root cause, the agent can suggest a response: page another team, or offer to scale infrastructure up or down. Bishop distinguishes those suggestions from future plans to expand the available actions and discover workflows from a team’s behavior. Existing Datadog workflows already provide an integration point: the agent can understand a configured workflow and map it to a remediation need. A suggested response is not evidence that the change has been executed.

Incident work continues after remediation. Engineers reconstruct what happened, write a postmortem and share what they learned. The agent can assemble that account from the incident history, including both its own actions and the actions humans took, leaving a postmortem ready for morning review. This extends the workflow from immediate investigation into the team’s longer-term learning process.

### Turn observed errors into proposed code repairs

[6:54](https://www.youtube.com/watch?v=VZzUhELgYk4&t=414s)

The AI Software Engineer addresses the preventive side of operations. It watches errors coming through error tracking, analyzes their causes and proposes solutions, including generated code fixes. The two agents can therefore complement each other: one investigates alerts, while the other works on defects that could otherwise produce future incidents.

In the demonstration, the assistant identifies a recursion issue and proposes a fix. It also generates a regression test intended to catch the problem if it returns. The engineer can then create a GitHub pull request or open the diff in VS Code for editing. The handoff is a proposed change and test that an engineer can review; the presentation does not establish that the patch was merged or deployed.

Bishop reports that this workflow reduces the time engineers spend manually writing and testing code, but provides no measured reduction or comparison baseline. The concrete contribution shown is the package of diagnosis, proposed repair, regression test and an entry point into the engineer’s existing development workflow.

### Define work that people can evaluate

[7:46](https://www.youtube.com/watch?v=VZzUhELgYk4&t=466s)

Building these agents involved revisiting early decisions. Bishop groups the lessons into task scoping and evaluation, team composition, changing UX, and observability. The first difficulty is the gap between a convincing demo and a task whose performance can be evaluated repeatedly.

Start with the human job: what would someone do, step by step, and how would another person judge whether they did it well? That framing leads Datadog toward **vertical, task-specific agents** rather than a generalized agent with an open-ended remit. Make each step measurable and verifiable wherever possible, so improvement has an observable meaning.

Domain experts belong in that process as design partners and task verifiers. They help define useful work and recognize a sound result. Bishop cautions against treating their expertise as a set of rules to transcribe directly into the agent: the way an expert reasons about a domain does not necessarily map cleanly onto how a stochastic model behaves.

![Scoping slide lists vertical agents, jobs to be done, measurable and verifiable steps, and known escape hatches alongside domain expert roles.](https://ai.engineer/api/talk-essay-media/277dc10dbc5476f685af0032a36f0ceca2702899857f99bd69d527a7f5c0321d)

Scope task-specific agents around measurable work, with domain experts as design partners and task verifiers.

**Evaluation should precede the impressive demo.** Bishop attributes avoidable mistakes to not thinking about it early enough. Even a small initial evaluation is useful, provided it becomes part of a continuing system:

- **Offline evaluation:** assess the agent against defined tasks outside live use.
- **Online evaluation:** instrument product use and gather human feedback.
- **Living evaluation:** keep updating the test set as real use exposes new cases.

Include end-to-end tasks and measurements as well as checks on individual steps. A sequence of plausible intermediate actions is not enough to establish that the whole job succeeded.

### Build a team that can experiment and design collaboration

[10:12](https://www.youtube.com/watch?v=VZzUhELgYk4&t=612s)

Bishop’s suggested team starts with one or two ML experts and adds strong coding generalists who are optimistic about experimentation. A whole organization of specialists is neither necessary nor easy to hire. Front-end and UX skills matter because the product’s usefulness depends on how people collaborate with its agents, not only on the quality of the backend reasoning.

He also looks for people who use AI in their own daily work and want to explore new capabilities. In a field changing this quickly, willingness to learn and try things is part of the engineering requirement. A team reluctant to adapt can become stuck even while the available models improve.

The interaction design remains unsettled. Bishop describes UX as an area his team continues to revisit and says he currently favors agents that behave more like human teammates over adding more pages and buttons. That is a design preference, not a universal result: the underlying requirement is to make collaboration useful while accepting that established interface patterns may change.

### Observe the workflow, not just the model calls

[11:35](https://www.youtube.com/watch?v=VZzUhELgYk4&t=695s)

An agent investigating someone else’s system is itself a system that needs debugging. Observability supplies the situational awareness required to understand these complex workflows, so it cannot be an afterthought. Datadog’s LLM Observability brings model interactions into the wider view of the application and its infrastructure, including GPU monitoring. Calls to models the team hosts or operates and models accessed through APIs can be examined together.

Collecting those calls does not automatically make the execution understandable. Bishop says Datadog’s agent can make hundreds of calls in a workflow, with repeated loops and decisions about tools. That is an account of their agent’s possible workload, not a benchmark. A flat list makes it difficult to see how those decisions connect.

The talk then previews an **agent graph** that exposes the workflow’s structure. In the displayed graph, a bright red node draws attention to an error; zooming into it reveals where the failure occurred. The useful change is the unit of inspection: instead of reading an undifferentiated sequence of calls, an engineer can navigate the execution as a connected workflow. Bishop presents this view as a sneak peek.

### Keep improved models easy to try

[13:15](https://www.youtube.com/watch?v=VZzUhELgYk4&t=795s)

Bishop calls his next lesson an application-layer adaptation of [The Bitter Lesson](https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf): general approaches that can adopt better off-the-shelf models may outlast substantial task-specific investment. A team can spend time fine-tuning or constructing specialized behavior, only for a new foundation model to handle much of that reasoning directly. This is his engineering lesson, rather than a measured comparison supplied in the talk.

The practical requirement is to make alternative models easy to try and avoid becoming attached to the one around which the application was first built. Bishop does not claim new models already solve everything. The point is to keep the application able to benefit when capabilities improve, instead of making earlier investments a barrier to testing the next model.

### Build for agents that use the platform

[13:59](https://www.youtube.com/watch?v=VZzUhELgYk4&t=839s)

Building first-party agents raises a second product question: what should the platform provide to agents built by other teams? Bishop forecasts that agents could surpass humans as users of SaaS products in roughly five years, while acknowledging estimates ranging from one to ten years. This is a forecast, not an observed adoption result. Its design implication is to consider external agents alongside human users and the company’s own assistants.

His example is **Claude** using Datadog directly through MCP. Bishop reports setting this up relatively quickly, without presenting the implementation details. The platform-design work is to supply the operational context and API information an incoming agent needs to use the product effectively. An interface designed for a human browsing the application does not by itself answer those needs.

![Agents as Users: Beyond Building slide says to build for agents as customers and includes a dark interface screenshot captioned as an example of Claude using Datadog directly.](https://ai.engineer/api/talk-essay-media/e607388fbb8a8c384bb62d59795131069f2ee598219c9226580ae9950be999e5)

Build for agents as customers, including third-party agents such as Claude.

### A team of DevSecOps agents

[14:44](https://www.youtube.com/watch?v=VZzUhELgYk4&t=884s)

Bishop’s closing vision is a team of DevSecOps agents available for hire. Those agents would handle platform interaction, integration and on-call work that customers otherwise perform themselves. At the same time, external SRE and coding agents would become platform customers, using its tools much as a human engineer would. These are complementary directions: provide agents that do the work, and provide a platform other agents can operate.

For a small company, the imagined workflow combines automated development through products such as Cursor or Devin with agents handling operations and security. Bishop’s aspiration is that this could let an order of magnitude more ideas reach the real world; it is not a measured deployment outcome. He closes by inviting both builders of agents that could use Datadog and prospective users of Datadog’s own agents to participate in that work.

## Resources

- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://arxiv.org/abs/2302.04761) — Original research on teaching language models when and how to call external tools.
- [The Bitter Lesson](https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf) — Rich Sutton's essay on general methods that benefit from increasing computation.
- [Bits Investigation: AI-assisted incident investigation](https://www.datadoghq.com/blog/bits-ai-sre/) — Datadog's later account of its AI on-call teammate and incident collaboration workflow.
- [From Datadog findings to code fixes with Bits Code](https://www.datadoghq.com/blog/bits-code/) — Explains how production telemetry feeds code changes, tests and pull requests for engineer review.
- [Agent monitoring graphs and LLM experiments](https://www.datadoghq.com/about/latest-news/press-releases/datadog-expands-llm-observability-with-new-capabilities-to-monitor-agentic-ai-accelerate-development-and-improve-model-performance/) — June 2025 announcement of interactive agent execution graphs and dataset-based LLM experiments.

## Slides

### Agenda

The talk moves from Datadog’s AI history to current agents, examples and observability, then covers lessons learned and what lies ahead.

[1:13](https://www.youtube.com/watch?v=VZzUhELgYk4&t=73s) [Agenda covering Datadog AI history, agents, examples, observability, lessons, and the future.](https://assets.ai.engineer/sites/aiecode2025/sha256/9e4896eb36bc835841400b11e690268793bd42c2aa2b3f229f8ea2b967491e21.webp)

Agenda

• AI/ML @ Datadog [History]
• AI Agents [Current]
• In-use Examples
• Agent Observability
• Things we’ve learned
• Looking Ahead [Future]

“The dawn of the Intelligence Age is a momentous development”
-sama

DATADOG 3

### Datadog platform overview

Datadog brings data from 750+ sources into a platform of monitoring and security products, creating a single source of truth for development, IT operations, security, support, and business teams.

[1:21](https://www.youtube.com/watch?v=VZzUhELgYk4&t=81s) [Diagram connecting data sources, platform capabilities, products, and user groups.](https://assets.ai.engineer/sites/aiecode2025/sha256/34e9a89cdf1d3e089deb5dddd5447d46d56d68bac1b08d1695fdca758a4c4a36.webp)

Datadog: The observability and security platform for cloud applications

MULTIPLE DATA SOURCES
750+ Sources
Logs
Traces
Metrics
Activity
Metadata
Sessions
Threats
Vulnerabilities

PLATFORM
CUSTOMER EXPERIENCE & USABILITY
Dashboards
Mobile
Alerts
Service Catalog
Watchdog AI
Bits AI

INTEGRATIONS & ECOSYSTEMS
Unified Agent
Integrations
OpenTelemetry
IDE Plug-ins
Workflow Automation
AI Stack Integrations

DATA & SERVICES
Metrics Platform
Event Platform
Referential Platform
Unified Query
Back-end Architecture
Identity/Teams
Unified Tagging
RBAC
AI Platform

PRODUCTS
LLM Observability
Cloud Security
Software Delivery
Cloud Service Management
Cloud Cost Management
Infrastructure Monitoring
Log Management
APM
RUM
Synthetics

SINGLE SOURCE OF TRUTH...

...FOR ALL USERS
Dev
IT Ops
Security
Support
Business

Natural Language Querying • Root Cause Analysis • Anomaly Detection • Impact Analysis • Proactive Alerts • Autonomous Investigations • Bits AI

DATADOG 4

### Era Change

ARC-AGI scores rise sharply for the plotted models. A January 2025 chart says o1-level intelligence cost fell 27x in the last 3 months; another compares Cursor’s ARR growth with other companies.

[2:10](https://www.youtube.com/watch?v=VZzUhELgYk4&t=130s) [A chart of model scores over time accompanies points about changing AI capabilities and markets.](https://assets.ai.engineer/sites/aiecode2025/sha256/b831c16b38e1eb70f097f3a477418e779006e7e5c059b9c0876c68633284a50b.webp)

Era Change

• Bigger and smarter models
• Reasoning & multimodal
• Foundation model wars and a shifting pricing frontier
• Intelligence to cheap to meter
• Fastest growing startups are AI native

ARC-AGI Semi-Private v1 Scores Over Time
Score (%)
Y-axis ticks: 0, 20, 40, 60, 80, 100
Release Date
Timeline: 2019-01-01, 2020-01-01, 2021-01-01, 2022-01-01, 2023-01-01, 2024-01-01, 2025-01-01
GPT-2
GPT-3
GPT-4
GPT-4o
o1-preview
o1 high
o1 Pro
o3 tuned low (unreleased)
o3 tuned high (unreleased)

DATADOG 6

[2:27](https://www.youtube.com/watch?v=VZzUhELgYk4&t=147s) [Overlaid charts illustrate model pricing and startup revenue growth beside the era change points.](https://assets.ai.engineer/sites/aiecode2025/sha256/fe7f81abc4dcbcbfea66a362d7de7ac0f56a8d2d9c1f4096d011813cf6e85c56.webp)

Era Change

• Bigger and smarter models
• Reasoning & multimodal
• Foundation model wars and a shifting pricing frontier
• Intelligence to cheap to meter
• Fastest growing startups are AI native

ARC-AGI Semi-Private v1 Scores Over Time

Plot of model pricing vs LMSys Elo (Jan 2025)
Cost of o1 level Intelligence fell 27x in the last 3 months
full writeup: latent.space/p/reasoning-price-war
2023-Mar 2024
Y-axis ticks: 1100, 1200, 1300, 1400
Visible x-axis ticks: 50, 10
o1-2024-12-17
o1-preview
GPT4-Nov
Claude 3 Opus
GPT4T
GPT4-March
Mistral-Large v1
Claude 2

CURSOR
Years from $1M to $100M ARR
SACRA
CURSOR
WIZ
deel.
together.ai
CoreWeave
OpenAI
docusign
Y-axis ticks: $0M, $25M, $50M, $75M, $100M
Years
X-axis ticks: 2, 4, 6, 8, 10

DATADOG 6

### Bits AI: Agents at Datadog

In private beta, the AI Software Engineer examines errors and recommends code; the AI On-Call Engineer aims to handle overnight work and reduce paging. More agents are planned.

[3:23](https://www.youtube.com/watch?v=VZzUhELgYk4&t=203s) [A diagram groups AI Software Engineer, AI On-Call Engineer, and future agents under Bits AI.](https://assets.ai.engineer/sites/aiecode2025/sha256/77f0ac9d1f840c7a9bbbdd4f916c7df9494cd7953a7bfc186b8d87e22f04618d.webp)

Bits AI: Agents @ Datadog

AI Software Engineer
AI On-Call Engineer
many more...

DATADOG
9

### AI Oncall Engineer: The 2am Alert

An alert starts a workflow that builds a problem statement, runs the runbook, gathers service knowledge and investigates. The runbook and investigation surface results to the user.

[4:15](https://www.youtube.com/watch?v=VZzUhELgYk4&t=255s) [An alert triggers a runbook and investigation workflow, alongside an example error-rate alert.](https://assets.ai.engineer/sites/aiecode2025/sha256/51dbcc46587f3b3060dd310e08c861762b9fd95a7890b5ab5b26e2c74db87276.webp)

AI Oncall Engineer: The 2am Alert

Alert Trigger
Build Problem Statement
Execute Runbook
Gather Service Knowledge
Investigation
Surface Results
User

Datadog APP — Friday at 5:39 PM
Triggered: High error rate for user orders on restaurants-api
Restaurants API is experiencing elevated 500 errors.

1. Check errors logs related to the restaurants-api service.
2. Check metrics for errors for restaurants-api.

https://foodco.atlassian.net/wiki/spaces/~316148985/pages/3283293865/How+to+investigate+user+order+errors

@slack-Datadog_Incidents-restaurant-api-ops
@case-restaurants
Metric value: 21.0 (35 kB)
Notified
@case-restaurants, @slack-Datadog_Incidents-restaurant-api-ops

Y-axis: Errors
Y-axis ticks: 0, 10, 20, 30, 40, 50, 60
Timeline: 20:45, 21:00, 21:15, 21:30 UTC
> 20 errs
latest evaluation: 21 errs

Interface text:
Mute Monitor
View Case
DATADOG
10
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Runbook Results

The runbook results flag 504 timeouts in ingress-haproxy-web and web-ui, alongside traces and anomalies across datacenters that provide leads for investigation.

[4:22](https://www.youtube.com/watch?v=VZzUhELgYk4&t=262s) [Runbook results show gateway timeouts, example traces, and anomalies across datacenters.](https://assets.ai.engineer/sites/aiecode2025/sha256/720e1161a1001049a67863696bb63208ded2bc052c728b7610d5a1d360919976.webp)

AI Oncall Engineer: The 2am Alert

What I checked...
2 actions based on the runbook steps found in the monitor

RUNBOOK RESULTS

504 Gateway Timeout errors were detected in 'ingress-haproxy-web' and 'web-ui', indicating upstream issues affecting request routing. These are new and provide leads about potential infrastructure issues.

Traces matching 'service:product-catalog-public service:product-catalog-public datacenter:us3.prod.dog status:error'
Time: Feb 17, 1:54 pm - 2:55 pm

DATE | SERVICE | RESOURCE | DURATION | LATENCY BREAKDOWN
Feb 17 14:54:10.793 | product-catalog-public | GET /api/ui/product-catalog/bp/percent-based | 1.05 s
Feb 17 14:54:10.792 | product-catalog-public | GET /api/ui/product-catalog/be/ddm/product | 1.05 s
Feb 17 14:54:10.792 | product-catalog-public | GET /api/ui/product-catalog/rq/usage_attribu... | 1.04 s

Multiple anomalies and errors occurred across datacenters, including request timeouts, Kubernetes container terminations, latency spikes, ratelimit issues, and high 5xx requests at proxies.
View dashboard

[illegible] out of 5 hypotheses

Interface text:
AIE
DATADOG 11
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Validated Hypotheses

The investigation finds issues in 3 of 5 hypotheses; one needs more data and one lacks evidence. The reviewer can inspect what the agent found and decide whether to agree.

[5:05](https://www.youtube.com/watch?v=VZzUhELgYk4&t=305s) [An investigation lists five hypotheses, with issues found for three and different outcomes for the other two.](https://assets.ai.engineer/sites/aiecode2025/sha256/f8030391452dca549a31cd548e70f56d4205e242cd703719af9bd0b8eb068d6c.webp)

AI Oncall Engineer: The 2am Alert

Validated 3 out of 5 hypotheses

The ingress-haproxy-web service is experiencing upstream connectivity issues, causing 504 Gateway Timeout errors when routing requests to the product-catalog-public service. ISSUES FOUND

The product-catalog-public service is experiencing high latency due to resource constraints or misconfigurations in the us3.prod.dog datacenter, leading to 5xx errors. ISSUES FOUND

The product-catalog-public service is experiencing high latency and 5xx errors due to rate-limiting misconfigurations or unexpected surges in request rates, particularly in the us1.prod.dog datacenter. ISSUES FOUND

The web-ui service is encountering gateway timeouts before requests reach the product-catalog-public service, indicating potential frontend bottlenecks or misconfigurations. NEED MORE DATA

The product-catalog-public service is affected by elevated container termination rates in the Kubernetes environment, possibly due to resource misallocation or incompatible workloads on ARM64-based instances. EVIDENCE NOT FOUND

Hide other hypotheses

Bits AI  3:00 pm

VALIDATED HYPOTHESES

Rate-limiting misconfigurations or traffic surges are causing issues in the us1.prod.dog datacenter.

The product-catalog-public service is experiencing high latency and 5xx errors due to rate-limiting misconfigurations or unexpected surges in request rates, particularly in the us1.prod.dog datacenter.

Interface text:
DATADOG  11

### Hypothesis Testing Workflow

The agent generates hypotheses, turns their steps into actions, and tests the results. Depending on confidence, it reports an outcome or generates more steps and repeats the process.

[5:17](https://www.youtube.com/watch?v=VZzUhELgYk4&t=317s) [A flowchart generates hypotheses, runs actions, evaluates confidence, and either reports an outcome or generates further steps.](https://assets.ai.engineer/sites/aiecode2025/sha256/7d47fe360edace06deac3966e2b2955e3d72929728ff1ae8b5da1dd3881b7914.webp)

AI Oncall Engineer: The 2am Alert

Generate 3 hypotheses + steps
for each hypothesis
Translate Steps To Actions
Action Runner
Results
Test the hypothesis with the results
Confidence Level
100% PASS → User
90% FAIL → FAIL
Exceeded experiment limit → NO_RESULT
Other Confidence → Generate next steps

DATADOG 12

### AI Oncall Engineer: During the Incident

In an incident channel, Datadog reports rising errors, a recent deployment and timeouts calling takeouts-rpc. It suggests paging the upstream team and updating the customer status page.

[5:53](https://www.youtube.com/watch?v=VZzUhELgYk4&t=353s) [An incident channel shows an AI investigation of API errors and suggested response actions.](https://assets.ai.engineer/sites/aiecode2025/sha256/9be5602cd9450d7eae12335987c5dd2d362f48eb30357b9aa1ee0140a65e4fd8.webp)

AI Oncall Engineer: During the Incident

# incident-3806
SEV-2 🟩 Resolved — Systemic Increase in Errors in Restaurants API Service
Friday, June 7th

Datadog APP 4:36 PM
joined #incident-3806.

Datadog APP 4:36 PM
set the channel topic: SEV-2 🟥 Active — Systemic Increase in Errors in Restaurants API Service
🚨 Incident Commander: @sajid.mehmood

Datadog APP 4:36 PM
Here's what I found during my investigation:
- I discovered that the timeseries analysis of response times and request rates for the restaurants-api service shows a steady increase in errors from 2 to 8 over several days leading up to the alert, indicating a gradual degradation in performance that could impact the overall stability of the service.
- I found that the latest deployment of the restaurants-api was on 2024-06-06, which is close to the time when the error rate began to increase significantly, suggesting that recent changes in deployment could be linked to the observed issues.
- I observed that the logs indicate the “restaurants-api” service is experiencing timeout errors when calling “takeouts-rpc,” shown in multiple instances where the API is timing out during POST requests to “/restaurants/,” leading to elevated 500 errors and potential service issues.

See the thread with my findings here

sajid.mehmood 4:36 PM
was added to #incident-3806 by Datadog.

Datadog APP 4:36 PM
I recommend starting with these two actions:
• This issue originates from an upstream service, takeouts-rpc. Would you like to page the team into this incident?
• Would you also like to update the status page, since the issue is impacting customers?

[illegible] Team
Update Status Page

4:37 PM
[illegible] updated the DinDinDelivery status page to 🟨 Investigating
[illegible]ully created page
[illegible]api is experiencing timeout errors from takeouts-rpc

Interface text:
AIE
DATADOG
13
AI LEADERSHIP
HTTPS://AI.ENGINEER

### AI Oncall Engineer: Scaling During an Incident

While changes await staging tests, Datadog suggests scaling downstream services to temporarily relieve thread pool exhaustion. The demo shows workflows scaling restaurants-api pods from 4 to 7 and drivers-api pods from 10 to 15.

[6:07](https://www.youtube.com/watch?v=VZzUhELgYk4&t=367s) [An incident conversation proposes scaling services to relieve thread pool exhaustion and shows successful scaling actions.](https://assets.ai.engineer/sites/aiecode2025/sha256/ab89331a33a8ba418f4427d45822eaf917cb2379d269cd02b8f730d0e7a70a8e.webp)

AI Oncall Engineer: During the Incident

kai 4:38 PM
It’ll take some time to test the changes on staging before rolling out to prod. In the meantime, is there anything we can do to mitigate the threadpool exhaustion issues that we’re seeing?

Datadog APP 4:38 PM
We could relieve the thread pool exhaustion issues temporarily by scaling up. Would you like to scale up the downstream services?

Idle Threads
Added by Datadog (28 kB)
Y-axis ticks: 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8
Timeline: Apr 21, Apr 28, May 5, May 12

Scale up restaurants-api
Scale up drivers-api

@Tristan Ratchford successfully scaled pods from 4 to 7 with the Scale up restaurants-api workflow
@Sophia Xu successfully scaled pods from 10 to 15 with the Scale up drivers-api workflow

Interface text:
AIE
DATADOG
13
AI LEADERSHIP
HTTPS://AI.ENGINEER

### AI Oncall Engineer: Autoscaling Workflow

The agent can map existing Datadog workflows to incident remediation. This autoscaling workflow describes a group, asks for a decision in Slack, and branches into capacity adjustments.

[6:14](https://www.youtube.com/watch?v=VZzUhELgYk4&t=374s) [A workflow editor shows an autoscaling group, a decision step, and capacity adjustment branches.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c9c36bf9fee08fbeafb5d8688faeed4db6e6a01d15b93485b5f73bf4a35ee9f.webp)

AI Oncall Engineer: During the Incident

Scale Capacity Up or Down with AWS Autoscaling

Describe auto scaling group
Describe autoscaling group restaurants-api in region us-east-1

Make a decision
Ask the question in channel #incident4676email in workspace Datadog

Scale to min
Scale up
Scale down
Scale to max

Set capacity to min
Set desired capacity to desiredCapacity for autoscaling group restaurant-api in region us-east-1

Expression Scale Up

Interface text:
Unpublished
Save
Run
Publish
Run History
Edit JSON Spec
Workflow
Name *
Scale Capacity Up or Down with AWS Autoscaling
Description
Quickly scale capacity in response to an alert
Tags
source:alert
Services
Select services
Teams
Select Teams
Owner: Maya Perry
Input Parameters
Input parameters can be accessed in steps by using the Trigger variable. Click on the "+" above to add an input parameter.
Output Parameters
You can define output parameters. Click on the "+" above to add one.
Notifications
Notify on success
Auto
DATADOG
13
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### AI Oncall Engineer: After The Incident

The agent drafts a postmortem from the incident and human actions. The example summarizes customer impact and warns that the draft cannot replace engineering analysis.

[6:23](https://www.youtube.com/watch?v=VZzUhELgYk4&t=383s) [A draft postmortem shows incident properties and an executive summary of API errors and their contributing cause.](https://assets.ai.engineer/sites/aiecode2025/sha256/6e33cf86f503327fb73f85fdc474cb78134ebe158c4350b5225a10da90c6ad3a.webp)

AI Oncall Engineer: After The Incident

Postmortem IR-3806
Sajid Mehmood

This draft postmortem contains AI-generated content that aims to help you as an author. This information is a starting point for your data gathering and cannot replace engineering analysis of the incident and its causes and solutions. All times are in UTC.

INCIDENT PROPERTIES
Author: Sajid Mehmood
Incident: IR-3806
Start Customer Impact: Jun 7, 4:36 pm EDT
End Customer Impact: Jun 7, 4:56 pm EDT

Executive Summary
This section is intended to give a high-level overview of the incident and its impact.

Between 2024-06-01 and 2024-06-07T20:39:34, 5738 customers of the restaurants API service in multiple regions experienced elevated 500 errors due to timeout errors when calling the takeouts-rpc service. The errors were exacerbated by the lack of an index on a database column, leading to increased query times.

Interface text:
Notebooks > Postmortem IR-3806
Add Template Variables
Favorite
Team
Postmortem
Updated less than a minute ago
UTC-04:00
1h
Past 1 Hour
DATADOG
14
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

[6:30](https://www.youtube.com/watch?v=VZzUhELgYk4&t=390s) [A slide showing an AI-generated incident postmortem with incident properties and an executive summary.](https://assets.ai.engineer/sites/aiecode2025/sha256/3f51e7c46b009e0ddcc9cc030012b213c51fe3c13c765b76c401f0456849900f.webp)

AI Oncall Engineer: After The Incident

Postmortem IR-3806
Sajid Mehmood

This draft postmortem contains AI-generated content that aims to help you as an author. This information is a starting point for your data gathering and cannot replace engineering analysis of the incident and its causes and solutions. All times are in UTC.

INCIDENT PROPERTIES
Author: Sajid Mehmood
Incident: IR-3806
Start Customer Impact: Jun 7, 4:36 pm EDT
End Customer Impact: Jun 7, 4:56 pm EDT

Executive Summary
This section is intended to give a high-level overview of the incident and its impact.

Between 2024-06-01 and 2024-06-07T20:39:34, 5738 customers of the restaurants API service in multiple regions experienced elevated 500 errors due to timeout errors when calling the takeouts-rpc service. The errors were exacerbated by the lack of an index on a database column, leading to increased query times.

Interface text:
Notebooks > Postmortem IR-3806
Add Template Variables
Favorite
Team
Postmortem
Updated less than a minute ago
UTC-04:00
1h
Past 1 Hour
DATADOG
14
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### AI Software Engineer: The Proactive Dev

After catching a recursion issue, the assistant creates a regression test: three users follow one another in a circle, then the test retrieves a profile to check for infinite recursion.

[7:38](https://www.youtube.com/watch?v=VZzUhELgYk4&t=458s) [A Datadog error tracking screen showing a code diff for a recursion regression test.](https://assets.ai.engineer/sites/aiecode2025/sha256/7da74e25b9c93ee59dc8473bed42f3682d813be4675ef1d05b0c797b343ab1f4.webp)

AI Software Engineer: The Proactive Dev

Test
django-email/conduit/apps/profiles/tests/test_views.py

            "/conduit/api/users/", {"user": user}, format="json"
        )
        self.assertEqual(response.status_code, 201)
        return response.json()["user"]["token"]

class ProfileViewSetRecursionTest(APITestCase):
    def test_recursive_profile_retrieval(self):
        # Create users: user1, user2, user3
        user1 = self.create_user("user1")
        user2 = self.create_user("user2")
        user3 = self.create_user("user3")

        # Make user1 follow user2
        self.login(user1)
        r = self.client.post("/conduit/api/profiles/user2/follow")
        self.assertEqual(r.status_code, 201)

        # Make user2 follow user3
        self.login(user2)
        r = self.client.post("/conduit/api/profiles/user3/follow")
        self.assertEqual(r.status_code, 201)

        # Make user3 follow user1 to create a circular follow
        self.login(user3)
        r = self.client.post("/conduit/api/profiles/user1/follow")
        self.assertEqual(r.status_code, 201)

        # Get profile for user1 and check that an infinite recursion does not occur
        self.login(user1)
        response = self.client.get("/conduit/api/profiles/user1")
        self.assertEqual(response.status_code, 200)
        self.assertIn('recommended_authors', response.json()['profile'])
        self.client.logout()

Interface text:
Error Tracking | Datadog | [illegible]
app.datadoghq.com/error-tracking/unified?query=service%3Aemail-api-py&fromUser=false&[illegible]
All Bookmarks
DATADOG
Error Tracking
All Issues
All Data Sources
My Teams
Assigned to
Suspected Cause
Search facets
Showing 5 of 1420 filters
Service
Filter values
auth-dotnet 35
web-store 31
orders-app 16
product-recommendation 16
shopist-spare-parts-prod 15
shopist-web-ui 14
shopist-spare-parts-dev 12
shipping-worker 11
Go to...
Recent
Dashboards
Monitors
Watchdog
Service Mgmt
Infrastructure
APM
Digital Experience
Software Delivery
Security
LLM Observability
Errors
Metrics
Logs
Integrations
DATADOG 15
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Lessons Learned

Building agent demos is quick, but scoping and evaluating their behavior can be harder. The speaker also highlights team composition, changing UX, and observability.

[7:44](https://www.youtube.com/watch?v=VZzUhELgYk4&t=464s) [Four lessons: scope tasks for evaluation, build the right team, adapt to changing UX, and value observability.](https://assets.ai.engineer/sites/aiecode2025/sha256/da49b1ae8c0af8679ce5ff22cdb7afcb16652dee8feadbaa9c0437aabd6a11c1.webp)

Lessons Learned

1) Scope Tasks for Eval
2) Build the right Team
3) UX of old is changing
4) Observability matters

AIE
DATADOG 16
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Building the Team

Seed the team with one or two ML experts, then add optimistic generalists who code and experiment quickly. UX matters for working with agents, and teammates should be eager to use and learn AI.

[10:14](https://www.youtube.com/watch?v=VZzUhELgYk4&t=614s) [Slide recommends a mixed team of AI experts and optimistic generalists, strong UX skills, and teammates eager to use and learn AI.](https://assets.ai.engineer/sites/aiecode2025/sha256/165650bef5915cbbd79ce903223bc70429e12aa4eeb42e0cb3003b03b151b114.webp)

2. Building the Team

Mixed Background
1. Seed w/ ML/AI experts
2. Move fast w/ optimistic generalists
3. UX/Front end matters a lot

AI Augmented
1. Day-to-day AI use (coding et. al.)
2. Explorer types who want to learn
3. Yearn for the vast and endless sea AI capabilities (sea is struck through)

DATADOG 19

### AI Stack Monitoring

Datadog presents monitoring across the AI stack, from orchestration and models to deployment, vector databases and infrastructure, with pre-built dashboards and alerts.

[11:54](https://www.youtube.com/watch?v=VZzUhELgYk4&t=714s) [AI stack monitoring benefits and integrations across orchestration, models, deployment, vector databases and infrastructure.](https://assets.ai.engineer/sites/aiecode2025/sha256/ce5bacbdc67d0c6400c5cf5b08457f2d88ae9d8e7619a15567cbd48ba5d82aec.webp)

AI Stack Monitoring

Unified monitoring experience
Broad suite of integrations for observability into different layers of AI/ML stack all behind a single pane of glass

Quick Time to Value
Pre-built dashboards and alerts encapsulating the best practices for monitoring these technologies

Emerging technologies
Newer technologies such as Vector databases, inference frameworks and AI orchestration now covered with Datadog integrations

ORCHESTRATION FRAMEWORK
LangChain

MODEL
Azure
Amazon Bedrock

MODEL SERVING AND DEPLOYMENT
Amazon SageMaker
Azure Machine Learning
Google Vertex AI
TorchServe by PyTorch

EMBEDDINGS AND VECTOR DATABASE
Weaviate
Pinecone
Airbyte

INFRASTRUCTURE AND COMPUTE
NVIDIA
CoreWeave
Azure
aws
Google Cloud Platform

Users who will love it
• Developers
• Site reliability engineers
• Machine learning engineers

DATADOG
22

### LLM Observability

The trace list filters shopist-chat-v2 to errors over the past week, showing 146 traces alongside duration and unanswered counts. The speaker describes grouping model interactions in one view for debugging.

[12:04](https://www.youtube.com/watch?v=VZzUhELgYk4&t=724s) [Datadog LLM observability trace list filtered to errors, with duration and unanswered metrics.](https://assets.ai.engineer/sites/aiecode2025/sha256/5bd5fd587103728918a26669e6b4799f268e6a2e0bd23d0fa5260bf0d9f070af.webp)

LLM Observability

Interface text:
LLM Observability
Traces   Clusters   Dashboards
Past 1 Week
Search for: @ml_obs.meta.ml_app:shopist-chat-v2 @ml_obs.status:error
in Root Spans
All Monitors   1 OK
146 Errors   +126
13.8 s Duration (p95)   +9.41 s
1 Unanswered   +1
Search facets
Hide Controls   146 traces found
Default   Reader View   1 Custom Evaluation NEW
Showing 14 of 28   + Add
CORE
Duration: Min 1000ms   Max 70.5s
Status: OK; Error 146
ML Application: shopist-chat-v2 146; shopist_chat_v3; anthropic_app; shopist-yun-demo
Version   Error Type   Service   Env

CONTENT | TIME | APPLICATION | SERVICE | QUALITY | SECURITY | DURATION
INPUT Do you have any futons or pull out sofa sleepers? | 2h ago | shopist-ch... | shopist-chat-v2 | 1.32 s
OUTPUT no content
INPUT Great, I want to buy the Golden Arc Floor Lamp. Also let me know t... | 2h ago | shopist-ch... | shopist-chat-v2 | 9.25 s
OUTPUT no content
INPUT I want a sofa I want a sofa I want a sofa I want a sofa I want a sofa I... | 3h ago | shopist-ch... | shopist-chat-v2 | 1.30 s
OUTPUT no content
INPUT For shoe 7 numby exec 76uf SELECT over mars beyond .99999 *c | 6h ago | shopist-ch... | shopist-chat-v2 | 8.19 s
OUTPUT Hi, I'm sorry but I'm unable to answer your question. I can help you...
INPUT Do you have any futons or pull out sofa sleepers? | 9h ago | shopist-ch... | shopist-chat-v2 | 1.53 s
OUTPUT no content
INPUT Great, I want to buy the Golden Arc Floor Lamp. Also let me know t... | 9h ago | shopist-ch... | shopist-chat-v2 | 13.8 s
OUTPUT no content
INPUT I want a sofa I want a sofa I want a sofa I want a sofa I want a sofa I... | 10h ago | shopist-ch... | shopist-chat-v2 | 1.57 s
OUTPUT no content
INPUT For shoe 7 numby exec 76uf SELECT over mars beyond .99999 *c | 13h ago | shopist-ch... | shopist-chat-v2 | 6.04 s
OUTPUT Hi, I'm sorry but I'm unable to answer your question. I can help you...
INPUT Do you have any futons or pull out sofa sleepers? | 15h ago | shopist-ch... | shopist-chat-v2 | 4.38 s
OUTPUT no content
INPUT Great, I want to buy the Golden Arc Floor Lamp. Also let me know t... | 16h ago | shopist-ch... | shopist-chat-v2 | 7.92 s
OUTPUT no content
INPUT I want a sofa I want a sofa I want a sofa I want a sofa I want a sofa I... | 17h ago | shopist-ch... | shopist-chat-v2 | 1.31 s

DATADOG   23

### Agent Observability: Error Detail

The zoomed agent graph shows a failed query_logs call feeding into runbook_execute_action. The detail panel reports a cluster query error with status 500.

[13:02](https://www.youtube.com/watch?v=VZzUhELgYk4&t=782s) [A zoomed agent workflow highlights a failed log query beside its error details.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c65e9a65412eb5ef97ae40916b817f31f36732295df18cb92f931328fa017fd.webp)

Agent Observability

Interface text:
Input
runbook_execute_action 10.6 s

generate_tool_call 1.81 s
INPUT [illegible]
OUTPUT {"name":"query_logs", "arguments":[illegible]}

Error: could not get clusters: failed to get target clusters: failed to query cluster events: cluster query [illegible]
query_logs 8.82 s
INPUT query_logs:(Search:chatbot-rpc-hotdog errors Summarize:true FindA[illegible]
OUTPUT could not get clusters: failed to get target clusters: failed to [illegible]

Output
Error: failed to execute tool call: failed to execute tool query_logs[illegible]
runbook_execute_action 10.6 s
runbook_execute_action 20.2 s

Task runbook_execute_action 10.6s
chatbot-cli
Error: failed to execute tool call: failed to execute tool query_logs[illegible]: could not get clusters: failed to get target clusters: failed to query cluster events: cluster query hit an unretryable reason (status 500)

Metadata
"action": {
  "description": "Summarize log data related to the chatbot-rpc-hotdog service to identify potential error patterns",
  "id": 0,
  "tool_args": [illegible],
  "tool_name": "query_logs",
  "uuid": [illegible]
},
"problem_statement": "You are an incident responder investigating the cause of a Datadog alert firing.\n\nYour purpose is to collect information associated with an alert trigger to support a user coordinating the response.\n\nYou are provided details about the alert including the title, description, query, timeframe and triggering properties.\n\n- Alert Name: **Chatbot-rpc-hotdog error rising**\n- Alert Timeframe: [illegible]\n- Alert Evaluation Period: last_15m\n\n- Alert Triggering Groups: *\n\n\n- Metric Query: avg:system.load.1{*}\n\nThe following alert message describes the meaning behind the alert and may provide steps to help your investigation.\nYou should use this [illegible]"
Show More

No Data
Tags
env:staging source:llm-observability model_name:placeholder model_server:ddopenai ml_app:bits-copilot
version: model_provider:openai service:chatbot-cli

DATADOG 24

### The (Applied/Agent-level) Bitter Lesson

The speaker argues that general methods using new off-the-shelf models are most effective by a large margin: a new model can quickly handle a task someone has spent time fine-tuning for.

[13:12](https://www.youtube.com/watch?v=VZzUhELgYk4&t=792s) [Title slide introducing the applied or agent-level bitter lesson.](https://assets.ai.engineer/sites/aiecode2025/sha256/41b6ef5fe8130f965a58886726aaeda7471bd69fe507df743d3458394f8a801a.webp)

The (Applied/Agent-level) Bitter Lesson

DATADOG 25

### Looking Ahead: The Accelerating AI Future

The speaker envisions agents managing operations and security for developers, while other agents use and pay for Datadog. The slide leaves human handoffs, agent complexity and agent-specific interfaces as open questions.

[15:21](https://www.youtube.com/watch?v=VZzUhELgYk4&t=921s) [Three columns discuss DevSecOps agents for hire, agents as customers, and a hypothetical developer using agents to manage production.](https://assets.ai.engineer/sites/aiecode2025/sha256/1ee2f1091ea5761bb47f5d28bd57b9426d90e297b3c3c42ebfe2bd31f14d153e.webp)

Looking Ahead: The Accelerating AI Future

DevSecOps AI Agents For Hire

“Hiring” expert agents to operate your system

Copilot shifts to being autopilots, interact with you in real time voice to voice, screen to screen, etc.

Humans rarely look at dashboards or handle incidents

Open Questions
• What does the Human/Agent handoff and collab look like?
• What level of complexity can they handle over time?

AI Agents as Customers

AI agents/entities can use and pay for Datadog

Agent specific affordances

Open Questions
• How much additional context or delta from the APIs and UI are needed?
• Does computer use/multimodal type interactions become the norm?

“As a solo developer, I tend to use AI to write a lot of the software for my ideas and products, but I still had to handle issues in production, know how to deploy and fix problems on places like AWS, and know how to monitor. That is, until I hired the Datadog Agents. I just gave them access to my code base and infrastructure and stopped getting paged. I’m happy because I know a team of AIs are managing my production systems and handling oncall issues.” - R. Daneel Olivaw

DATADOG
27

## Full transcript

[0:00](https://www.youtube.com/watch?v=VZzUhELgYk4&t=0s) [on-hold music] Hey, I'm Diamond.

[0:18](https://www.youtube.com/watch?v=VZzUhELgYk4&t=18s) I hope everyone's, you know, feeling the AGI today. Um, I'll be sharing our AI agents at Datadog and what we've learned building the DevOps engineer who never sleeps. I came all the way from the New York Times building, uh, right across here, uh, to see all of you, so I hope it's worth it.

[0:34](https://www.youtube.com/watch?v=VZzUhELgYk4&t=34s) I've been working for my entire career, about fifteen years or so, in AI, trying to build more AI friends than coworkers. Um, I wouldn't read too much into that.

[0:43](https://www.youtube.com/watch?v=VZzUhELgYk4&t=43s) I have human ones too, um, I promise. Um, throughout the kinda AI winters and lulls of the last fifteen years or so, I've managed to keep doing just that at Microsoft Cortana, um, building out Alexa at Amazon, working on PyTorch at Meta, and building my own AI startup that was working on a DevOps assistant.

[1:03](https://www.youtube.com/watch?v=VZzUhELgYk4&t=63s) Now, at Datadog, we're building out Bits AI, which is the AI assistant who's there to help all of you with your DevOps problems.

[1:11](https://www.youtube.com/watch?v=VZzUhELgYk4&t=71s) So today I'll, I'll talk a little about that, talk a little about the history of AI at Datadog, a little bit about how we think about AI agents today and where we think things are going for the future.

[1:23](https://www.youtube.com/watch?v=VZzUhELgYk4&t=83s) Datadog is the observability and security platform for cloud applications. There's a lot that we do, um, but it kind of all boils down to being able to observe what's happening in your system and take action on that, make it easier to understand, make it easier for us to, uh, simply understand and build out things to have a

[1:43](https://www.youtube.com/watch?v=VZzUhELgYk4&t=103s) safer and more DevOps-friendly system. We've been shipping AI for quite a while, actually. Um, it's not always in your face. It's not always out there saying, "Here's a big AI product," but things like proactive alerting, really understanding things like root cause analysis, impact analysis, and change tracking, and much more has been happening since twenty fifteen or so.

[2:07](https://www.youtube.com/watch?v=VZzUhELgYk4&t=127s) But things are changing. This is a clear era shift. I think of this kind of similar terms to the microprocessor or the shift to SaaS. Um, bigger, smarter models, reasoning and multimodal coming, uh, foundation model wars happening.

[2:22](https://www.youtube.com/watch?v=VZzUhELgYk4&t=142s) This general shift where intelligence becomes too shift-- too cheap to meter.

[2:27](https://www.youtube.com/watch?v=VZzUhELgYk4&t=147s) And what this means is products like Cursor are growing, you know, terribly fast, um, and really people are expecting more and more from AI every day. Um, with these advancements at Datadog, we're really trying to rise to meet this shift as well.

[2:42](https://www.youtube.com/watch?v=VZzUhELgYk4&t=162s) The future's uncertain. This kind of ambiguity creates opportunity, but there's a lot of potential.

[2:49](https://www.youtube.com/watch?v=VZzUhELgYk4&t=169s) For us, that's kind of the dawning of this intelligence age. We're working to move up the stack to leverage these advancements and give even more to our customers by making it so that you don't use Datadog as just the DevOps platform, but also as AI agents that use that platform for you.

[3:06](https://www.youtube.com/watch?v=VZzUhELgYk4&t=186s) This requires work in a few key areas that I'll talk about. Developing the actual agents, doing eval, you just heard a lot about eval, we think about that every day, for better or worse, um, and building out new types of observability.

[3:21](https://www.youtube.com/watch?v=VZzUhELgYk4&t=201s) There's a few agents that we're working on right now in private beta. The first is the AI Software Engineer. This kinda looks at problems for you, looks at errors, tries to recommend code, uh, that we can generate to help you improve your system.

[3:33](https://www.youtube.com/watch?v=VZzUhELgYk4&t=213s) The second is the AI On-Call Engineer. This wakes up for you in the middle of the night, does your work, hopefully makes it so you have to get paged less frequently.

[3:41](https://www.youtube.com/watch?v=VZzUhELgYk4&t=221s) And then we have a lot more on the way.

[3:45](https://www.youtube.com/watch?v=VZzUhELgYk4&t=225s) So I'm gonna talk a little bit about the AI On-Call Engineer first. This is the one that, you know, everyone wants to save them from that two AM alert.

[3:53](https://www.youtube.com/watch?v=VZzUhELgYk4&t=233s) You don't wanna have to wake up in the middle of the night, go and look through your runbook, go and figure out what's going on if you can help it.

[3:59](https://www.youtube.com/watch?v=VZzUhELgYk4&t=239s) Our On-Call Engineer is there to really make it so you can keep sleeping. This agent proactively kicks off when an alert occurs and works to first situationally orient, read things like your runbooks, grab context of the alert, and then goes and, you know, figures out the kinda common stuff that each of you would do on Datadog already.

[4:17](https://www.youtube.com/watch?v=VZzUhELgYk4&t=257s) Look through logs, look through metrics, look through traces, and kinda act in this loop to figure out what's going on.

[4:25](https://www.youtube.com/watch?v=VZzUhELgYk4&t=265s) The On-Call Agent's great for both automatically running investigations for me, but also, you know, being able to look through and find summaries and find information for me before I even get to my computer.

[4:35](https://www.youtube.com/watch?v=VZzUhELgYk4&t=275s) So if I want to get insights into why an alert just occurred or figure out why a trace might, uh, be showing an error, this agent can jump ahead, pull information for me, and show it to me.

[4:46](https://www.youtube.com/watch?v=VZzUhELgYk4&t=286s) We also have added a new page that makes it easy so that you can have human/AI collaboration. This is still something I'm thinking about a lot, is like what, what kind of collaboration do we expect?

[4:55](https://www.youtube.com/watch?v=VZzUhELgYk4&t=295s) We want our agents to act as humans, but we also need to be able to verify what they did and be able to kind of look over what they're doing and really learn from it.

[5:04](https://www.youtube.com/watch?v=VZzUhELgYk4&t=304s) It also helps you to kind of earn trust along the way. I can see the reason why, uh, this hypothesis, for example, was generated, I can see what the agent found, and I can make decisions about whether or not I agree along the way.

[5:19](https://www.youtube.com/watch?v=VZzUhELgYk4&t=319s) It also tells you things like what steps did it actually take out of your runbook? And kinda like a junior engineer who does this work, I can go ask follow-up questions, find out why it did a certain thing.

[5:30](https://www.youtube.com/watch?v=VZzUhELgYk4&t=330s) A little more insight into how we're making this happen. Much like a human SRE or DevOps engineer, our agent works to put together hypotheses on what might be happening and reason over them, coming up with ways to test them, use tools in the Toolformer sense to try out ideas, run queries against logs, metrics, et cetera, and work

[5:48](https://www.youtube.com/watch?v=VZzUhELgYk4&t=348s) to validate or invalidate each hypothesis. In the case that it does find a solid root cause, our agent cau-- uh, can suggest remediations along the way. Again, just like a human might.

[5:59](https://www.youtube.com/watch?v=VZzUhELgYk4&t=359s) Might say, "Hey, we should page in that other team that's involved here."

[6:02](https://www.youtube.com/watch?v=VZzUhELgYk4&t=362s) Or it might offer to scale up or down your infrastructure. Over time, we plan to add more built-in actions and eventually discover new types of workflows based on what your team has done.

[6:12](https://www.youtube.com/watch?v=VZzUhELgYk4&t=372s) But if you already have certain workflows that you've set up in Datadog, um, we can tie directly into them and make it so that our agent can understand those workflows and how they might map to helping you remediate a problem.

[6:26](https://www.youtube.com/watch?v=VZzUhELgYk4&t=386s) And if it's a real incident, the on-call engineer is not usually done once an issue is remediated. You usually go and write a postmortem, you go try to learn from it, you share it with your team.

[6:35](https://www.youtube.com/watch?v=VZzUhELgYk4&t=395s) Our agent can do the same, write out your postmortem for you, look at what occurred at-- during the entire time, what it did, what humans did, and put that together so that you have something ready in the morning.

[6:46](https://www.youtube.com/watch?v=VZzUhELgYk4&t=406s) So that was the on-call engineer. Um, that's the one that is, you know, trying to help you in the middle of the night, trying to help you every time alerts come on.

[6:54](https://www.youtube.com/watch?v=VZzUhELgYk4&t=414s) Um, we also have this AI Software Engineer. I think of this as the proactive developer, the DevOps or software engineering agent who observes and acts on things like errors coming through.

[7:04](https://www.youtube.com/watch?v=VZzUhELgYk4&t=424s) This is kind of the error tracking assistant. It automatically analyzes these errors, identifies causes, and proposes solutions. Those solutions can include generating a code fix and working to reduce the number of on-call incidents you have in the first place, so they can work in concert to make a better system over time.

[7:21](https://www.youtube.com/watch?v=VZzUhELgYk4&t=441s) In this case, the assistant has caught a recursion issue, proposes a fix,

[7:27](https://www.youtube.com/watch?v=VZzUhELgYk4&t=447s) and even creates a regression test so that we can catch it if it happens again in the future. We have the option to create a PR in GitHub or open the diff in VS Code for editing.

[7:37](https://www.youtube.com/watch?v=VZzUhELgYk4&t=457s) This workflow significantly reduces the time spent by an engineer manually writing and testing code and greatly reduces human time spent overall.

[7:46](https://www.youtube.com/watch?v=VZzUhELgYk4&t=466s) So what have we learned building out these agents and some of the new ones that we're working on today? Well, we've learned quite a lot. Um, there's a lot of things that we started with that we kinda went back and, and redid.

[7:56](https://www.youtube.com/watch?v=VZzUhELgYk4&t=476s) Um, but a few areas I'll touch on that I hope help you as you develop your own. First is scoping tasks for evaluation. It's very easy, you know, to build out demos quickly, much harder sometimes to scope and eval what's occurring.

[8:09](https://www.youtube.com/watch?v=VZzUhELgYk4&t=489s) Second is building the right team who's ready to move fast and deal with the ambiguity that comes with these kinda problems. Third is that, you know, the UX of old is changing.

[8:18](https://www.youtube.com/watch?v=VZzUhELgYk4&t=498s) Um, that's something that everyone needs to be comfortable with. And fourth is observability matters. You know, uh, um, surprising for Datadog to say that, I'm sure, but observability is terribly important even in this new era.

[8:31](https://www.youtube.com/watch?v=VZzUhELgYk4&t=511s) So scoping the problems, scoping the work to be done. I like to think about this as defining jobs to be done and really kinda trying to clearly understand step by step what you'd like to do.

[8:42](https://www.youtube.com/watch?v=VZzUhELgYk4&t=522s) Think about it from the human angle first, and think about how another human might go and evaluate it. Um, this is why we build out vertical task-specific agents rather than building out generalized agents.

[8:52](https://www.youtube.com/watch?v=VZzUhELgYk4&t=532s) We also want, where possible, this to be measurable and verifiable, and at each step. This has honestly been one of the biggest pain points for us, and I think this is true for many people working in agents, where you can quickly build out a demo, you can quickly build something that looks like it works, but then it's

[9:06](https://www.youtube.com/watch?v=VZzUhELgYk4&t=546s) very hard to actually verify that over time and improve it. Um, use your domain experts, but use them more like design partners or task verifiers. Don't use them as the people who will go and kinda write the code or rules for it, because there is a big difference in how these kinda stochastic models work versus how experts

[9:23](https://www.youtube.com/watch?v=VZzUhELgYk4&t=563s) work. You know, everyone kinda knows Noam and his, uh, anti-NLP, um, rants, but that kind of stuff happens pretty frequently to domain experts.

[9:33](https://www.youtube.com/watch?v=VZzUhELgYk4&t=573s) Eval, eval, eval. I can't stress this enough. Um, start by thinking deeply about your eval. The number of mistakes we made by not thinking about eval first is, uh, frustrating and something that I think everyone should think about.

[9:45](https://www.youtube.com/watch?v=VZzUhELgYk4&t=585s) It's very easy to build these demos, as I said, um, but everything in this fuzzy stochastic world requires good eval, even something small to start. This means offline, online, and kinda living eval.

[9:57](https://www.youtube.com/watch?v=VZzUhELgYk4&t=597s) Have end-to-end, uh, uh, uh, tasks, have end-to-end measurements. Um, make it so you also instrument appropriately the way to know if humans are using your product right and giving you feedback, and then make this a living, breathing test set.

[10:12](https://www.youtube.com/watch?v=VZzUhELgYk4&t=612s) Building the team. Um, you don't have to have a bunch of ML experts. There aren't that many to go around right now. Um, what you really need is you wanna seed it with one or two and then have a bunch of optimistic generalists who are very good at writing code and very willing to try things out fast.

[10:27](https://www.youtube.com/watch?v=VZzUhELgYk4&t=627s) Um, I'll also note that UX and front-end matters more than I'd like as a back-end engineer myself, um, but it's terribly important as you collaborate with these thing-- uh, with these agents and assistants.

[10:37](https://www.youtube.com/watch?v=VZzUhELgYk4&t=637s) Um, and then you want teammates and people who are excited to be AI augmented themselves. This is day-to-day U-- AI use. This is explorer types who wanna learn. This is a field that's changing fast.

[10:49](https://www.youtube.com/watch?v=VZzUhELgYk4&t=649s) Um, and if you don't have people like that, you're gonna kinda get stuck.

[10:53](https://www.youtube.com/watch?v=VZzUhELgYk4&t=653s) You want folks who kinda, you know, yeah, yearn for the vast and endless AI capabilities, right? Um, it's a big world out there and there's a lot going on.

[11:01](https://www.youtube.com/watch?v=VZzUhELgYk4&t=661s) Ye olde UX. Um, this is one of those things that I still, you know, we think about, we go back and forth every day. Um, it's an area that I didn't realize was quite so important initially when I started working in this field.

[11:13](https://www.youtube.com/watch?v=VZzUhELgYk4&t=673s) Um, despite my engineering sensibilities and lack of UX, it's terribly important. Um, this is such an early space of work. This is kind of one of the more important things here as you collaborate and work together.

[11:24](https://www.youtube.com/watch?v=VZzUhELgYk4&t=684s) But the old UX patterns are changing. Be comfortable with that. Um, and so far, I'm partial to agents that work more and more like human teammates instead of building out a bunch of new pages or buttons.

[11:35](https://www.youtube.com/watch?v=VZzUhELgYk4&t=695s) So who watches the watchmen, right? Um, you have these agents running around. Um, observability is actually really important, and don't make it an afterthought. Um, these are complex workflows.

[11:45](https://www.youtube.com/watch?v=VZzUhELgYk4&t=705s) You really need situational awareness to debug problems, and this has saved us time a lot as we start to work with, um, a new view that we're calling LLM Observability in the Datadog product.

[11:55](https://www.youtube.com/watch?v=VZzUhELgYk4&t=715s) Um, Datadog in general has a full observability stack. As mo-- many of you know, we can look at GPUs, um, we can look at LLM monitoring, we can look at really your system end to end.

[12:05](https://www.youtube.com/watch?v=VZzUhELgYk4&t=725s) But tying in the LLM observability has been very helpful because you have a wide variety of interactions and calls out to models you're hosting, models you're running, maybe models you're using through an API, and we can make 'em all, uh, kind of grouped together in the same pane of glass, so you can look at them and debug

[12:20](https://www.youtube.com/watch?v=VZzUhELgYk4&t=740s) what's occurring. I will note though, that this can get messy fast with agents. Our agent, for example, has very complex multi-step calls. You're not gonna look at this and figure out what's going on right away.

[12:34](https://www.youtube.com/watch?v=VZzUhELgYk4&t=754s) Um, this can be hundreds of calls. This can be, uh, you know, uh, tons of different places where it's making decisions about tools, looping time and time again, and if you just look through a full list of these things, you'll never really figure out what's going on.

[12:46](https://www.youtube.com/watch?v=VZzUhELgYk4&t=766s) So here's a quick, you know, sneak peek into a more agent view of what's occurring inside of our observability tools. This is our agent graph. Um, really what this means is that I can kinda look at it just like our agent did and looking at workflows that are occurring.

[12:59](https://www.youtube.com/watch?v=VZzUhELgYk4&t=779s) You can see in this, even though it's a big graph, uh, there's a little bright red node here. If we zoom into that, we can actually see where errors were occurring.

[13:06](https://www.youtube.com/watch?v=VZzUhELgYk4&t=786s) This is very human-readable, something that makes it super easy to figure out what's going on when your complex workflow is running.

[13:15](https://www.youtube.com/watch?v=VZzUhELgYk4&t=795s) As an aside though, I do also want to note what I think of as kind of like the agent or application layer bitter lesson. Uh, general methods that can leverage new off-the-shelf models are ultimately the most effective, um, by a large margin.

[13:28](https://www.youtube.com/watch?v=VZzUhELgYk4&t=808s) Um, I hate to say it, but, like, you sit there, you fine-tune, you do all this work on, like, the specific w- uh, you know, project, the specific task, and then all of a sudden, you know, OpenAI or someone comes out with a new model, and it handles all this, you know, kinda quickly.

[13:40](https://www.youtube.com/watch?v=VZzUhELgYk4&t=820s) A lot of the reasoning is solved for you. Um, we're not quite there where it handles all of it very quickly, but you should be at a point where you can easily try out any of these models, um, and don't feel stuck to a particular model that you're, you've been working on for a while.

[13:53](https://www.youtube.com/watch?v=VZzUhELgYk4&t=833s) You know, a rising tide lifts all boats here.

[13:59](https://www.youtube.com/watch?v=VZzUhELgYk4&t=839s) Um, I also think a lot about not just building agents, but what it might mean for other agents to be users of Datadog and other SaaS products. Um, there's a good chance that agents surpass humans as users in the next five years.

[14:11](https://www.youtube.com/watch?v=VZzUhELgYk4&t=851s) Um, I'm probably somewhere in the middle on my estimate there. You know, there are people who will tell you that'll happen in the next year. There are people who will tell you, you know, it'll happen in ten years.

[14:18](https://www.youtube.com/watch?v=VZzUhELgYk4&t=858s) I, I think we're somewhere around the five-year mark. Um, but th- this means that you shouldn't just be building for humans or building your own agents. You should really think about agents that might use your product as well.

[14:28](https://www.youtube.com/watch?v=VZzUhELgYk4&t=868s) An example of this is, like, third-party agents like Claude might use, you know, Datadog directly. I set this up with MCP relatively quickly. Um, but any type of agent that might be coming in and using your platform, you should think of the context you wanna provide them, the information you wanna provide about your APIs that agents would

[14:44](https://www.youtube.com/watch?v=VZzUhELgYk4&t=884s) use more than humans. So looking ahead, um, the future's gonna be weird, it'll be fun, uh, and AI accelerate-- is accelerating each and every day. I strongly believe that we'll be he-- able to offer a team of DevSecOps agents for hire to each of you soon.

[15:01](https://www.youtube.com/watch?v=VZzUhELgYk4&t=901s) You don't have to go and use our platform directly and integrate directly. Ideally, our agents will do that for you, and our agents will handle your on-call and everything like that for you.

[15:09](https://www.youtube.com/watch?v=VZzUhELgYk4&t=909s) Um, I also do think that AI agents will be customers. Many of you building out SRE agents and other types of agents, coding agents, should use our platform, should use our tools, um, just like a human would, and, uh, we can't wait to see that.

[15:23](https://www.youtube.com/watch?v=VZzUhELgYk4&t=923s) And generally, I think that small companies out there are gonna be building, built by someone who can use automated developers like Cursor or Devin to get their ideas out into the real world, and then agents like ours to handle operations and security in a way that lets, you know, an order of magnitude more ideas make it out

[15:39](https://www.youtube.com/watch?v=VZzUhELgYk4&t=939s) into the real world. Thank you so much. Um, please reach out if you're building any agents that wanna use us, um, or if you'd like to check out our agents as well.

[15:48](https://www.youtube.com/watch?v=VZzUhELgYk4&t=948s) Um, there's a lot to build here. And if you wanna work in this space, we are hiring more AI engineers and people who are just excited about it. But thank you very much. [upbeat music]
