Always-on agents run production without the on-call tax

[upbeat music] Hello. Hello. Hey, everybody.

Welcome, uh, to, um, this talk always, uh, on agents run production without the on-call tax. Um, my name is Justin Smith, one of the founding product engineers at Resolve AI.

Uh, been in the space for about fifteen plus years in the sort of monitoring, observability, how do you kind of operate production systems space. Um, was at Splunk, uh, for a while, was one of the architects on the Observability Suite there.

Uh, spent a good tenure at VMware. Um, really, really enjoy, like, product design and front-end architecture. How do you, how do you, how do people experience a, a product or a use case or something like that?

That's the stuff I like to dabble in.

Um, but I wanna talk a little bit about, uh, the first wave of AI, and it's, it's been a fun one. I think the first big wave, and I'm sure we've all experienced this, is just how we build software.

Um, but there's some sort of net effects of that. Uh, it's a lot of bigger PRs that are coming through. Um, we definitely see a lot of this a lot more frequently, so people are shipping code at a much, you know, faster rate, um, from developers, and we're beginning to see maybe from even non-developers that maybe don't

actually know the code or what it's doing or the sort of, like, operating principles behind it. Um, but we're getting developer productivity, and that's good, right? That's, that's a good thing that we're all able to sort of produce more and faster.

Kind of, sort of. Um, what we actually found out, um, and this was a survey study done, um, is that seventy percent of the time from an engineer is actually not just, like, is not focused just on writing code.

It's actually spent on actually running the code that is actually shipped into production. Um, maintaining all the, the platforms, scaling the infrastructure, debugging all the incidents and being on call, um, shipping hotfixes, right?

Dealing with alerts, um, updating all the sort of runbooks and operating procedures, restoring services, dealing with escalations, dealing with sort of, like, um, questions from other, you know, teams and things like that.

So really, coding was never the, the, the big bottleneck, right? Um, a lot of it was really around, uh... Thank you, Granola. Um, a lot of it was really around, like, how do we actually run these things sort of in production?

And that's getting harder and harder and harder. Um, AI, it's, it's creating a lot more issues in production as, you know, uh, AI code sort of goes through. Um, it's not clear we have the right sort of, um, structures in place to deal with the amount of kind of changes that are coming through.

Um, unlimited tokens is, is sort of coming to an end. The, the token max, right, they're starting to clamp down. Prices are going up. Companies are getting a lot more stringent on, you know, what's being used, um, for AI.

Um, you know, we need full stack AI. It's not just about the models anymore, it's about the context around the models and what the models can do inside of a specific domain.

These become the, the problem areas that we need to sort of, uh, focus and, and tackle on.

And this is true today, right? So it's, it's creating more sort of, uh, complexity inside of our environment. I mean, the, the reality is that systems have always been complex.

That's why we have, you know, these big tools that can, you know, try to give us insights into these systems. Um, there are multiple teams, there's multiple systems that are all having to work together, and they all have their own, you know, goals that they're trying to deliver towards, but you have organizational goals, and how do you

keep all of this sort of, uh, you know, um, in...

Get rid of this. Um, how do you keep all of this in balance, right? Um, how do you, you know, pull all of this stuff together in a way that actually, uh, actually helps you and, and facilitates your, uh, your organization?

Um, and the answer is, well, you gotta use AI inside of production to deal with, um, sort of the, the amount of increase of complexity that AI is kind of putting into your product or into your system.

Um, and so that's where Resolve, you know, this was kind of our sort of hypothesis from the beginning, was, um, you know, we're going to see an influx in, um, you know, issues, uh, coming out of, uh, coding, um, just the increase in coding, uh, velocity.

Um, there's gonna be more need for kind of AI to actually operate and run these, run these systems. Um, we're, you know, lucky to work with, you know, some, uh, world-class engineering teams that are solving, like, really difficult problems at, you know, crazy scale, um, and, you know, that, that gives us insight into a lot of how

bigger organizations are having to deal with the influx of AI, et cetera.

Um, Resolve itself, uh, hosts a bunch of different sort of capabilities. Um, we have a number of agents that sort of ex- you know, you get to kind of experience.

One of them is just an on-call agent, and this is kind of where we started, right? Um, so for every alert that comes in, um, we can do a triage of that alert.

We can do kind of a full root cause investigation of that alert. Um, and, you know, this is-- For anybody that's had to be on call before, um, you know, on call is a, it's a nightmare, right?

Um, you're often, uh, only going on call every few weeks. Uh, you, you maybe don't fully understand all the changes that have come in. You don't fully understand maybe all the different systems that you're having to interact with.

Um, and so, you know, the complexity is already there, and having an AI agent that's able to come support you and pull that context together is incredibly, incredibly valuable.

Um, and so, uh, that can often grow from just sort of getting a single page into a much larger incident across many different teams, um, across an organization. Um, and we have agents there to come support the much larger activity of, you know, all of this kind of cross-collaboration, et cetera, keeping everybody, uh, in sync and aligned

on, um, where the, where the incident is happening, what the impact of that is, um, et cetera.

And then we also focus a lot on background agents, and this is sort of covering the long tail of, you know, what happens when there's not a fire brewing, um, or, or going on at any one, you know, point.

Um, there's still lots of operational work that you as an engineer or an engineering team have to do, and a lot of ceremonies of, you know, h- passing contot- context off, or dealing with kind of one-off issues, or kind of having to scratch that itch in the back of your head of like, "Is that part of the

system okay or not okay?" Um, and you're constantly having to sort of balance across all the diff- these different things. Um, underneath all that, you know, the-- We have an agent, uh, architecture, um, deals with models and context and reasoning and, and actions.

Um, learning is a, a... I'll sort of like half pause on that one. I think, you know, some of the biggest issues that we've seen, it's, it's not that a, a model by itself is, is not smart or whatever.

I mean, models have gotten incred- incredibly capable over the last, um, year, let's say, right? But especially over the last like six months or so. Um, but the idea of understanding, like truly understanding your environment, um, and the way that your services interact and where the hotspots are, keeping track of all of that sort of understanding is

incredibly difficult. But it's incredibly important for any model to be successful at, you know, the task that it needs to do. It has to have an underlying sort of learning system to be able to capture that knowledge, um, and that sort of understanding of how your system operates.

Um, so we spend a lot of time thinking about, how do we have systems that not just can understand your environment at any one point, but grow as, as your system evolves?

'Cause again, your system is evolving faster and faster. We need to keep up with learning about what's the current state, um, what's the current sort of causal chains that we need to be sort of keeping, keeping an eye on.

Um, and then of course, all the enterprize, uh, enterprise stuff sort of underneath. Um, and so this is that same view kind of, uh, packed out. Um,

come on. Okay. Um, but so today-- So we do a lot at Resolve, um, the on-call and the incident stuff. I'm gonna focus a lot, uh, more just on the background agent, um, stuff.

So how do we deal with the things that maybe aren't sort of immediate fires? Um, if you have questions about the immediate fire stuff, um, we have a booth down in the expo.

Please come check it out. Uh, our team would love to demo to you, et cetera. Uh, but today we're gonna focus on the background agent.

So kind of a little pop quiz. Um, feel free to raise your hands. Is anybody using agents in part-- as part of your daily workflow? Maybe outside of the coding.

I'm assuming everybody's doing coding agents these days. Is anybody doing like actually running sort of, um, agents that are sort of helping in other ways? Okay. Oh, decent amount.

Any good examples? Any fun stuff that anybody has? You can just yell it out. Meeting review. Meeting reviews? Yeah, meeting reviews. I just had my granola show up and-

Market research. Market research? I do a lot of-- [laughs] Let's, let's have a good conversation about it. Yeah, yeah, I do that all the time. Any other ones? Maybe one more.

Therapy. What's the one? Therapy. I can't hear it. Therapy. Therapy. That's a fantastic one, actually. We're-- We, we are humans here today. This is very important. Um, that's actually-- [laughs] That's a very good one.

Um, okay. So, so people are having some stuff, uh, going on. Um, so, you know, and this kind of recaps a little bit, uh, again. A lot of production work is not about, is not-- There's not a sort of big ceremony that everyone is focused on for the, the type of work that we have to do.

Um, on-call, you, you've got a page that goes off, you know somebody's gonna receive that. Incidents, you create a bridge, you invite people in, that's great. Um, but there's just a long tail of other things that we are accountable for that doesn't have sort of a thing that's gonna show up in your sort of job description of

like, "This is what you're gonna be, you know, responsible for." Um, watching deploys that go out and make sure that they're actually getting out, um, healthy. Um, a morning report or incident digest of just like, what's the state of my system today so that we're all on the same page?

Um, "Hey, that P99 drift kind of came back. Is somebody looking at that or not?" Um, and you know, this is pulling people in to, to try to like figure out what's going on.

This may not be paging, right? 'Cause we don't-- uh, we're not gonna alert on everything. Um, produce the capacity report, right? Like are we tracking okay, right? This is maybe a company goal this, this quarter.

Are we tracking against that? Somebody's gonna have to be responsible for doing that. Um, the recurring health check and just kind of checking and making sure things are kind of running okay and not waiting for a customer to com- complain first.

So this work doesn't have like an obvious like, "Oh, this, you know, this now needs to go be done." Um, but it's work that we end up having to do.

So what is a task? A task is just execution and the context to understand how to actually execute the task. Um, execution, um, is very, very important. It's, um, understanding, uh, what to do and being able to execute that.

Um, maybe having access to the tools, et cetera, right? Um, obviously very important to do. Um, but we think the production context is just way more important. Um, because it's one thing to go check a dashboard, um, it's another thing to say, "That metric smells off."

Um, and the execution is-- can load the dashboard. It's the production context that's gonna say, "This feels wrong, and I don't know if I can even explain why it feels wrong.

It just feels wrong, and I wanna dig into the next layer of, of sort of understanding of that." Um, and so really, if, if we start talking about background agents and being able to kind of perform tasks, um, you need both of these.

You need the execution engine, that's great, um, but you really need that production context that tells you, is this important or not important?

Uh, so every background agent, um, you know, there's a, a few different principles that, that we like to think about, um, with our background agents. When does it work?

How does it work? How does it know what to go do? Uh, when does the agent work? Um, it can work in a bunch of different ways. It can just do it on a schedule.

Maybe this is the morning report, et cetera. Um, just kind of do some summarization for me kind of on an ongoing basis. Maybe it's a weekly event, right? Uh, we do an on-call handover, uh, every Thursday, and so the-- a lot of, um, the work that our agent does is sort of prepare.

Like what, what are the kind of interesting trends from the last week that the next on-caller needs to sort of understand as they pick up the rotation? Um, event streams.

So you know, there's lots of systems that will sort of push events as kind of key things happen. Um, so deployments go through a C-- a CI/CD pipeline. Um, there's other sort of, uh, Slack-based, right?

We get a lot of, uh, Slack things, messages coming through, um, et cetera. Um, and these are things that we can sort of pick up and trigger and say, "Oh, if this event happens, um, let me sort of understand what that event is and go do some work."

Um, and then message-based, so I can just tell it, "Hey, go do some work," and it will go do some work. That's fantastic.

Um, how does it run? It always runs. It's in the cloud, um, so if you close your laptop, it's okay. Um, runs inside of a sandbox, so it has kind of a file system underneath it.

Um, this, uh, allows it to sort of self-organize a lot of its work, et cetera, as it's doing, uh, doing things. Um, and then obviously back to the learning loop, right?

So that idea of knowledge and sort of a memory system underneath that, um, to really understand your systems, and as it's doing a task, able to sort of reflect on that task and, uh, you know, do a better job next time.

Or the things that it learned from one task, it can sort of apply into a different task, um, because again, this, this sort of shared, uh, sort of knowledge system, um, works across all the different tasks that we have.

Um, so how does the agent know what to do? Um, it has a task system. It can pull in all the skills that you have in other systems. That's fine.

You can connect those. Um, and it's got, obviously, the integrations that it's gonna plug into. Um, so let's talk a little bit about what types of things you, uh, can hand over, and we've got four sort of workloads that we're gonna talk about.

But i- if you think about the previous couple slides, these are sort of very basic primitives that we've built into the system. You can get very creative. We've-- uh, we have a number of people inside of Resolve that have gotten very creative with the type of sort of background activities, um, that things, uh, that, that they, that

they have. Um, so I want you to use these as kind of... These are things we've seen be very successful, um, inside of Resolve, but also with, uh, uh, you know, a number of our customers.

Um, but, you know, sky's the limit, um, and, and you can get really creative. So deployment monitoring. So this is a big one. You know, uh, any change inside of your environment is an opportunity for something to go wonky.

Um, and so, you know, having a-an agent that's able to watch as all these change events come in, just to do a sanity check of is everything stable is incredibly, incredibly important.

Um, and you know, a lot of people have a decent CI/CD system. I mean, this is, like, tried-and-true stuff that we've had as an industry for quite a while.

Um, but we, we notice a couple gaps, you know, from-- in most of our customers. Um, you know, typically the checks that it does are good. They're good baselines, but it's not exhaustive.

Um, based on the type of changes that are going and et cetera, there's certain signals you'd wanna s-watch or not wanna watch. Um, and so every rollout is a bit unique.

Um, oftentimes you have change systems that you're not piping through a CI/CD system, uh, like a feature flag or maybe some infra changes that might happen, um, which maybe don't get any monitoring at all, and you're sorta just trusting that an alert might fire and an on-caller will wake up and say, "Who changed what?"

Right? Um, and so d-deployment monitoring is, is actually a really big use case, um, that we suggest people sort of, uh, go through, and I'll show some examples of that in a second.

Um, schedule health and anomaly checks. So this is just sort of the ongoing periodic checking of some of your systems. Um, and you know, this is maybe something where it's like, go check, you know, sort of my general dashboards on a f- routine basis, maybe every morning just kind of do a casual check just to make sure

there's nothing kind of weird from last night that I might need to be aware of. Um, but, uh, this can also just be sort of a time-based thing. Like, I made a change in part of our system.

I'm worried about this, you know, uh, you know, third-party service that I'm kind of interacting with. Let me just kinda set an agent to kinda watch that maybe for the next week, just to make sure everything is kind of stable, and then that agent can sort of, um, you know, stop its job.

Uh, operational reports and handoffs, I talked a little bit about this. Uh, these are the sort of ceremonial things that we might wanna do just to, you know, spread information, summarize things, kind of bring things to the forward.

Um, and then a first responder to engineering questions, and this one's kind of fun a little bit, um, because the trigger for this is actually just a Slack message.

And I will say, um, one of my biggest, uh, like, let's call it responsibilities, uh, as an engineer is watching all my Slack channels and trying to make sure everyone's kind of happy, um, and, like, that nobody has any burning questions or anything like that.

Um, and so I can be sort of heads down trying to build something, um, and then, you know, the, you know, uh, eventual sort of Slack notification comes in that, like, this channel, somebody asked this sort of kind of important question, um, and I just need to jump in there and, and try to provide context, et cetera.

It's not hard work. It's not hard for me to go answer questions, but it's disrupting me, and if I don't go answer it, um, they won't get an answer for a while.

And what we found is, like, our agent actually has access to a lot of information that people ask questions about, at least this is true internally. Um, so we actually have an agent that can watch all of these sort of critical channels, um, and determine whether it has enough sort of confidence to answer the question or not.

Um, and one of the fun things is, like, it, uh, the-- our agents have access to, like, Slack DMs and things like that. Um, and so you can have an agent that basically will DM you to say, "I think I know the answer to this, but I'm not sure.

Can you confirm this for me before I r- you know, respond back?" Um, so this kind of emergent behaviors gets kind of fun and interesting as you just kind of build these things out.

Okay, so I'm gonna flip over and hope all of this works.

Um, cool. Um, so, uh, let me see if I can find the one that I wanted to show. So this is our, uh, sort of, uh, demo application running on our sort of demo sort of Slack environment.

Um, and, uh, what I wanted to show off was some of our deployment stuff and talk a little bit more about, um, what's kind of going on under the hood.

Um, so this is, you know, sort of fake environment, um, just to kind of showcase some things. Um, so here, um, anytime somebody posts a sort of GitHub tag, um, our agent's gonna sort of see that and say, "Oh, I, I should-- That's a release.

Is that a release? Yes, that is a release. Um, let me go watch that." Um, but it's not just gonna watch it. Um, it's gonna do something that's slightly more intelligent.

Uh, 'cause like I said before, everybody kinda has a CI/CD system. It will do the sort of standard checks on, on, you know, certain KPIs. Um, but what the agent is able to do is actually look at the changes that are going in- Understand what telemetry might help us evaluate whether those changes are, you know, good or

not good or like, are putting the system in an abnormal state. Um, and build a sort of customized plan that it's going to check, uh, just for this specific release.

And this is why I g- kinda go back to like, you know, our goal is not to sit here and say, "We're gonna replace an entire CI/CD pipeline." You've spent time organizing that.

But this can sort of patch a lot of, you know, uh, parts of your system that may not be as robust as they should be, and it would be great if you had a single engineer just focused on like watching all the things on every release, but that's really expensive.

There's a lot of cognitive load. You'd rather have them doing other things. Um, so now the agent can come and actually do a lot of that sort of dynamic understanding of, this is the change, um, so I'm gonna sort of check for these things.

And so here, um, the checkout replaces currency service. Um, you know, we'll monitor the checkout latency and the error rates. Well, let's take a look at the Kafka pipeline 'cause that sort of involves...

This is the sort of causal chain I want to sort of say I wanna make sure is, is healthy. And it'll check that, and it can check it not just once, um, but sort of o- on an ongoing basis.

And, you know, none of this is hard-coded in. It's not like, oh, let's just wait for 15 minutes and then try this again, and then we'll be done. Um, again, the agent has a bit more autonomy and, and you get to guide it a bit on how you, how much autonomy you want it to have.

Um, but it could decide, I wanna wait for another hour 'cause this type of issue might only hit every f- you know, every so often, so I really wanna spend a little bit more time focused on this.

Maybe I'll come back in three days and say, "Is this deploy still kind of healthy? Are we seeing the, the change and the effect that I expected to see, um, out of this?"

Um, so this is the type of, of thing that we can bring and, and this again, works for feature flags, for interchanges, um, sort of any sort of eventing system that you can think of.

The sort of on-call handoff reports. Um, let me scroll up just a little bit. Um, so this is just a summarization of all of the work that was done, you know, uh, over the last day, um, that the agent is kind of just summarizing up.

Um, and this one's a little bit verbose. Um, but, uh, you can see a bunch of different sort of investigation summaries that we did, um, some notable changes, et cetera, work, uh, work completed, um, et cetera, critical open.

Um, you know, I guess that's the on-call handoff. Um, but one of the nice things, and I don't know if I'll be able to watch it go all the way, but, um, you can always just come back in this thread and, you know...

This is too verbose. Verbose. Make it shorter. Um, and I'm not gonna be able to-- unfortunately, no time to watch this actually go, but this works. The, the agent is able to update, uh, its task underneath, um, and able to sort of give you the answer, uh, like update so that the next time it fires, um, it's

not going to, uh, be as verbose. Um, and I can tell it explicitly what I want, et cetera. Um, I was just kind of giving you an example. Um, this is kind of the more fun one.

Um, so, uh, you know, here I'm just like posting different problems. Um, I'm not having to know that Resolve exists. Um, I don't have to like, uh, @mention Resolve, whatever.

Um, I've set up this agent to sort of passively watch this channel. Um, if you see something that you think you have an answer for, that somebody's kind of, uh, you know, ch- digging into, um, go ahead and, uh, uh, respond.

Um, otherwise don't. So, you know, here's a message that I posted, um, that it's decided I don't need to respond to this. Um, so again, very kind of flexible system, um, that can kind of adapt to a bunch of different things.

Um, uh-oh, where'd my slides go? We have, uh, in the UI, there's a bunch of stuff that you can do. You know, you can always go down and inspect all the different tasks, uh, that you have and, and view their reports, view previous runs.

You can see all the work that the, that the agent has done underneath, um, to sort of accomplish that task. So you get a lot of visibility into what the agent is doing, but we think the surface area being where you live, right?

So Slack is this like kind of, um, or, or MS Teams, if you're on MS Teams, as this kind of first-party experience to sort of integrate the, the agent into, um, is incredibly important.

Um, and so, you know, how would you get this stuff, uh, sort of set up? Um,

it's really just through talking with the agent. Um, and so here this is me sort of saying, "Hey, I wanna do a new recurring health summary for my team."

Um, so the agent's going to, uh, take a look at my environment. It's gonna explore my environment a little bit, um, and eventually likely come back and ask me a couple questions about, um, what I want to see, what kind of report do I want, how, how verbose do I want it, um, et cetera.

Uh, the agent's gonna go ahead and do all that, and it's gonna set up that sort of initial thing for me so that I can test it out, make sure it's working, um, and then share it with the rest of, uh, with the rest of my team.

In the interest of time, I don't think we'll get to this, but come by the booth and you can see more. Um,

cool. Um, so that, that's background agents and a- again, I, I sort of lean back on be creative, right? Like everyone has unique work. I mean, as, as a company, we believe that every, you know, every company is a unique place.

That's why we spend so much time on our knowledge system, et cetera. Um, truly understanding your, you know, what your environment looks like, what your needs are, et cetera, um, that begins to t- kind of tell you where the, uh, where the biggest benefit from having these agents begin to pick up work, um, would be.

Um, also just to call out, if you have an agent harness, if you, you know, internally if you're building your own, um, uh, everything that I showed is accessible, uh, through kind of MCP servers, et cetera.

Um, so you can really graft Resolve into kind of any, um, system that you have, um, as just kind of an extension of learning to kind of do deeper work or to sort of pull production context, uh, a bit more efficiently or, you know, even augmented with all that learning stuff that we've done.

Um, and then obviously bring your own skills, um, along for the ride. Um, don't go duplicate a bunch of stuff. Um, so really the, the biggest things to take away, um, cost to operational work.

It's not navigating, uh, you know, it's, it's, it's not just in the task execution, it's in the environment complexity, right? Um, that's where the biggest issue is going to happen.

Um, background agents, they run on schedules, they run on triggers. They're very composable. You can sort of graft them into lots of different use cases. Um, it's fun to see people explore that.

Um, and, uh, yeah, um, you open Resolve just to kind of see what the top findings are, et cetera. Um, but you know, ideally a lot of your interaction is kind of in the places that you're already kind of doing work.

Um, so, um, if you have any questions, uh, you can find me down at the booth or, you know, just meet me out in the hall. Um, but thanks for coming.

Appreciate it. [clapping] [outro jingle]