The 6 Pillars of an Agentic Harness for Production — Varun Krovvidi, Resolve AI

Uh, can everyone hear me okay?

Awesome. Thank you so much, uh, for attending the session. Um, my name is Varun. I'm part of the team at Resolve AI. We are building AI for prod. Uh, basically, we're building agents to run and fix your software. So in this session, we'll cover what are the six pillars of an agentic harness that you'll need in a system that is there to fix and run your software. Uh, before I get into the session, just a quick show of hands just to understand-- for me to understand who's there in the audience. I'm

assuming most of you are engineers. Uh, how many of you have been on call before? Uh, quick show of hands. Awesome. Have you tried to use AI to help you with your on-call? Show of hands. Awesome. For the people who raised their hands, uh, can you tell me that, uh, ninety percent of your on-call work is being done by AI? Uh, can you comfortably say that? Awesome. Yes, that's what we realized when we started the company as well. This is a really hard

problem to solve. So, and in this session, I wanna share, like, our journey, what is the evolution we went through, what are some of the learnings we had, specifically in building such an agentic harness. Before I get in, um, I wanna set context on, uh, what we are experiencing today. So the first wave of AI really was focused on coding, and, uh, truthfully, I think most of the-- most-- for most of us, the way we code has fundamentally changed. And I also wanted to share

context on why that is the case. Uh, firstly, code in itself is self-documenting. It's easy for AI to actually parse and understand what's there to help you out with the next step. Second, inherently, code is very modular, so it's easy for, uh, for AI to actually break it down as chunks and navigate to the next step and also help you extend and build and bey-- build beyond it. Third, and the most important thing, is code is single domain. Um, you do not need to teach

AI multiple domains to reason across them. It's easily able to navigate to the next step, and that is one of the reasons why we have such defined benchmarks as well. You can objectively assess how good is AI in solving this problem for you. But in reality, for engineering, seventy percent of our time is spent on the other side, not generating new software, but rather in pr- in production systems, basically running and fixing software. Uh, and this ranges through a

broad range of issues as well, right? Like operating and running software. Typically, you can actually put it in three broad buckets. Like, one is, uh, think of the healthcare analogy. That's the easiest way I like to think about it as well. One is the regular on-call and maintenance. Think of this as regular small problems that are coming your way. Maybe you need a Tylenol for, uh, some, uh, for some pain that you have. Maybe you need an Advil. So these are the regular on-call style alerts that you need to

deal with. Second category is the incidents, where you need all hands on deck. It's equivalent to a surgery. Systems are shutting down. You need everyone's attention who's, uh, who's involved or responsible, uh, to come in and fix it. And the third category is more like your daily vitamins. So you need to do a lot of things on a regular basis to keep-- to maintain the health of your system going on. So be it things like taking a look at your infrastructure, taking a look at your costs, or maybe some platform engineering

work that you'll do to scale. So these things are fundamentally hard because they're not exactly like code. They cut across multiple different domains, multiple different tools, and that is the repercussions that we are starting to see when the euphoria of initial-- initial euphoria of AI is starting to settle in. And that's what you're seeing in the news right now. So there's a lot more talk about the number of issues maybe AI-generated code is starting to create in production, and there is talk around what are the right frameworks to deal with these, uh, issues in code.

Second, uh, the era of bottomless AI is over. Like, the unlimited AI is no longer there. You can increasingly see conversations about tokens, token optimization, token efficiency. How do you basically improve those architectures, those rigorous engineering architectures that you need to efficiently and precisely use AI? And third, uh, this is the conversation that is, uh, probably made popular by Satya Nadella, who's the CEO of Microsoft, where he started talking about the architecture you

need on top of your regular models for any domain-specific AI to work. So there is a broad classification, right? Like, whenever you look at a generation tasks, you don't have an output or an outcome in mind. Uh, you don't want-- You're not specifying to AI exactly how an image should look like, how a blog should look like, or how a website should look like. So that's where the euphoria sets in. But when you think of a flip side of a task where you have a single correct answer that you want AI to get to, that is when the disappointment starts to set in,

which, uh, in most cases, uh, most people dub it as a last mile, but that is the single longest mile that you have to run with AI. That is why you, uh, you need these architectures for you to focus on a specific problem. Now, beyond all this, uh, operating production systems is also a multiplayer and a multi-system problem. You're not just doing it on your own. You need to pull in SMEs for, for different teams. Over time, uh, all of us have specialized in different categories of engineering. We are put in different teams. Uh, we need to talk to,

to solve-- to resolve any issues, uh, across AI. So, sorry, across your production systems. All in all, uh, our company, Resolve AI, started with a single thesis. So we genuinely believe that all of this, uh, all of this work that you do on production systems like fixing issues or like, uh, running your software on a day-to-day basis should be done by agents in most part. That's one of the reasons why I was asking, uh, can you confidently tell ninety percent of your work or more is being done by agents? And engineers should just be running those

agents where agents take care of like fixing and running your software. So that's why we've built, uh, Resolve AI to-- as three categories of agents. There are on-call agents to fix your software issues that come in on a daily basis, and there are incident agents that'll help you drive incident channels and get you of-- get, get you from a complex incident to a root cause and fix. And lastly, there are ambient background agents that will help you with your production tasks, be it things like deployment monitoring or any of the analysis that you have to run,

or any of the conditions that you trigger where you want a specific investigation to happen. Underneath all this is the same Resolve AI agent architecture. That is what we'll be talking about today, uh, and, and what it takes to get there to build it. So let's start with a question. So why is this even required, right? Like, why can't we just point the biggest model at production data, models are, of course, getting better, the reasoning capacity is getting better, uh, to fix your software issues? It ge-it genuinely demos well. Whenever you focus on a particular use

case and you build your harness on top of it, it shows up with a very good demo. But slowly, as you start to scale across use cases and teams, that is where, uh, some-- that is where very familiar cracks start to appear in the system. So what are the kind of cracks that we have seen as you start to scale in the product ac-along the way, right? First thing, you'll notice that, of course, models will have anchoring bias. This is a very strong thing that you have to deal in your harnesses itself. They're, they're designed to give you like a coherent answer,

um, and first-- as in, as in when new models start to come out with better reasoning capabilities, you also need to maintain the treadmill. Uh, you also need to continuously change, uh, which-- evaluate-- firstly, evaluate which model is suitable for the task and also update, uh, the right model for your use case. And second, and a little bit of an overlooked item. So we always associate a model to a workflow or an outcome. But every workflow has hundreds of different types of tasks. So

there are reasoning tasks, there are deterministic tasks that you go through. There are things like image gen-- image reasoning or you're reasoning across images, you're reasoning across SQL or things like logs. Now, all the models, frontier models that you see around, each of them is good at a specific parti-- or a specific, uh, kind of task. So the orchestration engine that you need to imagine for, uh, moving across these models for a workflow, first of all, needs to keep pace with how models are evolving. And second,

it needs to be able to marry the best model for the best task. And second failure mode, uh, is, of course, the context windows. I know the context windows are expanding, but context windows, context windows are not just the problem, uh, for, for solving su- for, for solving a complex problem like this. The second part is, if you provide large enough context, models start to overexplore. It'll start to come up with hypothesis that don't even exist or sometimes might not even get you to the right answer. Now, if

you supply very limited context, of course, they underexplore. They won't be able to get to-- uh, they won't actually have visibility into what other pathways exist or probably what other potential hypothesis can exist in solving a problem, especially for production incidents. This becomes a huge issue because telemetry is literally infinite. You can start to create as many log lines as you want, as many metrics as you want. And now when you start to marry that with your code and infrastructure, that is when real st- real cracks in system, uh, starts to appear.

Third, and my, and my most favorite one is, uh, defining causal reasoning. So models are there to please you. I mean, we've all come across these use cases where you push the model hard enough, uh, it'll start to agree with you in every different direction. So models are specifically designed to give you coherent answers, but not causality. But production incidents, the only thing you're looking for is a causal chain of evidence, like a detective. Uh, what are the steps that took place that led to a particular incident so that you can actually

fix, uh, the right issue instead of creating a patch. So especially when you're solving a hard problem like production reasoning, you need causal reasoning built into the mod- built into the system, not just coherent answers. And the next one is, of course, guardrails. I won't harp on this topic too much, but all of us have read, uh, some news or the other where, let's say, AI is going and deleting a particular file system or like a database. Uh, so AI, it's-- uh, also this is a feature, right? Not a bug.

Uh, AI might have determined that the cleanest possible fix is just deletion of that particular code snippet. You can't fault it for it. You just have to build guardrails, uh, around the system so that it works in the right fashion. What is the least privileged access that AI can operate with in any given scenario? And lastly, learning loops. Any system that you build should be scalable across your team or your organization in the best case. Every time you're using AI for production incidents, like I said at the starting, it is a

multiplayer problem. You have multiple different teams, like SREs, platform teams, uh, backend engineers or service engineers. All of these people should be involved. And most importantly, all of these people should have the same context. Now, when you add temporal discussions on top of it, like, uh, your AI system should have the same context of a previous investigation in the hundredth investigation, or else you're starting the wheel from the same time again. So these are the failure modes where we've seen the cracks appear. So that's why we've

actually built an agent architecture that answers these same questions exactly. So there are six major parts, uh, for any AI architecture that has to work in a specific domain. The reason I'm mentioning it as a specific domain is models are there for general purpose reasoning. Now, when you want to harness these models into a specific answer, you need to take care of six different pillars. The first one is model orchestration. Like I mentioned, uh,

model orchestration is two layers. One, how do you keep up with the model treadmill with the latest models? And second, how do you match the best model for the task? Is it Gemini for image, image-specific reasoning, or is it OpenAI for deterministic steps, or is it Claude for open-ended investigations? Uh, you need to constantly keep evaluating what is the best, uh, possible model for the task at that point in time. So this is an engine that we built, uh, upfront very early. Second is context engineering. Now,

context engineering is often minimized to like, uh, the database or an execution solution. Oh, are you using a graph RAG, or are you using a knowledge graph? Are you using X, Y, and Z? Those are all implementation details. But what matters as a goal is what is the precise amount of context AI needs to solve that specific problem. And more often than-- most often than not, it's a combination of techniques. You might have to use a graph RAG kind of a solution for AI to start investigations. And when you actually further

go into the next steps, you need to determi-- you need to define very precise tool calls so that you don't burn through tokens very quickly when you're making multiple tool calls in, in form of log queries, in form of, in form of metrics or dashboards or code queries, et cetera. The third one is causal reasoning. Like I mentioned, so this is one of the first, uh, principles that we've built into our system at Resolve AI, where, uh, the eviden-- the root cause is always based on a causal chain of evidence.

What are the exact or precise steps that happened that led to this particular issue? If you're not able to establish that chain of evidence, we have to give out that information with a low confidence level and point the user in a different direction. This is true for any domain-specific AI, and this is true for Resolve AI as well, where we are pointing you in a different direction if we don't see the-- if we don't see the data beyond a particular point. The next one, like I said, is governed actions. This is, uh, quite straightforward. Uh, it depends upon

every team, every different organization, uh, depending on, uh, depending on the access levels or probably the restrictions that you have or the industry you operate in. What are those specific guardrails that you need to define for AI to operate under? Is it read? Is it write? And under write, what are the specific conditions for write? Uh, et cetera, et cetera. And lastly, a learning system. So e-- think of it like this, every interaction that, uh, you're doing with an AI system is an opportunity for learning. So, uh, AI system has to learn not just

from an investigation it is going through, but from how, from how a user is actually interacting with it. Like, am I giving you positive reinforcement? So is this turning into a positive eval, or am I giving you some negative reinforcement? Am I g-- am I guiding you or steering the investigation in a specific direction? Those are different kinds of evals. So that leads into the last piece, which-- where evals is also a very critical step. So this is the starting point and ending point for any agent architecture. So the way we think about evals is in five

different levels, like I mentioned. What is the positive reinforcement you can give? What is the negative reinforcement you can give? And can we trace your path? How did you come up with the solution? Can we score you like an engineer based on that? Uh, how would the b-- how would a best engineer perform this investigation, and how do you compare against it? And on top of it, how are you calibrating yourself as, as AI? How are you giving out that confidence level? So all these are systematic eval interes-- uh, uh, evals that you build into one platform that you constantly have

to push the architecture against whenever there's a new model, whenever there's a new use case, or whenever there's an architecture change.

Cool. This is enough talking, so let me actually show you this in action on how it works in Resolve. Like I mentioned, our platform, uh, is specifically there. Uh, so there are two categories of agents that we work with. One is an on-call agent, one is an incident agent, and of course, the background agent. So, uh, since, like I've, uh, since most of you are engineers over here, you might be familiar with the system. Of course, uh, most of your telemetry or observability, uh, alerts are firing into a collaboration tool like Slack or Microsoft Teams or whatever you might be using. In this

case, what we are seeing is Grafana fire-firing alerts into our common Slack channel. Um, so Resolve automatically picks up those alerts and kickstarts an investigation. What you see here is, uh, it is throwing an error about a log scale, and it starts to provide, uh, what is the, what are the, uh, what is the root cause that it found for the, uh, alert that came in, and also the other contributing factors or other theories it rejected. It gave a quick root cause, uh, and evidence and recommendations. But let's

actually deep dive. So in most cases, like I said, this should be a multiplayer experience. So let's actually go into the UI and see what's happening here behind the scenes. So, uh, it-- the work starts off from Resolve AI when it treats the alert that's coming in as a prompt in itself. So it takes the prompt, and it spins up a bunch of different agents, actually two different categories of agents, uh, to get the work done. So first category of agents are basically investigators. So they are, uh, trained to actually, uh,

go through the investigation just like how an engineer would. Firstly, find out what is the issue, gather more information from metrics and traces, et cetera, and figure out where the issue is happening from traces, logs, and now start to correlate that with things like change events, uh, or any of the new deployments that would have gone in, and your code and your infrastructure. So primarily, it's doing this translation or reasoning across four types of data sources: your code, your infrastructure, your knowledge bases, and your observability platforms.

Based on all that, it starts to give out this, uh, root cause or, uh, this root cause what we've seen. There are m-- So one of the log scale is actually experiencing high f-high failure rate. It trace the issue all the way down to some stale integrations that are still live in the system. The interesting part is it's also showing out some other ruled out theories where generally we would start an investigation. At the same time, there was a GCP outage. So, uh, you would actually, uh, this, this would be a perfect correlation

as well, right? Going back to AI that is designed to give you coherent answers. So a GCP outage happened at the same time, so of course, this might be the issue, uh, that we are seeing. Or there are other traffic spikes that it is experiencing, uh, without missing integration, so, uh, that would be the other thread that you would start to pull on. So let's say if I'm a new engineer, if I'm starting in, okay, Resolve, uh, gave, uh, this as a root cause. Of course, it's-- We have also designed to go operate with the least trust policy. So let's say if I, I can start

to highlight, and it'll start to-- it'll pull me in so that I can interrogate this further. Uh, in this case, let's say, uh, let's ask Resolve, uh, "Is this related to, uh, GCP outage?" So it's giving out the answer here, but there is other, other precise questions that I ran I want to share with you. So there was other deployment, uh, errors that happened at the same time, so that is what I was pushing Resolve on. Going back to what

I mentioned, if you push, uh, an AI hard enough, it'll start to agree with you. We've designed the system to go against it. It'll only base its answers based on the causal, uh, causal chain of evidence that it created. Since it is able to establish that, it's able to give out a very coherent answer. So this, uh, also lets me actually pull in my teammates so that this truly becomes like a multiplayer experience. Let's say if I give one, uh, I can pull in Ranvir. Can you take a look at this?

This can pull in my teammate directly from Slack so that we can collaborate on this virtual war room experience.

This was a quick demo. Like I said, uh, the-- currently... And what I talked about today is just one, uh, axis of the architecture, right? Like, now there is the other axis of the architecture where you just need to productize an AI. Like, if you want to put this in, uh, in front of, uh, any user or a customer, you need to make sure it runs on a robust platform. Those are the other aspects that you have to think through, but this is something that we are al-always familiar with. So that's been my time and, uh, if you're interested to

learn more about, uh, about Resolve AI and which customers are currently using us and how are they using it, uh, please check us out at the booth L twenty-eight, I believe. Uh, and by the way, before you leave, take out those cool bags, uh, uh, that are right outside. Thank you so much.