Building Reliable Agentic Systems

[upbeat music] So for this talk, uh, basically what we thought was it'd be cool to give kind of practical examples and lessons of, uh, problems and solutions that we identified while

building, uh, the Droids. Uh, and so for context about Factory, uh, our mission is to bring autonomy to software engineering, and what that means concretely, we build these products that we call Droids.

They are, uh, autonomous systems that are applied to different stages of the software development life cycle. Think code review, documentation, testing, all the way to end-to-end coding tasks like a refactor, a migration, feature work.

Uh, and so each of the Droids, uh, has, like, separate cog-cognitive architectures which are mapped to the tasks at hand. And, uh, in particular, the, uh, you know, Droids like ReviewDroid or, uh, the DocumentationDroid, which handle processes which we think are kind of more on guardrails, are pretty different from something like the CodeDroid, which is able to

take nearly any natural language task and make an attempt or complete, uh, a coding task that is associated with that, like, request. Uh, and so the idea for this talk is let's just start describing what we think of as an agentic system.

Uh, this sy-- like, agentic systems have a lot of different interpretations, a lot of different definitions. We think that, uh, there's three characteristics which are kind of most, uh, representative of an agentic system.

The first is planning. I think you've probably seen a lot of this in this track. The idea that the agentic system can make decisions about one or future-- or, or many future actions that it's going to take.

Uh, decision-making. Some people call this reasoning. Uh, I think that on a lot of these systems, just the ability to make a decision, have some criteria and some algorithm that's associated with making that decision is kind of critical in order for agentic systems to take on more general or broad tasks where the decision space is very wide.

Uh, and then you have environmental grounding. Uh, and so, you know, a lot of, uh, systems have planning, they have decision-making, but I think the existence of an agent within an external environment is very critical to understanding some of the unique properties of agentic systems when you're actually implementing them.

Um, so being able to read and write to these environments is, is a, you know, a, a critical part of this process. Uh, and so... Oh. Let's talk about planning first.

Uh, so the first idea that we kind of encountered is inspired by control systems and robotics. The idea of the pseudo Kalman filter is as you are working through a plan, what happens is you tend to notice that your agentic systems will lead-- will be led astray.

Their reasoning will change rapidly, and as you kind of iterate through especially long plans, you can imagine that in order to migrate even a small section of a code base, you might have literally hundreds of steps in a process.

Uh, and so inspired by a lot of the folks that on our team come from backgrounds in self-driving and robotics, uh, the pseudo Kalman filter is basically passing intermediate reasoning.

And you can get pretty complex about how you modify or share the intermediate reasoning through the different plan steps, but as you pass that intermediate reasoning through the execution of the plan steps, uh, it allows the, like, individual decisions that happen on the plan to slowly converge towards at least consistent reasoning.

Uh, the kind of core issue with this is it also facilitates error propagation. A simple mistake, especially early on in the plan, can lead to, like, you know, strong downstream effects.

Sub-task decomposition. This is pretty well known. Uh, even ReAct, uh, like some of the earlier agentic, uh, action systems, uh, have some form of sub-task decomposition. Uh, but you know, what we found is that experimenting with different forms or structures of sub-task decomposition in our planning process, uh, has led to a lot of, like, pretty interesting downstream

positive effects. Uh, in particular, it-- when you basically increase the resolution or the fidelity of your sub-tasks in a given plan, it gives you more fine-grained control. It allows you to define kind of the action space a lot more clearly.

However, the risk is you're introducing a lot of decisions for the LLM to make. The more, like, small, tiny tasks you introduce, the harder it is for the system or the agentic system to decide what's right.

Um, then you have model predictive control. This is again, you know, not a new idea, not invented by anyone here, but the idea of evaluating outcomes of your sub-tasks in your current state, uh, and enabling kind of adaptive replanning based on real-time feedback that occurs during the execution of the plan.

So, you know, if you have, you know, rapidly changing environment or information, this can occur especially in, you know, situations for us where there are other humans actively engaged in either a development process or some other workflow while your agent is executing perhaps a long-running workflow.

Uh, and so if you have, you know... You, you can honestly, like, find a lot of information about these techniques and kind of reason around, you know... I, I-- You don't have to necessarily jump into the true, like, Bayesian statistics, predictive modeling stuff.

I think a lot of the, like, general ideas of replanning, taking in the trajectory information, and making sure that you keep that up to date, uh, is honestly enough to see some, like, pretty solid quality improvements on your agents.

And then finally, uh, explicit plan criteria. So I think you actually kind of see this pretty often for very simple agents, and then people start to remove explicit plan criteria in order to increase generalizability.

Uh, but, uh, what we found is that clearly defining kind of successful structures or at least successful initial states for plans, uh, can lead to, uh, very strong kind of downstream effects.

Uh, and, and, you know, this can be done in a lot of different ways: instruction tuning, uh, you know, few pro-- uh, few-shot prompt examples. You can validate, like, your plans with different kind of hard-coded logic.

Uh, you know, ultimately, all of this is about error reduction and keeping your, like, trajectories as successful as possible as long as possible. Uh, but it's difficult to build and scale these 'cause basically what I'm recommending is that you hard code a lot of logic into your system.

Uh, I think that this is like a general idea that people maybe, you know-- Depending on your domain, this may seem appealing or not appealing, but ultimately, we're probably not building AGI like tomorrow or next week or in the next six months.

And so I think taking some of the lessons from the kind of like symbolic era of AI is useful if you have like a domain challenge that you wanna solve in the next three months.

Uh, for us, you know, we think about delivering real value to customers today, and that means that a lot of what we do is thinking about things like explicit plan criteria, hard-coded logic, uh, and I think that a lot of folks may not, like, be open to admitting that, but I think it's a very important part of

the, the process of building these kind of agentic systems. Um, so now talking about decision-making, and I know this is also meant to be kind of like a word vomit, honestly, because I think hopefully you guys are like building agents and you care a lot about this stuff, and it can spark or inspire some ideas in your

own systems when you're working on them. Um, so first for decision-making, consensus mechanisms. Uh, there's a lot of these. Self-consistency is a very popular one. Prompt ensembles, cluster sampling.

You know, basically, the more inference at runtime that you run and the more that you can like build a clever way of selecting ideal or optimal samples from those like many number of inferences, the higher the accuracy.

It's just gonna cost more. It may introduce longer inference wait times if you're not parallelizing, uh, but we found this very important, uh, to getting high-quality decisions that are consistent.

Uh, next you have explicit and analogical reasoning. These also have a lot of different names: chain of thought, uh, checklists, chain of density, analogical prompting. You know, basically you want the system to explicitly outline its reasoning or decision-making criteria to reduce the complexity of the decision-making process.

So when you have, for example, a checklist of things that-- Let's say you wanna make a decision about left or right. If you just create a checklist of what constitutes a reasonable left and what constitutes a reasonable right, your system will do better at choosing left or right.

Um, obviously though, this introduces challenge if you have like very broad domain stuff, uh, and-- or you like have a decision which has a huge action space, in which case techniques like chain-of-thought reasoning, uh, you know, chain of density, uh, honestly, like there's like galactic tree of mega brain thought that like...

There's a lot of these like techniques that exist and I think, uh, you know, exploring these is super worth it, uh, for improving performance. Uh, fine-tuning, this is kind of a cop-out answer, but I actually think it is pretty valuable once you really get into having data, uh, for specific decisions that you wanna make.

It may just be true that the best thing to do is to spend a weekend pulling the latest open source model, generate a bunch of training data with a high-quality model, validate it with a bunch of your team members, and just train like or fine-tune a model.

Uh, this is expensive and it locks in the quality of your system. Like a lot of the like benefit of relying on basically being able to sample from different models is every time a new state-of-the-art model comes out, your system gets better.

I think that's a huge benefit. However, for certain decisions, especially those that are really out of distribution, uh, fine-tuning is a pretty effective way, uh, to make a good decision.

Uh, and then simulation. So simulation of decision-making is super tricky. Uh, this is definitely gonna be very domain-specific. If you're working with software development, simulation is luckily kind of built into like the thought process here.

The ability to execute code, the ability to reason through code trajectories, uh, is super doable. And so for us, simulation makes up a huge amount of how we think about decision-making processes.

So sampling multiple decision paths, uh, simulating them both with real and s-- and kind of like LLM imagined execution paths, uh, techniques like language agent tree search, which basically says amongst this simulation of decision nodes, let's implement like a fancy Monte Carlo tree search algorithm and, uh, y- you know, it's k- it's kind of like Monte Carlo

tree search. It's not exactly, but you know, let's do fancy algorithms to decide where we want to move based on the simulation results. Uh, and so you can do all these things and ultimately the, the kind of core goal here is y- you wanna have some evaluation of whether or not your system makes good decisions when you

want it to make good decisions. You can take all of these techniques, apply them, and then see what's working, what's not working, look at the pros and cons, and then adapt.

Um, so that's decision-making. Now we have environmental grounding. So the first, uh, you know, this is oftentimes called tool use. I think that that's like an e-equivalently like valid way to describe AI computer interfaces.

But I think the idea of like building these is kind of the interesting challenge. So there's dedicated tools that are very common. Like you can like Pull down LangChain and start using a calculator, a sandboxed even Python script execution.

Like, I'm pretty sure you can clone an open source repo today that implements Claude's artifacts, like, in your own, like, environment. It's awesome how great the open source community has pushed tool use.

Uh, but I think a lot of the kind of edge of tool use and where you start to move towards building custom AI computer interfaces is when you need workflows or trajectories which don't exist with the kind of known tool set that you have today.

So if you have a calculator, you can definitely run a calculation. But what if you know that very consistently you are going to use a calculator, take the output of that calculator and maybe pass it to another system, and then you're gonna read them and parse the logs and then take the output of those parsed logs and

do some, like, additional transformation? If you're going to consistently do that and you want your agent to just come up with that every single time, then that's probably not the right mental model for what, like, the agent should be thinking about or making a decision about.

Uh, instead, what you wanna do is say, "How can we build this tool and then build the interface to the LLM for this tool so that you can kind of streamline those types of, uh, actions that you wanna take?"

Uh, and so this is honestly really effective, especially in domains like code where there's tons of dev tools. Like, I mean, there's infinite numbers of tools that people have made to be really good at developing software that are honestly really effective, and they just have kind of weird interfaces.

Maybe it's a CLI. Maybe it's like Command+Shift clicking to go to definition on your VS Code. All these things that exist in, like, the world need a way for an LLM-based system to invoke and reason around the outputs.

Uh, so we spend a ton of our time building these AI computer interfaces. Um,

then I, I kind of alluded to this, but, uh, designing explicit feedback processing is I think a very critical step of grounding your agent in an external environment. Uh, in particular, if you have, for example, logs I think is the great-- is like a great example of this.

Uh, your CI/CD probably outputs an enormous amount of data about, you know, all the tests that ran, all the debug statements, all this kind of garbage that you don't care about.

If you know that you're going to need to process that data and use it at some point, it's definitely good to build into the system explicit kind of paths or decisions or tools that take all of this feedback, process it, and then maybe even do like a step which is solely about LLM reasoning.

Uh, I guess the example here is you have your logs, you parse through it, and then you say, "Well, what is-- You know, I know the LLM is going to want the failing tests, so let me get the failing tests and also provide a brief ex-- like explainer of what the rest of the logs were doing."

Uh, that type of, like, feedback processing is pretty critical, uh, into making these systems work really well. It also applies to-- I'm talking a lot about external tools and external mechanisms like the LLM or agentic system interacting with something else.

There's also the LLM or agentic system interacting with itself. And so as you, as the agent kind of reflects and reasons about error trajectories, processing that feedback in a meaningful way and passing it back to the LLM, uh, is super important because a lot of the time the LLM is not great at actually criticizing its own action.

It's just good at listening to you tell it to criticize itself. Uh, and so that we found to be very important. Uh, bounded exploration. So you definitely want your agents to be able to gather as much context as they can about the problem space.

As you introduce additional ways of gathering this context, probably at the beginning of the problem, not always, but probably, uh, there is a huge benefit for co-- uh, models that can handle this, like very long-context models.

If you use Gemini Pro 1.5 or, uh, Sonnet 3.5, or honestly, there's a lot of models now that have very long context windows. Um, you can continue to, you know, include information, but at a certain point you need to kind of say, "Let's jump into the problem."

And what we found is that finding the right balance of bounded exploration time is very difficult to actually know in advance, uh, and honestly requires a lot of evaluation.

Uh, and so if you can allow your agent to gather this context, have longer exploration phases, collect data, decide which data is or isn't relevant, and kind of begin the problem with the maximum likelihood of success, then that is, like, one of the single most important things to having a successful trajectory, uh, in the, in the kind

of extent. Um, and then human guidance. Uh, so this is also a little bit of a cop-out answer, but I think it's important to, like, describe. You know, as you have your agentic system interact with humans, you want to decide when in those interaction patterns do we wanna ask the human to interfere or when do we want

them to provide guidance? So as-- You know, with careful UX desi- design and, like, interaction design, I, I think that this can be extremely effective at allowing your systems to go from thirty or forty percent reliability to ninety or a hundred percent reliability.

Um, but you know, ultimately this is balancing autonomy with human oversight, and so it's a trade-off that you have to make. Um, anyway, so those are all of the, uh, things that we learned at Factory.

There's many more, and if you'd like to join us, we are hiring AI engineers, software engineers, go-to-market, everybody. So please give me an email. [upbeat music]