Beyond the Harness: A Journey Towards Adaptive Engineering - Rajiv Chandegra, Annicha Labs

[lips smack] Hello, I'm Rajiv. I'm a practicing medical doctor here in London, uh, and been doing some AI engineering for the past few years. Uh, particularly interested in real-world application and really the future, which is multi-agent, multi-human, multi-institutional collaboration, uh, through a company called Annicha Labs.

Uh, and really the premise of my talk today is to introduce a new design philosophy for the future of AI engineering, which is beyond a harness or beyond a fixed static harness and towards adaptive engineering.

This slide basically summarizes my entire talk. Um, and really it's, uh, about exploring what the current AI engineering paradigm is, which is to use fixed harnesses, um, to steer agents.

Um, so we either use or build an existing harness like Pi, Claude Code, Cursor, Codex, whatever it is, ahead of runtime that remains fairly consistent throughout the engineering process and is largely constrained, um, with fixed roles and tools and sequencing, which again, ahead of the process, beginning of the process, we can tweak.

Um, and it essentially produces very reliable and orderly work, and it's great for most engineering problems and ideas. Um, and so your role as an AI engineer right now is to use a harness and steer agents.

But the future's gonna be different in at least two ways. Number one, models are gonna become so powerful that existing fixed harnesses are constantly gonna become outdated. And number two is there's gonna be a huge exposure to the real world.

Right now, a lot of AI engineering is software-based. Uh, it's within behind a screen. Um, but actually as things get more powerful, as models get more powerful, we're gonna have exposure to the real world where I, I'd argue that this is where the real challenges are.

Um, and the real world is dynamic and messy. You know, it's full of multi-agent, multi-human, cross-institutional, and really touching the physical world, which means that a fixed harness is quite brittle.

And so that's where we enter what I coin as adaptive engineering, where you actually allow the harness to emerge and adapt mid engineering to find its most optimal position and structure.

Uh, and essentially the engineer's role is gonna be to design some of the constraints, which are kind of the rules of play, um, and really allowing the harness, um, to emerge, stabilize, change, and eventually dissolve, um, as you go through the runtime of engineering.

So let's look, uh, firstly at the current, uh, paradigm for AI engineering. Um, and it's essentially, um, where a harness, um, serves as the primary method to guide an LLM, which is typically stateless, um, and make it into something quite useful.

And there's a couple of aspects of a harness which I won't go to in detail. Um, but essentially things like system prompts, which is established by the harness vendor, uh, meaning that users can't actually modify it.

And this essentially, um, tells the harness what it can and can't do. Uh, you've also got the agents.md files or the claude.md if you're using Claude Code, uh, which is loaded into every context window at the start of a session, uh, tool calling.

And essentially then you have, um, the genesis of agents, which are specialized entities that have been harnessed, uh, granting them unique capabilities, um, and, and allowing them to differentiate from other agents.

And these capabilities are typically manifest as specific skills when it comes to role, typology, their capabilities, the rules. Um, and that enables them to deliver targeted outputs and specified outputs, whether it be code, documentation, issues, handoffs.

And, and most recently we've h- seen the genesis of kind of loops. Loop engineering has become a thing. Um, so that eventually we reach some sort of outcome,

uh, whether it's a feature, uh, a problem solved, which is eventually reviewed by a human. Um, so what you see, uh, from this is that before running the engineering process, the entire harness has been predetermined or pre-engineered, and that's harness engineering.

And, uh, it's been remarkable at getting the most out of the latest models. [lips smack]

Um, and just sort of stepping a bit back, um, defining exactly what a harness is. Essentially the model is the engine and the harness is everything built around to make that engine useful.

Um, there's many different types of harnesses. I mean, there's hundreds, uh, actually, but I guess the most common ones are, um, you know, CLI coding, um, such as Claude Code, Codex, Pi.

You have IDEs like Cursor, uh, multi-agent orchestration through LangChain and, and things like Hermes as well, Cline and Goose. And, and these are all differentiated based on number one, properties, uh, and how they're configured.

But actually more importantly, uh, number two, um, based on some sort of design and engineering philosophy that they believe in. And what makes them so great and so distinct is that they are opinionated to a lesser or greater degree, um- And they allow for, I guess, different use cases.

Um, and depending on your temperament as an engineer, you'd pick one over, uh, the other.

But what we do see in all of these is that everything is predefined. You know, you can customize them, of course. Like, for example, the Pi Agent, it's minimalist and it's maximally extensible.

Um, but everything is predefined, you know, the roles, fixed outputs, sequencing protocols, how memory is created. Uh, and so I want to be absolutely clear here. I don't mean that you can't customize harnesses.

Of course you can. But the customization occurs ahead of the engineering runtime, uh, and not mid-engineering, and that's largely directed by us as humans. And for someone like me who started AI engineering about two years ago, it's probably the most powerful and predictive, uh, predictable method.

Um, and it's got really three payoffs. It's reliable. Um, you know, the same input leads to a similar set of outputs. Obviously, there's going to be some variation. It's auditable.

Um, you can inspect exactly what changed and when. And there's some sort of linear causality where, uh, you know, if something breaks, you can actually trace it back to its source.

Um, this is like a factory line though. Like think Taylorism for AI. Just like a factory assembly line, every station is engineered in advance. Um, each agent has one job, a fixed place, a sequence, and maybe a defined handoff to the next agent.

I mean, there's clearly some incredible uses for this and, you know, factory engineering gives you, you know, accuracy, speed, reproducibility, behavior that you can certify, and it really shines in these kind of closed deterministic systems where you have a well-defined product, uh, problem, and you know the kind of features you want to build, you know the sort

of solutions that you need. Um, but here's a catch, um, and, and it's a real trade-off that that reliability isn't free. Um, you buy it by suppressing, um, variance that I'd argue novelty requires.

Um, and determinism and emergence pull kind of in opposite directions. Um, and this is where we have to pivot to the real world, because in real-life situations, in, when, when AI eventually, uh, when it becomes a norm for AI to capture real-world scenarios where you've got multi-agents, multi-humans, multi-institutions interacting and also

touching the physical world, um, there's a hard ceiling on novelty, um, because the real world is messy, and it keeps changing, and so a fixed harness starts to break down.

Um, so like I mentioned, there's a hard ceiling on novelty, um, because, uh, essentially its reliability is predicated on suppressing, um, variance. Um, the models are gonna keep accelerating.

Um, you know, you can build a careful harness today or you can use one off the shelf. Um, and but it could be irrelevant in the next month. Uh, the model just got so good, uh, that it didn't need that kind of scaffolding anymore.

Um, and that kind of leads to this whole dilemma of brittleness, where every situation you didn't anticipated, uh, you didn't anticipate needs a human to patch the harness. Um, the more real world it meets, the more rules you bolt on, and eventually the harness just becomes ever more complicated than the actual problem that you need to solve.

And I guess if there's one takeaway here, is that the factory method

is the right answer to a fixed problem and the wrong answer to a moving problem.

And so let's really examine what the real world is about, and, and I'm gonna go down a bit of a philosophical route here.

Um, as we've said, um, you know, agentic systems will eventually collide with the real world, and that raises a question. What is the real world actually made of? Um, because there's two ways to answer this actually.

So the first paradigm, the first way to see the world is through what we call a reductionist or analytical view, um, which we're all very used to actually. It says, "To understand the whole, a big problem or a system or a big thing like an organization, we've got to take it apart and study the individual parts."

So essentially, the world is made up of all these individual parts, which are things that are stable, and change is just something that happens to those stable things now and then.

The relationships amongst those things are secondary or an afterthought. And this is really the metaphysics of a factory and of nearly all software that we build. You know, you make the components, uh, you wire them together,

and you get a product. The second paradigm, what we call the systems or relational paradigm, um, takes the emphasis away from there being separate things or components and actually puts more what we call ontological emphasis on the relationships amongst those things.

So it flips the order. It says that the real world isn't made up of things at all. It's made up of processes and, um, relationships which come first.

So what we call a stable thing really is just a slow pattern in an ongoing flow. Think an organism. It's more like a flame than a crystal. A flame looks like an object, but it isn't.

It's a pattern held together moment by moment by a process. You stop the process and the thing doesn't exist.

Uh, and there's a really p- good payoff in understanding this paradigm. Um, and once you see the world in this way, you start to notice something remarkable, where, where you have simple local rules between, let's say, particles or between people or between creatures,

that local interaction gives rise to a whole new level

of order that can't be predicted from its constituent parts and that no constituent part designed ahead of time. For example, water. Water has a property of being wet,

but its contit- constituent parts, oxygen and hydrogen, are not wet. But they're coming together, again, putting emphasis on the relationships, they're coming together is where you have the emergence of a novel new property, i.e.

wetness. Another example is a flock of birds. Not one bird wakes up every morning intending to make a flock. Each probably just follows three simple local rules, which is align with your neighbor, don't crash into them,

stay close, and that's it. Local rules, local interactions, and out of that, you get a flock, a new pattern, alive and coordinated, that you could never understand by pulling it apart and studying one bird, because no single bird has a flock in it.

The flock lives in the relationships, not the parts. So this factory method of engineering assumes a world of parts.

But the non-machine world, and I'd argue that AI is sort of on the cusp of that, the non-machine world is a world of patterns and moving patterns of emergence.

And that sort of reflects what sort of problems we face in this moving, messy world. Um, and there's a line from Russell Ackoff that I absolutely love. He says that managers aren't handed neat, separate problems.

They're handed dynamic situations, tangles of problems that keep changing and keep bumping into each other. And he, and he gave that a name. He called it a mess. A mess is a, isn't just a pile of separate parts.

It's actually a system of relationships in motion, which is exactly why the factory based on a fixed harness way of engineering will struggle when it collides with real-world live scenarios.

Because you can't decompose a mess into tidy boxes and solve each of them, because they're not individual parts. They're a moving pattern of relationships.

We can then look at problems through this dis- this distinction between different types of problems. So, um, engineering is all about solving problems, and I'd argue that very simply put, there are two kinds of problems.

There are complicated problems, um, uh, and there are complex problems. So a complicated system or a complicated problem is like building a jumbo jet or a clock. There are passive parts, um, but experts can take them apart, analyze them, plan, predict, document.

It's hard, but it's actually knowable, and it works in a very predictable way.

In contrast, when you have a complex system or a complex problem, like a flock of birds, a market, a s- human organization, it's slightly different. Its parts are constantly interacting and adapting to one another.

So the whole can't be derived from analyzing the parts. So you don't analyze and plan a complex system. Instead, you probe and sense and respond. And this is probably one of the most expensive mistakes we make in the modern world of design and engineering, which is that we treat a complex problem like a complicated one.

Um, and, and things fail not due to lack of execution, but essentially because there's a failure in categorizing the problem spaces.

If we go just one step deeper into what a complex system is, um, essentially, you know, the ingredients of it are first, you need a diverse set of agents or actors if it's a human organization.

Um, not clones essentially, but different from one another in some sort of way, so because the diversity is the fuel there. Um, they need to interact locally, uh, 'cause no agent, no individual actor, no individual component sees the whole.

Um, instead, each respond to whoever's next to it, and this is like the bird aligning with the neighbor.

And they learn. They learn recursively. They adapt, then they adapt to the results of their own adapting, and it constantly loops. And, um, you know, with respect to one another and the environment.

Everything is moving in response to everything else, and nothing holds still. And out of all of that local adaptive looping interaction, you get something called emergence, which is a key concept in complexity science.

Um, basically it's a new novel pattern, uh, or organization or behavior that, uh, no single part designed on its own and can't be explained by splitting that whole into its parts and studying them.

Again, think the flock of birds. No bird has a flock in it. Think about the wetness of water. None of the individual oxygen, hydrogen molecules have wetness within them, and yet it leads to the emergence of something quite novel.

Now, these sy- systems don't fly apart into chaos. They tend to settle into attractors, which are essentially stable states. Uh, they keep pulling back towards some sort of equilibrium.

So for example, water is stable at room temperature. So you get this sort of beautiful tension. You've got constant change at a local level, but then you've got recognizable, stable patterns at a whole system level.

And crucially, nobody's steering here. It's self-organizing, and that's the whole mystery, and I'd argue that's the whole opportunity for design and engineering. And this is where really wanna introduce this new design and engineering philosophy called adaptive engineering.

Before that, let's just take stock a bit. So everything we've covered so far lives inside a particular framework, which is this kind of fixed or factory harness. Uh, the human engineer designs it all up front, the sequencing, the roles, capabilities.

That's really the factory model, and it works. Um, and we've seen that, um, because problem spaces that are complicated yet predictable, it's exactly right. And I'd argue for most engineering problems, that is the perfect method.

But the moment AI becomes exponentially better and it collides with the real world, which is, as we found, messy and complex, not just complicated, we need something genuinely new, a different idea of what optimal design and engineering even is.

Um, and it's, uh, obviously it rests on those two assumptions that models are gonna become exponentially better,

uh, and that AI will, uh, come outside of a sandbox situation and actually, um, engineering will run in real time in contact with the social and physical environment. Um, so essentially what we're saying is that the harness must adapt mid-flow, and that's the crux of harness engineering.

And I guess if I were to crystallize, um, the definition, um, adaptive engineering is the discipline of designing constraints to the extent that the harness emerges on its own, stabilizes, and adapts as needed in response to the changing environment in ways that you could not

specify in advance. Essentially, the harness becomes the ongoing output rather than the input. And you know what this means in practice, uh, just to make it a bit more concrete, is given that agents have capabilities, um, to interact with each other,

organization emerges from those interactions in response to the environment or a goal

such that new levels of order appear, ones that you could never have specified up front. Um, so think bird and a flock, think agent and harness. So the agents are actually creating the harness in the same way as the birds are relating to one another so that the flock can emerge.

So you don't build the harness anymore. You let the agents form the harness that best fits the environment in that moment, in that context. The harness then becomes a self-organizing, constantly evolving multi-agent system.

And if we were to just simulate what it may become, because this is when agents become a lot more capable, um, essentially you have agents that are isomorphic, um, but yet undifferentiated, um, but fully capable of interacting and, um, learning and changing.

Um, you let those agents interact. At first there's nothing. There's a bit of few exchanges here and there, but the coupling continues and slowly builds until you reach a point where roughly one connection on average, um, occurs, um, with each agent, and suddenly you start to see this whole emerge, um, a system you could say.

Uh, and I guess as an engineer, your new lever, uh, becomes, uh, how the rate of coupling, like do you dial it up or do you

dampen it down? Um, again, the adaptive path focuses on the interactions,

um, until, you know, agents start to actually specialize with respect to one another. So let's say you got two agents doing the same thing in the same place. That's just redundant, and the environment knows that.

The pressures from the environment knows that. So re- it rewards anything that essentially breaks the tie. Um, so I guess a tiny difference, um, you know, who got there first, who is slightly better, gets amplified by the feedback until those two agents are no longer interchangeable.

And so the pool stops becoming something uniform and it actually falls into, uh, niches. And really the key insight here is that the agent's identity isn't something you gave it.

It's actually the position, role, or capability it took relative to the others and its environment. So here we see specialization emerges from the interaction.

And if you go on a bit further, uh, they stop connecting at random. Uh, in fact, they start to work well together, um, in terms of sort of clusters.

Um, and you see the first boundaries appear, and crucially, the system drew them, uh, you didn't as an engineer ahead of time. Um, so these are kind of emergent clusters.

Um, until a new convention crystallizes, you know, a division of norms and protocols, and the system produces some level of governance without a governor. Conventions emerge spontaneously from their local coordination with no central authority, because as we've seen, central authority can lead to brittleness in a changing environment.

Um, so there's just enough repeated interaction to tip the group into some sort of shared norm, and this is crucial really, because it means that it can continue to adapt in a decentralized way based on the changing environment or problem space.

And essentially a new, um, emergent order, um, arises. So from those tiny little interactions amongst those agents, you have a new level of organization that is fairly stable up until the next iteration, up until the environment changes and, um, the order needs to essentially restructure itself.

So the question is that what is the role of an engineer? And, and really in adaptive engineering, uh, you don't abolish the engineer, you're just relocating the emphasis of engineering.

So if you can exploit the model's capabilities, um, so agents can essentially, um, interact, learn, and change, what you're doing as an engineer is essentially, um, exploring the cl- constraints.

You're dictating the constraints, the rules of the game. You're not deciding what should happen or what you want to happen, but you're giving the agent space to explore that field.

And then once that emergent harness, uh, starts to take shape, you are sensing it and responding to that rather than, um, stopping the engineering process and starting from scratch.

Um, and I think this slide sort of, um, portrays that. This is not binary. This is not either/or. Uh, in fact, it's very much a continuum. Um, if you, you-- There's no purely fixed harness way, and there's no purely, uh, completely adaptive, autonomous way.

Um, fixed engineering is essentially, in one hand, um, prescribing the structure which agents run, and you rarely change the structure mid engineering. Um, whereas on the other end of the scale, you've got adaptive engineering, where you allow the agents to freely interact and allow the harness or the structure that guides them, uh, to be created by them

and change mid engineering. Uh, so it's not like a swarm of agents loose with no roles and this sort of intelligence that magically appears. Um, and it's not sort of better.

One, one way is not better than the other. They just have two different use cases. And, um, if I could just, you know, bring it back to some examples of existing, uh, harnesses.

So for example, Hermes, it, um, on, on the website, it, it states that it's a self-improving AI agent that creates skills from experience and learns from it. So obviously, this is a huge step in terms of adaptive direction.

Um, but I do wanna make a distinction here, um, because, um, that adaptation is in the form of vertical intelligence, which is about making individual agents smarter. What I'm talking about here is horizontal intelligence, which is all about how groups of agents coordinate.

And so those two directions are a bit orthogonal. Um, and the thesis that I'm presenting is that it's horizontal intelligence, uh, that is the most adaptive method. It's the most agile method, um, and, and, and arguably the higher leverage point, um, [lip smack] uh, compared to vertical.

Um, another example is Pi. Um, you know, a great harness, uh, which is, uh, sort of essentially minimal, um, yet it's maximally extensible. Um, so it's fully customizable to an extent.

Um, so you could class that as adaptive, but again, there's a distinction here. There's adaptive at the design stage, like when you're preparing for that engineering process, um,

uh, where the engineer can customize and the tool's quite malleable. But when it comes to adaptive engineering, that's all about being adaptive during the runtime, where the system reorganizes itself whilst running in response to the changing problem space of the environment.

And, you know, Pi is a strong example of the first kind, um, but it doesn't make any claims, um, to be anything other. Uh, it's not a self-organizing multi-agent system yet.

I'm just gonna skip over some of the, uh, nitty-gritties of what that could mean. Um, so for example, um, [lip smack] uh, when it comes to agent capabilities, um, you know, obviously they need to interact, learn, and change.

That's kind of a prerequisite for adaptive engineering, and I don't doubt that That will, um, become the case as models get, um, exponentially more powerful. Um, there's a few things that the designer can actually do in the form of constraints, and I'm not gonna list out all the different types of constraints, uh, because they're sort of largely

dictated by some of the properties, like the roles and the sequencing and the memory. These are all sort of categories or properties of harness. But essentially, there'd be questions that you could ask yourself as an engineer, which is, um, do you want to, um, enable, [keyboard clicks] uh, the agents more, or do you wanna close them in and govern

them more, uh, and produce more-- g- uh, give them more guardrails? Um, are you rewarding them to cohere towards a particular goal, or are you costing them, um, if they fall outside a particular container?

Um, and how fast or slow do you want this, um, [lips smack] to happen, some of the properties? Uh, you know, what's the speed, uh, for example, the rate of coupling?

So these are some of the engineering tweaks that you can make when it comes to adaptive engineering. [lips smack]

And then finally, like this emergent harness that is constantly shaping and reshaping itself based on the problem space, um, you can only really sense and respond to that emergent harness.

Um, you're not gonna, uh, uh, completely, um, be able to, I, I guess, um, edit it in some sort of hard engineering fashion, because that's not in the best interest of you as an adaptive engineer.

This slide is just a more detailed, um, comparison of both approaches. And again, I wanna emphasize that one is not better than the other. It's just two different design engineering philosophies.

Um, where on one hand, in the factory model, um, you know, the structure is imposed upfront by the engineer, and so the harness becomes the input. Um, whereas in the adaptive path, the harness emerges and is essentially the output.

Um, and, and one of the crucial things here is that control is decentralized in the adaptive path,

um, in the field continuously, um, because the problem space, eventually when AI, um, comes, uh, or AI engineering comes off a screen into the real world, um, the problem space is constantly changing.

Um. [lips smack] Now, it's important to understand that, you know, there are particular use cases for this, um, but there are a huge number of failure modes, um, in this as well.

Um, [lips smack] you know, just to list a few of those failure modes. You know, emergence leans towards, uh, stability, what we call an attractor, um, where you find a structure that feels stable and optimal.

But that doesn't necessarily mean it's the best. Um, [lips smack]

you need some sort of genuine selection pressure. So if you imagine in the social world or the physical or the natural world, the environment is, um, giving us certain, uh, pressures.

You know, that's what encourages us to adapt. That is evolution. You know, there is some sort of selection pressure here. And you kind of need to find that, um, in...

Or we as a community need to find that in adaptive engineering, otherwise you just get drift.

There is a risk of monoculture, where you don't get genuine diversity amongst agents because they're all trained on the same data. Um, legibility does collapse. Um, you know, you, as adaptability increases, um, [lips smack]

you can't really pin down or explain things that well. And again, that's a feature of complex systems. You can't really pin them down. [inhales]

And, and, and obviously, it goes without saying that there's no more predictability ahead of runtime because things are constantly moving. Um, [lips smack]

and, and, and these are the real failure modes that we must apprehend, um, because I do feel like adaptive engineering is something that we're gonna be more inclined towards, and it's gonna race forward, um, from the future. [clears throat]

I guess a bit of a takeaway then, um, you know, as models improve exponentially, um,

AI engineering will move into the real world scenarios. You know, the real world where it touches, um, the physical space, uh, on an ongoing basis. It's, uh, in contact with multiple agents across different institutions.

It's in contact with multiple human beings. So, so there's a real cusp here where AI is heading towards. And so the discipline of design and engineering is gonna be forced to rethink itself around continual production.

And the limiting factor, which is the case now, but probably more so in the future, is not gonna be the strength of the model. It's gonna be the adaptability of the harness.

Um, and adaptability really here means multi-agent, um, which is essentially about coordinating amongst agents. A multi-agent orchestration, which is decentralized,

um, and it's not adaptable ahead of runtime. In fact, it starts becoming adaptable mid-runtime or during an entire engineering, uh, process. And I think that's real intelligence. That's...

Well, when I say real intelligence, I mean that's the intelligence when it comes to adaptability of the harness. [lips smack]

Uh, I'd be really interested in, um, hearing your thoughts on this, um, and whether this is something that, uh, other people are also exploring. Um, please do reach out to me.

Thank you.