AI Engineer World's Fair 2026
What if the harness mattered more than the model? - Aditya Bhargava, Etsy
Read the talk
What if the harness mattered more than the model?
A coding agent’s journey from an inaccessible file to a tested repair shows how tools, permissions, feedback, subagents, and prompt optimization change what a model can accomplish.
From a talk by Aditya Bhargava
Before you start: Familiarity with basic Python functions, automated tests, and LLM tool calling will help; no prior knowledge of Agency is required.
Could a better harness make local models good enough?
What would it take to get the performance you need from a local, open-source model instead of depending on a powerful proprietary one? Aditya Bhargava, who introduces himself as an Etsy staff engineer and IC initiative lead for agentic commerce, approaches that question through harness design. His background includes writing Grokking Algorithms and illustrated technical posts on ducktyped.org. The engineering question is whether more of an agent’s capability can come from software developers control.
The familiar advice is to keep the harness simple: give a strong model a few tools and let it work. That can be effective, but it makes the model provider responsible for more of the system’s intelligence. Investing in the harness offers another direction: build enough support around a model that a locally runnable alternative becomes useful for the task. Matching cutting-edge performance this way is the ambition that motivates the experiments.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What changes when the harness changes?
Harness-Bench gives this question an empirical starting point. Bhargava introduces it as a way to compare harnesses while keeping models and evaluation conditions consistent. The matching paper evaluates 106 tasks across eight model backends. Its aggregate composite scores range from 52.4 for OpenClaw to 76.2 for NanoBot, a 23.8-point spread. Those are averages across backends, rather than one model’s success rates; the composite combines completion, security, and process quality, with an LLM judge contributing to process assessment. The comparison measures complete harness configurations, so it does not isolate which individual feature produced the difference.
The study also observes greater sensitivity to harness choice among weaker backends. That makes harness engineering particularly interesting for developers who cannot train a leading model themselves. They can still build tools, execution policies, feedback loops, and context management. The evidence supports taking that work seriously, while leaving local-model parity as a question to investigate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A language for the harness
Bhargava’s exploration led to Agency, a language for building agents. At the time of the talk, he describes it as roughly six months into development, motivated by capabilities he could not obtain from existing tools and frameworks. His proposed solution puts those capabilities into the language itself.
Here, an agent means a model plus its harness. In his example, Claude Code is the agent, Opus is the model, and the surrounding machinery is the harness. Agency supplies that surrounding machinery. Its design follows the Alan Kay principle that simple things should be simple and complex things possible: common agent operations should require little ceremony without ruling out more elaborate execution patterns.
The demonstration develops a coding agent through seven stages. Initially, the model and repair task stay the same while the harness changes; later, subagents deliberately broaden the assignment. Safety develops alongside capability throughout. An agent needs enough authority to act on the user’s behalf, but unrestricted authority would make that usefulness dangerous.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A median bug, then an observable tool call
The repair target is a Python function in demo/median.py that calculates the median of an ordered list. For an odd-length list, the answer is the middle element. For an even-length list, it is the mean of the middle two elements. The existing function gets the latter case wrong, and an existing test exposes the bug. A compact implementation of the intended behavior for nonempty, ordered inputs is:
python
def median(numbers):
middle = len(numbers) // 2
if len(numbers) % 2:
return numbers[middle]
return (numbers[middle - 1] + numbers[middle]) / 2
assert median([1, 3, 5]) == 3
assert median([1, 3, 5, 7]) == 4
The illustrative assertions make the distinction concrete: selecting one middle element is insufficient when the list has two central values.
The first Agency program does almost nothing around the model. Agency resembles TypeScript with some Python-inspired syntax; its entry point is node main. This program asks the LLM to fix the Python median function, receives a result, and prints it. The model responds that it needs to see the existing code. Naming a file in a prompt does not grant access to it.
Before adding filesystem access, Bhargava demonstrates the tool interface with a greet function. Any Agency function can be passed directly as a tool: Agency generates its JSON Schema and uses its docstring as the tool description. The function’s parameters become the arguments the model can supply.
Running the example through npx produces a greeting, but the greeting alone would not establish that a tool ran—the model could generate one itself. Agency’s built-in log viewer exposes the actual sequence. In the inspected terminal frame, greet receives the name Adit, the tool returns Howdy, Adit!, and the assistant returns the same text. The execution trace distinguishes an action from a plausible description of one.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
File access needs an approval policy
The next version passes Agency’s built-in read and write functions directly to the model. That creates the ability to request filesystem operations, but unrestricted reads and writes would be unsafe. Agency refuses to proceed without an approval mechanism. The example stops with an unhandled standard-read interrupt; the error asks whether the user wants to read the requested file, identifies that file, and points to the handlers guide.
A handle block supplies the missing policy. It wraps the LLM call and connects raised interrupts to a handler function. The handler prints the interrupt information, obtains a response using the built-in input function, and approves the operation when the user agrees. An interrupt pauses execution so that a decision can happen before the operation proceeds. Bhargava explains that standard-library operations involving mutations, destructive actions, or sensitive reads raise interrupts first.
In the live run, he approves the read, the agent reads the file, and then a write request follows. This establishes a usable human approval path, but it also puts the human on the critical path for each action. The next requirement is more demanding: let the agent work autonomously within an appropriate scope without handing it destructive access to the rest of the filesystem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Bind the scope before exposing the tool
Partial function application fixes selected arguments before a function is called. Agency’s read tool accepts a filename and a directory. Calling .partial with the directory set to demo produces a tool whose directory is already chosen by the program. The model sees only the filename parameter; it neither sees nor supplies the bound directory argument. The same approach applies to writes.
This changes the capability exposed to the model. Instead of choosing both where to operate and which file to use, it chooses a filename within a scope supplied by the harness. Bhargava describes the resulting tools as confined to demo and suitable for use without repeated human approval. Actual filesystem confinement also depends on the file tool enforcing that boundary, including path traversal, absolute paths, and symlinks; binding an argument alone does not establish those protections.
The run now reads the file without asking for permission and produces the correct proposed repair. Yet it stops short of applying it, asking whether the user wants the file updated. The harness has removed an access obstacle and an approval bottleneck, but the task is still incomplete: a proposed fix is not an edited file.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Close the loop with failing and passing tests
The next change introduces ReAct: reasoning and acting interleaved with observations from the environment. For the median repair, the loop gives the agent a concrete procedure:
- Read the implementation and its test file.
- Run the tests to observe the failure.
- Use the failure to decide what to change.
- Write the repair.
- Run the tests again, then continue if they still fail.
The harness changes are a revised prompt and the ability to run the tests. The agent now has both an action it can take and an observable condition for deciding whether its work succeeded.
Bhargava enables printed tool calls to make the execution visible. After a brief interruption in the live demonstration, the trace shows the intended sequence: the agent reads both files, runs the tests, observes failure, writes the implementation, reruns the tests, and confirms success. This is the first version that completes the repair and checks the result.
Each harness change has solved a different problem:
| Harness stage | What it enables | Remaining obstacle |
|---|---|---|
| Model alone | A conversational response | Cannot access the file |
| File tools | Requests to read and write | Unhandled approval interrupts |
| Approval handlers | Authorized operations | Human input slows progress |
| Partial application | Scoped autonomous access | Stops at a proposed repair |
| Feedback loop | Applies and tests the repair | Confidence remains tied to the tests |
The final step supplies a completion process, not just more permissions. The agent must compare the consequences of its changes with the task’s test results.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Expand capability without mixing every tool together
The subagent example changes the assignment. Alongside fixing the failing test in test_median.py, the agent is asked to research Jensen’s inequality for medians using a Wikipedia agent and explain it in limerick form. This exercises a broader set of capabilities than the original repair.
The top-level agent no longer receives raw file tools. It receives two functions that act as subagents. In Agency, a subagent is an ordinary function containing its own LLM call, so the same function-as-tool interface works at both levels:
- Coding subagent: Keeps its own system message, LLM call, and tools for reading, writing, and running tests.
- Wikipedia subagent: Has its own LLM call and the standard library’s Wikipedia search tool.
Callbacks print the tool-call information, preserving visibility into the delegated work.
In the demonstration, the top-level agent calls the two subagents in parallel. The Wikipedia agent searches while the coding agent modifies the file, and the result includes a repair and a limerick. This shows delegation and combined output; the mathematical correctness of the generated explanation is not established by the demonstration.
The architectural benefit is separation of context and capabilities. An agent with many unrelated tools must choose among them while processing unrelated concepts in the same context. Subagents let the parent make a coarser decision—coding or research—while each specialist works with a smaller, relevant tool set. Bhargava identifies context bloat and unrelated tools as recurring sources of confusion; grouping them by responsibility gives the model a cleaner decision to make.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure and revise the prompts
The final example uses Agency’s built-in GEPA optimizer, one of several optimizers Bhargava says the language provides. Variables marked with the optimize modifier become candidates for revision. Here, two prompts begin with very little information, and the optimizer receives the goal of fixing median.py using test-driven development.
The optimizer identifies the marked prompts and runs the agent to establish a baseline before attempting improvements. The live optimization run reports a baseline objective value of 0.2 for the median-repair task. The scoring definition is not supplied, so that value should be read as the demonstration’s objective score, not as an accuracy percentage.
Because optimization takes time, Bhargava switches to an earlier completed run. He reports that it achieved its objective in the first iterations and shows a rewritten prompt. The useful shift is from informal prompt tweaking to systematic evaluation and reflection. GEPA still proposes and tests candidates; it organizes experimentation around execution feedback and a measured objective rather than eliminating trial and error.
By this point, the seven-stage progression has moved from a model that cannot read the target file to an agent with tools, approval handling, scoped autonomy, test feedback, specialized subagents, and prompt optimization. The last two stages broaden what the agent can do and provide a process for improving it. Whether that support can make a local model sufficient for a particular workload remains the motivating experiment: define the task, improve the harness, and measure whether the resulting agent meets the need.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Preserve execution through a human decision
The strongest justification for language-level support arrives at the end: true pause and resume. Function-based tools, interrupts, handlers, and partial application simplify common harness operations, but an interrupt also needs to preserve a computation that may be deeply nested. Agency’s intended behavior is to return control to the user while retaining the execution state needed to continue later.
Consider an interrupt raised inside a for loop, inside a tool call, inside a subagent. Bhargava says Agency can pause that computation and later resume at the exact point inside the loop. The caller should not need to flatten the program or manually reconstruct its position just to obtain approval. He contrasts this with restrictions he has encountered in framework-level human approval features, while qualifying his comparison with other languages as based on what he knows.
He further describes serializing execution and returning to it a week later without changing the demonstrated code. That makes human approval an interruption of an ongoing computation rather than a requirement to start the task again. Together with built-in optimizers, these are the language features intended to support more ambitious harnesses than a single prompt-and-tools call.
To try Agency, the installation command visible on the slide and given in the current project README is:
sh
pnpm install agency-lang
Bhargava describes the package and an API key as the starting requirements. The linked README is current documentation rather than a pinned release of the demonstration. From there, the practical experiment is the same one developed through the median repair: keep the task concrete, give the agent appropriate authority, and use observable results to improve the harness around the model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
A study of harness effects across 106 sandboxed workflow tasks, with composite scoring and a six-harness, eight-model evaluation matrix.
Getting-started instructions for Agency, including the agency-lang package and a minimal program.
The original ReAct paper explains how reasoning and actions interleave with observations from an external environment.
Prompt optimization through reflection on execution trajectories and evaluation of revised candidates.
Further reading
Public benchmark assets for evaluating agent frameworks in filesystem workspaces.
Read the complete timestamped transcript
- 0:01
Hi everyone. My name is Aadit, pronounced Aadit like a tax audit, um, and I'm here to talk to you about what if the harness mattered more than the model?
- 0:11
A little bit about me first. I'm a staff engineer at Etsy. I'm also Etsy's IC initiative lead for agentic commerce. I wrote Grokking Algorithms, which is an illustrated book on algorithms, uh, and I also do illustrated posts on ducktyped.org.
- 0:27
Um, and I wanna talk to you about
- 0:32
what if the harness mattered more than the model, by which I mean what if we should be paying more attention to building expertise and harnesses? Um, and my controversial take here is I hear a lot of people say, "Oh, the models are so good that you can just keep the harness simple.
- 0:54
Just give the model a few tools and it'll do the rest." And I've been hearing this so often that it has now, uh, started becoming the wisdom of the tech industry.
- 1:06
Uh, to which my answer is yes, but that's moving in the wrong direction because that is making us reliant on fancy proprietary models that can't be run locally. So what if we instead talked about the other direction?
- 1:23
What if we focused on open-source models that can be run locally? What if we focused on building a harness that is so good that we can get the performance of a cutting-edge model through a local open-source model?
- 1:37
So what if the harness mattered more than the model? What if our focus should be on the harness?
- 1:45
Um, how much does the harness matter, right? This is something I have been researching a lot in my spare time. Um, there's a really interesting paper that talks about HarnessBench, which is a benchmark for harnesses, which is something we haven't seen a lot of yet.
- 2:02
Um, it's got 106 tasks. Um, it's testing these different models and different harnesses. Uh, but the point is that it's running the same setup, it's doing the same evaluation, same model, but testing different harnesses.
- 2:22
Um, and the results are pretty interesting. So scores range from fifty-two point four percent to seventy-six point two percent, so more than a twenty-point difference, and only the harness changed.
- 2:36
Um, and the really interesting thing is that for weaker models, the harness matters more.
- 2:44
Um, if the harness matters more than the model, that's great news, uh, because if the model matters more, then we're dependent on a handful of companies who are able to build and train these models.
- 2:58
But if the, if a good harness can compensate and can make a weaker model perform better, then we can build our own harnesses, and that's something that any of us can do, and we don't have to depend on paid models.
- 3:13
Um, so I have been spending a lot of time and doing a lot of research specifically into what would it take to build a really good harness. Um, and
- 3:29
I realized early on that none of the existing tools or frameworks, uh, were able to do what I wanted, and I realized that what was actually needed was a language-level solution.
- 3:43
So this is my other controversial take in this talk is I think building a good harness actually requires language-level support. Um, and this is kind of where my research has gone in the last six months, and I've spent a good part of that six months building a new language called Agency, which is a language for
- 4:07
building agents. Um, and just to talk about the terminology there real quick, because the terms agent and harness aren't super crisp in the industry yet. They're not super well defined.
- 4:24
Um, but in this talk I'll kind of use them interchangeably. But when I talk about the agent, I really mean the harness and the model. So if you take Claude Code, for example, Claude Code is an agent.
- 4:38
It uses Opus, which is the model, and then everything else is the harness. Um, but you know, when we talk about Agency, it's really for building agents, but the thing it's providing is the harness.
- 4:55
Um, Agency is really good. Uh, you know, it's still new. Building a language is a lot of work, and it's only six months in. Uh, but I love this quote by Alan Kay, "Simple things should be simple.
- 5:10
Complex things should be possible." And that's something I love about Agency is that I think, you know, when people build agents, there are a lot of primitives, a lot of things that every agent
- 5:26
will need to be able to do, and those things should be simple to do. Um, and there's a lot of complex things that we want to be able to do with agents, and it should be possible to do those things in the framework of your choice, and with Agency it is possible.
- 5:46
Um, so now we have a clear goal, which is try to build the best possible harness, and we have a great tool to do it, which is Agency. Uh, so the rest of this talk I'm going to talk about building a coding agent in Agency.
- 6:02
And we'll see the same agent evolve over seven different examples. We'll see the same model, the same task, but the harness will improve each time, and we'll see the harness improve the agent's performance each time.
- 6:19
Um, so you know, like I said, I've been doing a ton of research on this. My goals with this talk are to give you, like, the basics of building a harness.
- 6:29
When we talk about building a good harness, what does that mean? You know, if you're starting from zero, what are the basics? Um, I'll cover a good bit about agent safety because, you know, when we talk about building agents, what we really want them to be able to do is take action on our behalf.
- 6:50
Um, but the big problem with that is how do you get-- how do you give agent capabilities, so they can take actions,
- 7:01
but not so much capability that they do something unsafe? So when we talk about agent quality and capability, agent safety goes hand in hand with that.
- 7:14
Um, at the end, I'll do an introduction to some of the more advanced concepts like subagents and self-optimization. I think that's where, you know, more interesting things come in, and we really start to talk about, like, what are all the possibilities, you know, because when you talk about building a good harness, there are so many things to
- 7:34
explore. And so I hope you stick with me till the end there, because that's where we get into some really interesting stuff.
- 7:45
Um, so this is the code we're going to ask our coding agent to edit. This is a code to calculate the median of an ordered
- 7:57
list of numbers. Um, and if it's an odd length list, you just take the middle element. If it's an even length list, you need to take the middle two elements and take the mean of those two elements.
- 8:13
But right now, this function doesn't do that. So there's a bug in this function. There's a test that shows the failure, um, and we're going to ask our agent to fix this code.
- 8:30
So let's go ahead and start. Let's just dive in. Um, this is the first example. This is Agency code now, and you'll notice that Agency looks a lot like TypeScript.
- 8:42
It's heavily inspired with T-- by TypeScript with some Python syntax thrown in. Um, so the entry point is this node main. I have a simple prompt, the Python function median in demo/median.py has a bug.
- 8:58
Please fix the bug. And I just pass that prompt into the LLM function, and I get the result, and I print the result. Um, and of course, this isn't going to work because it can't read or write to this file.
- 9:13
So let's run this example real quick. Great. So of course, as expected, it says, you know, "In order to help you fix the bug, I'll need to see the existing code."
- 9:32
Um, so this is just a model, very little harness, and of course, it can't do anything. So the most obvious improvement to any harness is give it some tools.
- 9:45
So real quick, let's talk about how tools work in Agency, because it's very simple. Um, I'll define a function. In this case, just for an example, I'm going to define a greet function that greets a user, and then I'll make an LLM call that says, "Use your greet tool to greet Aadit," and then I pass in that
- 10:08
tool. Um, and every function can automatically be used as a tool in Agency, so there is nothing else to do. Uh, Agency creates a JSON Schema out of this function and sends it to the LLM call.
- 10:24
The doc string here will get used as a tool description. So let's go ahead and run the second example.
- 10:32
So NPX Agency... Oops. Actually, NPX Agency example, and it's actually zero one B, because we're gonna talk about tools first.
- 10:47
So howdy, Aadit. And how can we know that it ran that tool and didn't just generate an example, right? So I'll just also show real quick,
- 11:00
um, Agency writes out logs, and it has this great log viewer built in. So I'll just expand this last run, and you can see here it's making this tool call to the greet function with these parameters.
- 11:16
The tool responds by returning howdy, Aadit, and the assistant just passes that straight back to us. Um,
- 11:26
so that's tools. Now let's look at using the tools in Agency. So again, this is, you know, the next step of our agent, which is give it tools. Um, here is an example where we're giving the agent read and write tools.
- 11:48
Again, read and write are just functions that are built into Agency, and because every function can be used as a tool, I can just pass this straight to the LLM.
- 11:58
Uh, except giving the agent the ability to read and write arbitrary files on our file system is really unsafe. So by default, Agency won't allow you to do this.
- 12:12
Agency has a human-in-the-loop feature that will require some sort of approval. We're not providing it anywhere, so this code will crash and print an error. So let me show you what that looks like.
- 12:33
Great. So interrupt standard read was not handled. Here's the interrupt message. "Are you sure you want to read this file?" You can see the file name it's trying to read, and it's basically telling us that we didn't approve this interrupt, and it's giving us a pointer to the guide where it tells us how to use handlers.
- 12:54
Um, so we started with just the model. We tried tools. Now the next step is to make the tools safe. So we're going to do exactly what that message asks us to do and use a handler.
- 13:12
So now here's the same code, you know, passing through that read and write, uh, those read and write functions as tools, um, but now it's wrapped in a handle block.
- 13:25
Uh, and it has a handler function down here. So now when those tools call or raise an interrupt,
- 13:34
this code is going to execute. So it'll print the information about that interrupt. Um, it'll ask the user for input using the built-in input function, and if the user approves, the tool call is approved.
- 13:52
And I know I'm saying interrupt, and I'm not really defining what that means, um, and I can get to that a little bit later, but it's essentially a way to pause execution at some point, um, and ask for human input.
- 14:08
And so interrupts are a great feature in Agency, and all the functions in the Agency standard library that mutate code or mutate something or have some sort of destructive action or read sensitive data, data all throw an interrupt first.
- 14:29
So here is the safe version of our agent, where the tools
- 14:38
will raise an interrupt by default, and now this code is going to ask the user, "Do you approve?" So let's run this example.
- 14:53
Great. "Are you sure you want to read this file?" I'll just say yes.
- 14:58
You can see it's reading. Now it's trying to write to that file. So, you know, that's an example of us
- 15:08
making the agent safer by asking for user input. And this is better, and it is safe, but it's also very slow. So the next thing we want to do is make the agent autonomous
- 15:22
while still keeping it safe, and that's the tricky part, right? How do you give it just enough capability so that you don't have to manually approve every single action without giving it, without giving it so much capability that it's, like, able to do something destructive with your file system?
- 15:44
Um, so Agency has a really cool feature to handle this. It, that's called partial function application. Um, it's a concept borrowed from functional programming. It's a really fancy term for a pretty simple concept.
- 16:02
So here's a signature of the read tool, right? It takes a file name and a directory, um, and when you read a file, it's going to look inside that directory for that file name.
- 16:15
And with partial function application, what we do is we take that read function and call .partial on it
- 16:22
and give it a directory. And what we're doing here is we're locking the directory argument to demo. Um, the LLM isn't going to be able to change that argument.
- 16:34
It's not even going to know that that argument exists. When we pass in the read tool now, it's just going to see one parameter, which is file name. It's going to pass in the file name, and it will read that file from this directory.
- 16:49
So we've locked the directory parameter. Now the agent can only read files from and write files to this directory. Uh, partial function application is a really great way to constrain the capabilities of your agent.
- 17:07
Now, no human input is needed, but it's still safe. So let's see that in action. So again, we run NPX Agency examples. PFA is partial function application. So here
- 17:23
it's thinking, and you can see it read the file, didn't need to ask us for permission, and now it said, you know, to fix this issue, it's kind of giving us a new code, but it's not updating the code.
- 17:38
It asks, "Would you like me to update the file with this corrected function?" But it didn't do that itself. So, you know, this version of this agent is the best one we've seen so far because it does give us the correct solution without any human input needed, but it hasn't actually fixed the code.
- 18:01
So the next step is to get it to actually fix the code. Um, and that's where the feedback loop comes in. Um, so there's a very popular agent pattern called ReAct, and ReAct stands for reason and act.
- 18:18
And essentially, you're asking the agent to work in this loop where you reason, act, observe the consequences of your action, and then decide what to do next. So in this case, reason, read the two files, act, run the tests,
- 18:40
observe if the tests fail, read the error, and then reason again. So look at that test failure and think about what you want to do next, and just do this in a loop until the tests pass.
- 19:00
So this is a really key pattern to making the agent better. And so now let's see the same agent. The only thing we're changing is the prompt and giving it the ability to run these tests, and now the agent should work better.
- 19:24
In this case, I'm gonna also print all the tool calls just so you can see what's happening. Um...
- 19:47
Well, shoot. Maybe... [laughs] All right. Hang on. There's always where... Uh-huh.
- 20:02
There we go. Okay. Now let's see it in action. Um, so now you can see it's calling the tools. It's reading those two files, like we said. It's running the tests.
- 20:16
The tests fail. It's writing to the file. It's running the tests again and confirming success. So this is a really important
- 20:27
stage in the harness development because now we finally have an agent that successfully
- 20:37
figures out the task, checks that the code is failing,
- 20:43
modifies the code, and checks that the code now passes. Um, so let's talk about how we have improved the harness so far. So we started with just the model, couldn't do anything.
- 20:57
Then we added tools, but the agent wasn't safe. Then we added handlers, which added safety but needed human input. Then we added partial function application, which was safety without requiring human input.
- 21:14
And finally, the last one that we just looked at was the feedback loop, which added reasoning. So the agent made the change and confirmed that the change worked.
- 21:29
So I have two more examples. This one, we're going to go in a slightly different direction, and this is where we're getting into some of the more advanced concepts.
- 21:38
So this one is subagents. Um, everything so far we've done has made the same task better. But with subagents, we're asking, how do you make the agent do more things?
- 21:49
Um, and let me show you what this code looks like real quick.
- 22:02
Okay, so this program, I'll start by just showing you the prompt we're giving because it's slightly different. So fix the failing test in test median.py, then research Jenso-Jensen's inequality for medians with the Wikipedia agent and explain it to me in Limbic form.
- 22:22
So here is a case where we're actually asking the agent to do a little more, right? And this one is interesting because this is our main agent, but for tools, we're not giving it r-the read and write functions.
- 22:38
Instead, we're giving it two subagents as tools. And I really like this pattern in Agency because I've seen a lot of frameworks make subagents their own concept and make them kind of hard to grasp.
- 22:55
Um, in Agency, a subagent is just another function, and you just call it, use it like a tool, just like you would anywhere else. And that consistency has a lot of benefits, makes it really easy to write and reason about.
- 23:09
So here's the coding agent subagent. Um, same one as before, you know, there's a system message.
- 23:19
This subagent has its own LLM call, gets its own set of tools. Similarly, you have the Wikipedia agent. That has its own tool, which is the search tool, uh, which is a tool built into this Agency standard library that lets you search Wikipedia.
- 23:36
Uh, so Agency has a big standard library that lets you do all kinds of things, including this Wikipedia search tool. I'm also using the callback feature to print the tool call information.
- 23:49
And now let's see this example in action, and this is an example where we're actually asking the agent to do a little bit more, and we're giving it two subagents to accomplish its task.
- 24:04
So here you can see it's calling those two subagents. Uh, it's calling them in parallel, so the Wikipedia agent is doing the search, the coding agent is modifying the file, and That fixed the error, and now here is a limerick.
- 24:21
So subagents are cool because they let you add new capabilities without bloating context. Um, and this is a good pattern because it allows your agent, when it has a lot of different tools at its disposal, it allows your agent to do the right
- 24:46
thing more consistently. Often when agents get confused and fail, it's because they have too much context bloat, but also because they have too many unrelated concepts in their context, and they have too many tools that are unrelated, and so they have a hard time picking the right tool to use.
- 25:06
In this case, we make that problem a lot cleaner. For the top-level agent, we just say, "Pick the subagent you want to call," and then the subagents have those tools that are kind of grouped in a way that makes sense.
- 25:21
Um, so now let's look at the last example, which is self-optimization.
- 25:28
Uh, Agency has a JEPA optimizer built in. It actually has a few optimizers built in, but the JEPA one, uh, is a really interesting one. And the way the optimization feature works is you mark any variables that you want the optimizer to optimize.
- 25:49
You can just use this optimize modifier, and then you run the optimizer and give it a goal. So, you know, in this case, you can see that the two initial prompts I have are very basic.
- 26:03
I'm giving it hardly any information. And then the goal I give it is fix the bug in median.py using TDD. And the reason this is such a great step for our harness is we're no longer doing trial and error.
- 26:22
We're no longer just trying stuff to see what works. Instead, now we have a way to systematically measure and improve performance. So I'm just gonna go ahead and run this code, um, and I'm actually gonna also reset.
- 26:42
Here, I'm gonna modify this code. Cool. So now... [clears throat]
- 26:59
And I won't show the full run here because the optimizer takes a while, but you can see first it's, like,
- 27:09
checking what prompts it has that it needs to optimize. It's going to run the agent to establish a baseline. So the baseline objective is zero point two, and it's going to try to improve upon that.
- 27:25
So in this case, I did run the optimizer earlier just so I could show you what a full run looked like. Um, and so here is one where you can see
- 27:37
it achieved the objective in the first iterations. It was actually quite quick in this case, um, and you can see that it's rewritten this prompt. Um,
- 27:51
so the reason this self-optimization is so great is because we're not guessing and checking. We're systematically measuring and improving, which is a big leap forward.
- 28:04
So this is the ladder we just went through for improving the harness. Um, you know, first step was nothing, just the model, can't even read the file. Then we added tools, but in a way that was unsafe and not allowed in Agency.
- 28:19
Next, we added handlers, which added safety but made things slower because they required human input. Then we added partial function application, which was safe and fast. Then we added reasoning, which gave us higher confidence in the output of the agent.
- 28:39
Uh, and then we switched tracks to subagents, added more capabilities, and finally, we did self-optimization, so we got measured improvement and not guess and check. So, you know, I hope this gives you an idea of how you can build harnesses and then slowly improve those harnesses.
- 29:03
And I hope, you know, you try doing that and come along on this, you know, experimentation with me to see if we can build a better harness, um, because what if the harness matters more than the model?
- 29:17
You know, what if we can build a harness that's good enough that we can now start using local models and still get the performance we need for the tasks we wanna do.
- 29:30
So what language level support does Agency provide? You know, we used Agency for all these examples. How did it help?
- 29:36
Um, simple syntax for defining tools. Again, you know, the simple things should be simple. Every agent is going to need tools. Let's make it easy to use them. Tools for safety, very important for agents, so interrupts, handlers, PFA.
- 29:52
True pause and resume execution. I didn't cover this a ton 'cause I didn't wanna make this talk too long, um, but one of the really killer features of Agency is that when you raise that interrupt, it's actually going to pause execution and return control back to the user, but in a way where you can resume execution later.
- 30:15
Uh, and as far as I know, very few languages allow you to do something like this. And so typically, most frameworks you'll use, if they have a human-in-the-loop feature, it has a lot of restrictions.
- 30:29
There's a lot of things you need to do to work around it. Agency is completely different. If you wanted, you could, you know, raise interrupts inside of a for loop, inside of a tool call, inside of a subagent, um, and it would correctly pause execution, go back to the user, and then resume execution at that exact point
- 30:52
inside that subagent, inside that tool call, at that exact point in that for loop, um, which is really cool. You can serialize the execution. You can come back to it a week later, and you don't need to change anything about all the code we have been looking at.
- 31:08
It just works, which is really cool. Uh, it also has these built-in optimizers, which are great, you know, which kind of get you to a more analytical, mathematical way of doing the improvements.
- 31:20
So if you wanna try it out, please do. Pnpm install agency lang. All you need is the package and a API key, and you're set. Um, again, takeaways, you know, an agent is a model plus a harness for this talk, at least.
- 31:37
Uh, better harness equals better performance, especially for weaker models. And then how do you build a better harness? Give it tools, make it safe, make it autonomous, make it reason, give it subagents, and have it self-optimize.
- 31:52
Thank you so much. Again, my name is Aadit Bhargava. Check out agency, agencylang.com, uh, and then please follow me on Blue Sky if you wish. Thank you. Take care.