What if the harness mattered more than the model? - Aditya Bhargava, Etsy

What would it take to get the performance you need from a local, open-source model instead of depending on a powerful proprietary one? Aditya Bhargava, who introduces himself as an Etsy staff engineer and IC initiative lead for agentic commerce, approaches that question through harness design. His background includes writing Grokking Algorithms and illustrated technical posts on ducktyped.org. The engineering question is whether more of an agent’s capability can come from software developers control.

The familiar advice is to keep the harness simple: give a strong model a few tools and let it work. That can be effective, but it makes the model provider responsible for more of the system’s intelligence. Investing in the harness offers another direction: build enough support around a model that a locally runnable alternative becomes useful for the task. Matching cutting-edge performance this way is the ambition that motivates the experiments.

Harness-Bench gives this question an empirical starting point. Bhargava introduces it as a way to compare harnesses while keeping models and evaluation conditions consistent. The matching paper evaluates 106 tasks across eight model backends. Its aggregate composite scores range from 52.4 for OpenClaw to 76.2 for NanoBot, a 23.8-point spread. Those are averages across backends, rather than one model’s success rates; the composite combines completion, security, and process quality, with an LLM judge contributing to process assessment. The comparison measures complete harness configurations, so it does not isolate which individual feature produced the difference.

The study also observes greater sensitivity to harness choice among weaker backends. That makes harness engineering particularly interesting for developers who cannot train a leading model themselves. They can still build tools, execution policies, feedback loops, and context management. The evidence supports taking that work seriously, while leaving local-model parity as a question to investigate.

Bhargava’s exploration led to Agency, a language for building agents. At the time of the talk, he describes it as roughly six months into development, motivated by capabilities he could not obtain from existing tools and frameworks. His proposed solution puts those capabilities into the language itself.

Here, an agent means a model plus its harness. In his example, Claude Code is the agent, Opus is the model, and the surrounding machinery is the harness. Agency supplies that surrounding machinery. Its design follows the Alan Kay principle that simple things should be simple and complex things possible: common agent operations should require little ceremony without ruling out more elaborate execution patterns.

The demonstration develops a coding agent through seven stages. Initially, the model and repair task stay the same while the harness changes; later, subagents deliberately broaden the assignment. Safety develops alongside capability throughout. An agent needs enough authority to act on the user’s behalf, but unrestricted authority would make that usefulness dangerous.

The repair target is a Python function in demo/median.py that calculates the median of an ordered list. For an odd-length list, the answer is the middle element. For an even-length list, it is the mean of the middle two elements. The existing function gets the latter case wrong, and an existing test exposes the bug. A compact implementation of the intended behavior for nonempty, ordered inputs is:

def median(numbers):
    middle = len(numbers) // 2
    if len(numbers) % 2:
        return numbers[middle]
    return (numbers[middle - 1] + numbers[middle]) / 2


assert median([1, 3, 5]) == 3
assert median([1, 3, 5, 7]) == 4

The illustrative assertions make the distinction concrete: selecting one middle element is insufficient when the list has two central values.

The first Agency program does almost nothing around the model. Agency resembles TypeScript with some Python-inspired syntax; its entry point is node main. This program asks the LLM to fix the Python median function, receives a result, and prints it. The model responds that it needs to see the existing code. Naming a file in a prompt does not grant access to it.

Before adding filesystem access, Bhargava demonstrates the tool interface with a greet function. Any Agency function can be passed directly as a tool: Agency generates its JSON Schema and uses its docstring as the tool description. The function’s parameters become the arguments the model can supply.

Running the example through npx produces a greeting, but the greeting alone would not establish that a tool ran—the model could generate one itself. Agency’s built-in log viewer exposes the actual sequence. In the inspected terminal frame, greet receives the name Adit, the tool returns Howdy, Adit!, and the assistant returns the same text. The execution trace distinguishes an action from a plausible description of one.

The next version passes Agency’s built-in read and write functions directly to the model. That creates the ability to request filesystem operations, but unrestricted reads and writes would be unsafe. Agency refuses to proceed without an approval mechanism. The example stops with an unhandled standard-read interrupt; the error asks whether the user wants to read the requested file, identifies that file, and points to the handlers guide.

A handle block supplies the missing policy. It wraps the LLM call and connects raised interrupts to a handler function. The handler prints the interrupt information, obtains a response using the built-in input function, and approves the operation when the user agrees. An interrupt pauses execution so that a decision can happen before the operation proceeds. Bhargava explains that standard-library operations involving mutations, destructive actions, or sensitive reads raise interrupts first.

In the live run, he approves the read, the agent reads the file, and then a write request follows. This establishes a usable human approval path, but it also puts the human on the critical path for each action. The next requirement is more demanding: let the agent work autonomously within an appropriate scope without handing it destructive access to the rest of the filesystem.

Partial function application fixes selected arguments before a function is called. Agency’s read tool accepts a filename and a directory. Calling .partial with the directory set to demo produces a tool whose directory is already chosen by the program. The model sees only the filename parameter; it neither sees nor supplies the bound directory argument. The same approach applies to writes.

This changes the capability exposed to the model. Instead of choosing both where to operate and which file to use, it chooses a filename within a scope supplied by the harness. Bhargava describes the resulting tools as confined to demo and suitable for use without repeated human approval. Actual filesystem confinement also depends on the file tool enforcing that boundary, including path traversal, absolute paths, and symlinks; binding an argument alone does not establish those protections.

The run now reads the file without asking for permission and produces the correct proposed repair. Yet it stops short of applying it, asking whether the user wants the file updated. The harness has removed an access obstacle and an approval bottleneck, but the task is still incomplete: a proposed fix is not an edited file.

The next change introduces ReAct: reasoning and acting interleaved with observations from the environment. For the median repair, the loop gives the agent a concrete procedure:

1. Read the implementation and its test file.
2. Run the tests to observe the failure.
3. Use the failure to decide what to change.
4. Write the repair.
5. Run the tests again, then continue if they still fail.

The harness changes are a revised prompt and the ability to run the tests. The agent now has both an action it can take and an observable condition for deciding whether its work succeeded.

Bhargava enables printed tool calls to make the execution visible. After a brief interruption in the live demonstration, the trace shows the intended sequence: the agent reads both files, runs the tests, observes failure, writes the implementation, reruns the tests, and confirms success. This is the first version that completes the repair and checks the result.

Each harness change has solved a different problem:

The final step supplies a completion process, not just more permissions. The agent must compare the consequences of its changes with the task’s test results.

The subagent example changes the assignment. Alongside fixing the failing test in test_median.py, the agent is asked to research Jensen’s inequality for medians using a Wikipedia agent and explain it in limerick form. This exercises a broader set of capabilities than the original repair.

The top-level agent no longer receives raw file tools. It receives two functions that act as subagents. In Agency, a subagent is an ordinary function containing its own LLM call, so the same function-as-tool interface works at both levels:

• Coding subagent: Keeps its own system message, LLM call, and tools for reading, writing, and running tests.
• Wikipedia subagent: Has its own LLM call and the standard library’s Wikipedia search tool.

Callbacks print the tool-call information, preserving visibility into the delegated work.

In the demonstration, the top-level agent calls the two subagents in parallel. The Wikipedia agent searches while the coding agent modifies the file, and the result includes a repair and a limerick. This shows delegation and combined output; the mathematical correctness of the generated explanation is not established by the demonstration.

The architectural benefit is separation of context and capabilities. An agent with many unrelated tools must choose among them while processing unrelated concepts in the same context. Subagents let the parent make a coarser decision—coding or research—while each specialist works with a smaller, relevant tool set. Bhargava identifies context bloat and unrelated tools as recurring sources of confusion; grouping them by responsibility gives the model a cleaner decision to make.

The final example uses Agency’s built-in GEPA optimizer, one of several optimizers Bhargava says the language provides. Variables marked with the optimize modifier become candidates for revision. Here, two prompts begin with very little information, and the optimizer receives the goal of fixing median.py using test-driven development.

The optimizer identifies the marked prompts and runs the agent to establish a baseline before attempting improvements. The live optimization run reports a baseline objective value of 0.2 for the median-repair task. The scoring definition is not supplied, so that value should be read as the demonstration’s objective score, not as an accuracy percentage.

Because optimization takes time, Bhargava switches to an earlier completed run. He reports that it achieved its objective in the first iterations and shows a rewritten prompt. The useful shift is from informal prompt tweaking to systematic evaluation and reflection. GEPA still proposes and tests candidates; it organizes experimentation around execution feedback and a measured objective rather than eliminating trial and error.

By this point, the seven-stage progression has moved from a model that cannot read the target file to an agent with tools, approval handling, scoped autonomy, test feedback, specialized subagents, and prompt optimization. The last two stages broaden what the agent can do and provide a process for improving it. Whether that support can make a local model sufficient for a particular workload remains the motivating experiment: define the task, improve the harness, and measure whether the resulting agent meets the need.

The strongest justification for language-level support arrives at the end: true pause and resume. Function-based tools, interrupts, handlers, and partial application simplify common harness operations, but an interrupt also needs to preserve a computation that may be deeply nested. Agency’s intended behavior is to return control to the user while retaining the execution state needed to continue later.

Consider an interrupt raised inside a for loop, inside a tool call, inside a subagent. Bhargava says Agency can pause that computation and later resume at the exact point inside the loop. The caller should not need to flatten the program or manually reconstruct its position just to obtain approval. He contrasts this with restrictions he has encountered in framework-level human approval features, while qualifying his comparison with other languages as based on what he knows.

He further describes serializing execution and returning to it a week later without changing the demonstrated code. That makes human approval an interruption of an ongoing computation rather than a requirement to start the task again. Together with built-in optimizers, these are the language features intended to support more ambitious harnesses than a single prompt-and-tools call.

To try Agency, the installation command visible on the slide and given in the current project README is:

pnpm install agency-lang

Bhargava describes the package and an API key as the starting requirements. The linked README is current documentation rather than a pinned release of the demonstration. From there, the practical experiment is the same one developed through the median repair: keep the task concrete, give the agent appropriate authority, and use observable results to improve the harness around the model.