# What if the harness mattered more than the model? - Aditya Bhargava, Etsy

Aditya Bhargava

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/2e9ANoOEn28-what-if-harness-mattered-more-than-model) · [Video](https://www.youtube.com/watch?v=2e9ANoOEn28)

## What if the harness mattered more than the model?

A coding agent’s journey from an inaccessible file to a tested repair shows how tools, permissions, feedback, subagents, and prompt optimization change what a model can accomplish.

Familiarity with basic Python functions, automated tests, and LLM tool calling will help; no prior knowledge of Agency is required.

### Could a better harness make local models good enough?

[0:11](https://www.youtube.com/watch?v=2e9ANoOEn28&t=11s)

What would it take to get the performance you need from a local, open-source model instead of depending on a powerful proprietary one? Aditya Bhargava, who introduces himself as an Etsy staff engineer and IC initiative lead for agentic commerce, approaches that question through harness design. His background includes writing *Grokking Algorithms* and illustrated technical posts on `ducktyped.org`. The engineering question is whether more of an agent’s capability can come from software developers control.

The familiar advice is to keep the harness simple: give a strong model a few tools and let it work. That can be effective, but it makes the model provider responsible for more of the system’s intelligence. **Investing in the harness offers another direction:** build enough support around a model that a locally runnable alternative becomes useful for the task. Matching cutting-edge performance this way is the ambition that motivates the experiments.

![Controversial take slide questioning reliance on proprietary models and asking whether a strong harness could enable cutting-edge performance with a local open-source model.](https://ai.engineer/api/talk-essay-media/b1d0cb2b982b4fed66819b4a84ec2f31e285c31ea2a532d0d5e3ecce1cc05d04)

Could a better harness bring cutting-edge performance to a local open-source model?

### What changes when the harness changes?

[1:45](https://www.youtube.com/watch?v=2e9ANoOEn28&t=105s)

[Harness-Bench](https://arxiv.org/html/2605.27922v1) gives this question an empirical starting point. Bhargava introduces it as a way to compare harnesses while keeping models and evaluation conditions consistent. The matching paper evaluates 106 tasks across eight model backends. Its aggregate composite scores range from 52.4 for OpenClaw to 76.2 for NanoBot, a 23.8-point spread. Those are averages across backends, rather than one model’s success rates; the composite combines completion, security, and process quality, with an LLM judge contributing to process assessment. The comparison measures complete harness configurations, so it does not isolate which individual feature produced the difference.

The study also observes greater sensitivity to harness choice among weaker backends. That makes harness engineering particularly interesting for developers who cannot train a leading model themselves. They can still build tools, execution policies, feedback loops, and context management. The evidence supports taking that work seriously, while leaving local-model parity as a question to investigate.

### A language for the harness

[3:13](https://www.youtube.com/watch?v=2e9ANoOEn28&t=193s)

Bhargava’s exploration led to [Agency](https://github.com/egonSchiele/agency-lang/blob/main/README.md), a language for building agents. At the time of the talk, he describes it as roughly six months into development, motivated by capabilities he could not obtain from existing tools and frameworks. His proposed solution puts those capabilities into the language itself.

Here, an **agent** means a model plus its **harness**. In his example, Claude Code is the agent, **Opus** is the model, and the surrounding machinery is the harness. Agency supplies that surrounding machinery. Its design follows the Alan Kay principle that simple things should be simple and complex things possible: common agent operations should require little ceremony without ruling out more elaborate execution patterns.

![The Agency Language slide showing agency-lang.com, an Alan Kay quote about simplicity and complexity, and pnpm install agency-lang.](https://ai.engineer/api/talk-essay-media/98e4b4ce82749acbc0e42a38ba7c3e782cf460fe8bd09167b90129408f5fd990)

Agency introduces its language and the principle that simple things should be simple.

The demonstration develops a coding agent through seven stages. Initially, the model and repair task stay the same while the harness changes; later, subagents deliberately broaden the assignment. Safety develops alongside capability throughout. An agent needs enough authority to act on the user’s behalf, but unrestricted authority would make that usefulness dangerous.

### A median bug, then an observable tool call

[7:45](https://www.youtube.com/watch?v=2e9ANoOEn28&t=465s)

The repair target is a Python function in `demo/median.py` that calculates the median of an ordered list. For an odd-length list, the answer is the middle element. For an even-length list, it is the mean of the middle two elements. The existing function gets the latter case wrong, and an existing test exposes the bug. A compact implementation of the intended behavior for nonempty, ordered inputs is:

```python
def median(numbers):
    middle = len(numbers) // 2
    if len(numbers) % 2:
        return numbers[middle]
    return (numbers[middle - 1] + numbers[middle]) / 2


assert median([1, 3, 5]) == 3
assert median([1, 3, 5, 7]) == 4
```

The illustrative assertions make the distinction concrete: selecting one middle element is insufficient when the list has two central values.

The first Agency program does almost nothing around the model. Agency resembles TypeScript with some Python-inspired syntax; its entry point is `node main`. This program asks the LLM to fix the Python median function, receives a result, and prints it. The model responds that it needs to see the existing code. Naming a file in a prompt does not grant access to it.

Before adding filesystem access, Bhargava demonstrates the tool interface with a `greet` function. Any Agency function can be passed directly as a tool: Agency generates its JSON Schema and uses its docstring as the tool description. The function’s parameters become the arguments the model can supply.

Running the example through `npx` produces a greeting, but the greeting alone would not establish that a tool ran—the model could generate one itself. Agency’s built-in log viewer exposes the actual sequence. In the inspected terminal frame, `greet` receives the name `Adit`, the tool returns `Howdy, Adit!`, and the assistant returns the same text. **The execution trace distinguishes an action from a plausible description of one.**

![Terminal log showing greet called with name Adit, followed by tool and assistant responses reading “Howdy, Adit!”](https://ai.engineer/api/talk-essay-media/b473a71f42a7d884494767279fb6f2d6200d7a052ac2092999c0e90f3fb6ec68)

The expanded log shows the greet tool call, its response, and the assistant returning the same greeting.

### File access needs an approval policy

[11:26](https://www.youtube.com/watch?v=2e9ANoOEn28&t=686s)

The next version passes Agency’s built-in `read` and `write` functions directly to the model. That creates the ability to request filesystem operations, but unrestricted reads and writes would be unsafe. Agency refuses to proceed without an approval mechanism. The example stops with an unhandled standard-read interrupt; the error asks whether the user wants to read the requested file, identifies that file, and points to the handlers guide.

A `handle` block supplies the missing policy. It wraps the LLM call and connects raised interrupts to a handler function. The handler prints the interrupt information, obtains a response using the built-in `input` function, and approves the operation when the user agrees. An **interrupt** pauses execution so that a decision can happen before the operation proceeds. Bhargava explains that standard-library operations involving mutations, destructive actions, or sensitive reads raise interrupts first.

In the live run, he approves the read, the agent reads the file, and then a write request follows. This establishes a usable human approval path, but it also puts the human on the critical path for each action. The next requirement is more demanding: let the agent work autonomously within an appropriate scope without handing it destructive access to the rest of the filesystem.

### Bind the scope before exposing the tool

[15:44](https://www.youtube.com/watch?v=2e9ANoOEn28&t=944s)

**Partial function application** fixes selected arguments before a function is called. Agency’s `read` tool accepts a filename and a directory. Calling `.partial` with the directory set to `demo` produces a tool whose directory is already chosen by the program. The model sees only the filename parameter; it neither sees nor supplies the bound directory argument. The same approach applies to writes.

![Partial function application slide showing read and write tools bound to the demo directory, the read function signature, and explanatory bullets.](https://ai.engineer/api/talk-essay-media/4b7647a68b484bfdd6106b5cd015931cfa8a9d16ea49364c9875e083489999bb)

Partial application binds the tools’ directory argument to demo.

This changes the capability exposed to the model. Instead of choosing both where to operate and which file to use, it chooses a filename within a scope supplied by the harness. Bhargava describes the resulting tools as confined to `demo` and suitable for use without repeated human approval. Actual filesystem confinement also depends on the file tool enforcing that boundary, including path traversal, absolute paths, and symlinks; binding an argument alone does not establish those protections.

The run now reads the file without asking for permission and produces the correct proposed repair. Yet it stops short of applying it, asking whether the user wants the file updated. The harness has removed an access obstacle and an approval bottleneck, but the task is still incomplete: **a proposed fix is not an edited file.**

### Close the loop with failing and passing tests

[18:01](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1081s)

The next change introduces [ReAct](https://arxiv.org/abs/2210.03629): reasoning and acting interleaved with observations from the environment. For the median repair, the loop gives the agent a concrete procedure:

1. Read the implementation and its test file.
2. Run the tests to observe the failure.
3. Use the failure to decide what to change.
4. Write the repair.
5. Run the tests again, then continue if they still fail.

The harness changes are a revised prompt and the ability to run the tests. The agent now has both an action it can take and an observable condition for deciding whether its work succeeded.

Bhargava enables printed tool calls to make the execution visible. After a brief interruption in the live demonstration, the trace shows the intended sequence: the agent reads both files, runs the tests, observes failure, writes the implementation, reruns the tests, and confirms success. This is the first version that completes the repair and checks the result.

Each harness change has solved a different problem:

| Harness stage | What it enables | Remaining obstacle |
|---|---|---|
| Model alone | A conversational response | Cannot access the file |
| File tools | Requests to read and write | Unhandled approval interrupts |
| Approval handlers | Authorized operations | Human input slows progress |
| Partial application | Scoped autonomous access | Stops at a proposed repair |
| Feedback loop | Applies and tests the repair | Confidence remains tied to the tests |

The final step supplies a completion process, not just more permissions. The agent must compare the consequences of its changes with the task’s test results.

### Expand capability without mixing every tool together

[21:29](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1289s)

The subagent example changes the assignment. Alongside fixing the failing test in `test_median.py`, the agent is asked to research Jensen’s inequality for medians using a Wikipedia agent and explain it in limerick form. This exercises a broader set of capabilities than the original repair.

The top-level agent no longer receives raw file tools. It receives two functions that act as subagents. In Agency, a subagent is an ordinary function containing its own LLM call, so the same function-as-tool interface works at both levels:

- **Coding subagent:** Keeps its own system message, LLM call, and tools for reading, writing, and running tests.
- **Wikipedia subagent:** Has its own LLM call and the standard library’s Wikipedia search tool.

Callbacks print the tool-call information, preserving visibility into the delegated work.

In the demonstration, the top-level agent calls the two subagents in parallel. The Wikipedia agent searches while the coding agent modifies the file, and the result includes a repair and a limerick. This shows delegation and combined output; the mathematical correctness of the generated explanation is not established by the demonstration.

The architectural benefit is **separation of context and capabilities**. An agent with many unrelated tools must choose among them while processing unrelated concepts in the same context. Subagents let the parent make a coarser decision—coding or research—while each specialist works with a smaller, relevant tool set. Bhargava identifies context bloat and unrelated tools as recurring sources of confusion; grouping them by responsibility gives the model a cleaner decision to make.

![Subagents slide listing codingAgent for reading, writing and running tests, and wikipediaAgent for general knowledge, with bullets about context bloat and consistency.](https://ai.engineer/api/talk-essay-media/d4a0699ce2e80fc58849c92b3cb348a7bc6fc2d9ff28e9be05d92e75b07cd566)

Coding and Wikipedia subagents provide separate capabilities as tools.

### Measure and revise the prompts

[25:21](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1521s)

The final example uses Agency’s built-in [GEPA optimizer](https://arxiv.org/abs/2507.19457), one of several optimizers Bhargava says the language provides. Variables marked with the `optimize` modifier become candidates for revision. Here, two prompts begin with very little information, and the optimizer receives the goal of fixing `median.py` using test-driven development.

The optimizer identifies the marked prompts and runs the agent to establish a baseline before attempting improvements. The live optimization run reports a baseline objective value of 0.2 for the median-repair task. The scoring definition is not supplied, so that value should be read as the demonstration’s objective score, not as an accuracy percentage.

Because optimization takes time, Bhargava switches to an earlier completed run. He reports that it achieved its objective in the first iterations and shows a rewritten prompt. The useful shift is from informal prompt tweaking to **systematic evaluation and reflection**. GEPA still proposes and tests candidates; it organizes experimentation around execution feedback and a measured objective rather than eliminating trial and error.

By this point, the seven-stage progression has moved from a model that cannot read the target file to an agent with tools, approval handling, scoped autonomy, test feedback, specialized subagents, and prompt optimization. The last two stages broaden what the agent can do and provide a process for improving it. Whether that support can make a local model sufficient for a particular workload remains the motivating experiment: define the task, improve the harness, and measure whether the resulting agent meets the need.

### Preserve execution through a human decision

[29:30](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1770s)

The strongest justification for language-level support arrives at the end: **true pause and resume**. Function-based tools, interrupts, handlers, and partial application simplify common harness operations, but an interrupt also needs to preserve a computation that may be deeply nested. Agency’s intended behavior is to return control to the user while retaining the execution state needed to continue later.

Consider an interrupt raised inside a `for` loop, inside a tool call, inside a subagent. Bhargava says Agency can pause that computation and later resume at the exact point inside the loop. The caller should not need to flatten the program or manually reconstruct its position just to obtain approval. He contrasts this with restrictions he has encountered in framework-level human approval features, while qualifying his comparison with other languages as based on what he knows.

He further describes serializing execution and returning to it a week later without changing the demonstrated code. That makes human approval an interruption of an ongoing computation rather than a requirement to start the task again. Together with built-in optimizers, these are the language features intended to support more ambitious harnesses than a single prompt-and-tools call.

To try Agency, the installation command visible on the slide and given in the current project README is:

```sh
pnpm install agency-lang
```

Bhargava describes the package and an API key as the starting requirements. The linked README is current documentation rather than a pinned release of the demonstration. From there, the practical experiment is the same one developed through the median repair: keep the task concrete, give the agent appropriate authority, and use observable results to improve the harness around the model.

![Slide listing simple tool definitions, safety tools, true pause-and-resume execution, built-in optimizers, and the Agency installation command.](https://ai.engineer/api/talk-essay-media/1b1c5dd998c703b478e2f1344b73d15e5e9aab6dac405c7581fa9f9a86bdf517)

Agency lists pause-and-resume execution among its language-level features.

## Resources

- [Harness-Bench](https://arxiv.org/html/2605.27922v1) — A study of harness effects across 106 sandboxed workflow tasks, with composite scoring and a six-harness, eight-model evaluation matrix.
- [Harness-Bench code and benchmark assets](https://github.com/Qihoo360/harness-bench) — Public benchmark assets for evaluating agent frameworks in filesystem workspaces.
- [Agency installation and first program](https://github.com/egonSchiele/agency-lang/blob/main/README.md) — Getting-started instructions for Agency, including the agency-lang package and a minimal program.
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629) — The original ReAct paper explains how reasoning and actions interleave with observations from an external environment.
- [GEPA: Reflective Prompt Evolution](https://arxiv.org/abs/2507.19457) — Prompt optimization through reflection on execution trajectories and evaluation of revised candidates.

## Slides

### About me

Aditya Bhargava is a staff engineer and IC initiative lead for agentic commerce at Etsy. He wrote Grokking Algorithms and publishes illustrated posts at ducktyped.org.

[0:06](https://www.youtube.com/watch?v=2e9ANoOEn28&t=6s) [Introduction listing the speaker’s role, book, and blog.](https://assets.ai.engineer/sites/aiecode2025/sha256/626e16c2d755f458eab9564a6295f58ee20d4f4eb6fe95bd747226b08bffd3ed.webp)

About me

• Staff engineer at Etsy
• Also Etsy's IC Initiative Lead for Agentic Commerce
• Wrote Grokking Algorithms, an illustrated book on algorithms
• Blog at ducktyped.org

Aditya Bhargava  What if the harness mattered more than the model?

Interface text:
slides
-bash
slides
-bash

### Controversial take

Bhargava asks whether better harnesses could bring cutting-edge performance to local open-source models, reducing reliance on proprietary models.

[0:25](https://www.youtube.com/watch?v=2e9ANoOEn28&t=25s) [A proposed focus on locally run open-source models instead of relying on proprietary models.](https://assets.ai.engineer/sites/aiecode2025/sha256/4fb99d8e538af292fb9055ce5c125be6cd1e007bc9186d5e3cfccfcde0c2c6ee.webp)

Controversial take!

The models are so good, that you can keep the harness simple. Just give the model a few tools and it'll do the rest.

• Yes, but this makes us reliant on fancy proprietary models
• What if we focused on open source models that can be run locally?

Aditya Bhargava  What if the harness mattered more than the model?

Interface text:
slides
-bash  slides  -bash

### Harness-Bench results

Harness-Bench tested six harnesses on 106 tasks. Bhargava reports scores from 52.4% with OpenClaw to 76.2% with NanoBot, with the harness mattering more for weaker models.

[1:43](https://www.youtube.com/watch?v=2e9ANoOEn28&t=103s) [Harness-Bench setup and reported scores ranging from 52.4% to 76.2%, with a larger harness effect for weaker models.](https://assets.ai.engineer/sites/aiecode2025/sha256/b0d998a730af1fdc4c322e9849738cfd7b0101b7d752cb4f1ff55f62e14b10da.webp)

How much does the harness matter? Results from Harness-Bench

Setup
• 106 tasks
• 8 models (Anthropic, Google, OpenAI, Qwen, z.ai, Moonshot Kimi, DeepSeek)
• 6 harnesses (OpenClaw, ZeroClaw, Hermes, Moltis, NullClaw, NanoBot)
• Same setup, same evaluation

Results: scores ranged from 52.4% (OpenClaw) to 76.2% (NanoBot)!
• Only the harness changed.
• For weaker models, the harness mattered more

https://arxiv.org/html/2605.27922v1

Aditya Bhargava What if the harness mattered more than the model?

Interface text:
slides
-bash
slides
-bash

### Building a good harness

Bhargava found existing tools and frameworks insufficient for the harness he wanted to build. He argues that a good harness needs language-level support.

[2:42](https://www.youtube.com/watch?v=2e9ANoOEn28&t=162s) [The speaker set out to build a harness and found existing tools and frameworks insufficient for the intended purpose.](https://assets.ai.engineer/sites/aiecode2025/sha256/17d884fed6e933a5d76c75b97e1bed84464ee1248810d3ef4fa1838edf130379.webp)

Okay, lets try to build a really good harness

• I set out to build a really good harness
• I realized that none of the existing tools or frameworks can do what I want

Aditya Bhargava  What if the harness mattered more than the model?

Interface text:
slides
-bash  slides  -bash

### The Agency Language

Bhargava introduces Agency as a language for building agent harnesses. Its design aims to make common agent tasks simple while allowing more complex behavior.

[5:06](https://www.youtube.com/watch?v=2e9ANoOEn28&t=306s) [Slide introducing Agency, its design philosophy, and installation command.](https://assets.ai.engineer/sites/aiecode2025/sha256/5a1246b817452e665407acfe16caa70bf4df5a2547fad7e67ece99758707192c.webp)

The Agency Language

• agency-lang.com
• It's really good!
• "Simple things should be simple, complex things should be possible - Alan Kay"
• Lets you do a lot of cool things pretty easily

pnpm install agency-lang

Aditya Bhargava  What if the harness mattered more than the model?

Interface text:
slides
-bash

### Building a Coding Agent in Agency

Bhargava will build a coding agent across seven examples, keeping the model and task fixed while changing the harness. He will cover harness basics, safety, subagents and self-optimization.

[5:55](https://www.youtube.com/watch?v=2e9ANoOEn28&t=355s) [Slide outlining seven examples and goals including harness basics, safety, subagents, and self-optimization.](https://assets.ai.engineer/sites/aiecode2025/sha256/4167a6b32b9874708f07432c670cb2ecd59baf79d0270ecf0c60d2c3f9241df5.webp)

Rest of this talk: Lets build a coding agent in Agency

• We'll see the agent evolve over 7 examples
• Same model, same task, different harness each time
• We'll see the harness improve the agent's performance over time

Goals

• Understand the basics of building a harness
• Learn about agent safety
• Intro to more advanced concepts like subagents and self-optimization

Aditya Bhargava  What if the harness mattered more than the model?

Interface text:
slides
-bash
slides
-bash

### The Code to Edit

The agent will be asked to fix a median function that returns one middle element for an even-length list. The failing test expects 2.5 but gets 3; the correct calculation averages the two middle elements.

[7:46](https://www.youtube.com/watch?v=2e9ANoOEn28&t=466s) [Slide showing a median function that mishandles even-length lists and a test expecting 2.5 instead of 3.](https://assets.ai.engineer/sites/aiecode2025/sha256/d140a9087cacb6d2292ee8147624cecdb7ffbb14775f26a9a350e66f2022c76c.webp)

The code to edit

One bug:

demo/median.py:

def median(numbers):
    ordered = sorted(numbers)
    middle = len(ordered) // 2
    return ordered[middle]  # wrong for even-length lists!

demo/test_median.py fails:

AssertionError: median[illegible] returned 3, expected 2.5

Same model, same task for every example. Only the harness changes.

Aditya Bhargava  What if the harness mattered more than the model?

Interface text:
slides
-bash  slides  -bash

### 01 - Just the Model

The first Agency example sends a bug-fix prompt directly to the model and prints its response. Without file access, the model asks to see the code.

[8:27](https://www.youtube.com/watch?v=2e9ANoOEn28&t=507s) [Agency code sends a request to fix the median function to a model; the slide notes that it cannot read or write the file.](https://assets.ai.engineer/sites/aiecode2025/sha256/391f0f25843f72918dc30d2e2c28bacb0d915ca0a114a762d15bc03644e0a771.webp)

01 - Just the model

node main() {
  const prompt = """
  The Python function median(numbers) in demo/median.py has a bug. Please fix the bug.
  """
  const result = llm(prompt)
  print(result)
}

It can't read or write to the file.

npx agency examples/01-llm.agency

Aditya Bhargava  What if the harness mattered more than the model?

Interface text:
slides
-bash
slides
-bash

### Model-only example output

Asked to fix median, the model requests the existing code. The model-only example cannot read or write the file.

[9:20](https://www.youtube.com/watch?v=2e9ANoOEn28&t=560s) [Terminal output asks for the existing median function before attempting a fix.](https://assets.ai.engineer/sites/aiecode2025/sha256/6257218c58256d33a2156e56cce2c796e8fbdec47a54f1d77731e2ada389f335.webp)

ai-talk (main) $ npx agency examples/01-llm.agency
examples/01-llm.agency → examples/01-llm.js (in 109.36ms)
Running examples/01-llm.js...
---
To help you fix the bug in the `median` function within `demo/median.py`, I'll need to see the existing code for that function. Please provide the code, and I'll assist you in identifying and correcting the issue.

Interface text:
esbuild
esbuild
slides
-bash

### Intro to tools in Agency

The LLM call passes the greet function as a tool. Agency creates a JSON Schema from the function and uses its docstring as the tool description.

[9:44](https://www.youtube.com/watch?v=2e9ANoOEn28&t=584s) [A greet function is supplied as a tool to a model call; the slide says every function is a tool in Agency.](https://assets.ai.engineer/sites/aiecode2025/sha256/03103ef4229336cc05e0be061f8f88d463c48ba0687fb32d5a7d1ad257ff69aa.webp)

Intro to tools in Agency

export def greet(name: string): string {
  """
  Greet the user with a message. This tool is used to demonstrate how to create and use custom tools in an agent.
  """
  return "Howdy, ${name}!"
}

node main() {
  const result = llm("Use your `greet` tool to greet Adit", {
    tools: [greet]
  })
  print(result)
}

Every function is a tool in Agency.

Aditya Bhargava  What if the harness mattered more than the model?

Interface text:
slides
-bash  slides  -bash

### Entering the next example

The previous model response asks to see the median function’s code. Bhargava starts typing the read-write example, then corrects himself: he wants to discuss tools first.

[10:30](https://www.youtube.com/watch?v=2e9ANoOEn28&t=630s) [Terminal shows the previous model response and a command for a read-write example being entered.](https://assets.ai.engineer/sites/aiecode2025/sha256/91cdf3536d22dee40362373710d3d9bb645fc634223a7272be856c627f11f721.webp)

ai-talk (main) $ npx agency examples/01-llm.agency
examples/01-llm.agency → examples/01-llm.js (in 109.36ms)
Running examples/01-llm.js...
---
To help you fix the bug in the `median` function within `demo/median.py`, I'll need to see the existing code for that function. Please provide the code, and I'll assist you in identifying and correcting the issue.
ai-talk (main) $ npx agency examples/02-read-write.agency

Interface text:
-bash
slides
-bash

### Greeting tool trace

Agency’s log viewer shows the assistant calling greet with the name Adit, then passing the tool’s greeting straight back to the user.

[11:15](https://www.youtube.com/watch?v=2e9ANoOEn28&t=675s) [Agent log showing a greeting tool call, its response, and the assistant response.](https://assets.ai.engineer/sites/aiecode2025/sha256/3be64a6ebc98d3f3d20dffedd15ebd04f0c57ac3708a890d6d2ad54e309575bb.webp)

▸ threadCreated thread #0
▾ nodeExecution "main" (1.7s, 207 tok, $0.000)
  ▸ enterNode "main"
  ▾ llmCall gpt-4o-mini · "Use your `greet` tool to [illegible]
    [user] "Use your `greet` tool to greet Adit"
    [assistant] tool call: greet({"name":"Adit"})
    ▾ toolExecution greet (1ms)
      ▸ toolCallStart "greet"
      ▸ toolCall "greet" (1ms)
    [tool: greet] "Howdy, Adit![illegible]
    [assistant] "Howdy, Adit!"
    ▸ raw data
  ▸ exitNode
▸ agentEnd (1.7s)

### Tools, the unsafe way

The agent gets read and write tools to fix median.py, but Agency requires human approval for file access. With no approval supplied, the example will crash.

[11:26](https://www.youtube.com/watch?v=2e9ANoOEn28&t=686s) [Code passes read and write tools to an agent; bullets explain the need for human approval.](https://assets.ai.engineer/sites/aiecode2025/sha256/625b135d9d10bbd85dd7c642199e48e3413df1449ccb7b4ba31c42397096d66d.webp)

02 - Tools, the unsafe way

node main() {
  const prompt = """
  The Python function median(numbers) in demo/median.py has a bug. Please fix the bug.
  """

  const result = llm(prompt, {
    tools: [read, write]
  })
  print(result)
}

• Giving the agent the ability to read and write files is unsafe
• Agency's human-in-the-loop feature will require human approval
• We're not providing it anywhere, so this code will crash

Aditya Bhargava  What if the harness mattered more than the model?

Interface text:
slides
-bash  slides  -bash

### Unhandled read interrupt

The run stops at an unhandled read interrupt for demo/median.py because no approval was provided. The error points to a handler guide.

[12:31](https://www.youtube.com/watch?v=2e9ANoOEn28&t=751s) [Terminal reports an unhandled read interrupt and directs the user to handler guidance.](https://assets.ai.engineer/sites/aiecode2025/sha256/353167ba4e6fe0b56d6768e6d5059f32fef5b9d0dc10e7c6403b6cf03ce659a4.webp)

ai-talk (main) $ npx agency examples/02-read-write.agency
examples/02-read-write.agency → examples/02-read-write.js (in 77.77ms)
Running examples/02-read-write.js...
---

Interrupt "std::read" was not handled:
  Are you sure you want to read this file?
  {"dir":null,"filename":"demo/median.py","offset":0,"limit":0}

You need to handle your interrupts by wrapping them in a handler.
See the guide: https://agency-lang.com/guide/handlers.html
ai-talk (main) $

Interface text:
-bash
-bash  slides  -bash

[12:50](https://www.youtube.com/watch?v=2e9ANoOEn28&t=770s) [Terminal reports that a read interrupt was not handled and shows the requested file.](https://assets.ai.engineer/sites/aiecode2025/sha256/d5cbac9a694864865462b8f90d1e0689df8d5d03d2fd28695a46686608d534b9.webp)

examples/02-read-write.agency → examples/02-read-write.js (in [illegible]
Running examples/02-read-write.js...
---

Interrupt "std::read" was not handled:
  Are you sure you want to read this file?
  {"dir":null,"filename":"demo/median.py","offset":0,"limit":[illegible]

You need to handle your interrupts by wrapping them in a handl[illegible]
See the guide: https://agency-lang.com/guide/handlers.html
ai-talk (main) $

### Handlers

The handler prints each tool interrupt and asks the user to approve or reject it. Human approval makes the example safer, but slows the agent down.

[13:11](https://www.youtube.com/watch?v=2e9ANoOEn28&t=791s) [Code wraps read and write tools in a handler that asks the user to approve or reject an interrupt.](https://assets.ai.engineer/sites/aiecode2025/sha256/5ee3a6894acc4901f20e0f68c9df442ee6eaaf235d369ac89cef6461324fba03.webp)

03 - Handlers

handle {
  const result = llm(prompt, {
    tools: [read, write]
  })
} with (data) {
  print("Tool wants to run:")
  print(data.message)
  const approval = input("Approve? (y/n) ")
  if (approval.toLowerCase() === "y") {
    return approve()
  }
  return reject()
}

Ask for user input

This works, and it is safe, but it's also very slow.

Aditya Bhargava What if the harness mattered more than the model?

Interface text:
slides
-bash
slides
-bash

### Partial function application

The example fixes the read and write tools’ directory argument to "demo", so the agent can work there without approving each tool call. The model supplies the filename but cannot change the directory.

[15:46](https://www.youtube.com/watch?v=2e9ANoOEn28&t=946s) [Code fixes the directory argument for read and write tools, limiting their access to a designated directory.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b8aaf6bbb075c55fc5d2b84d80a37a7d41a5d89ca0bc6fafaa49bcaec5bca7f.webp)

04 - Partial function application (PFA)

handle {
  const result = llm(
    prompt,
    {
      tools: [read.partial(dir: "demo"), write.partial(dir: "demo")]
    },
  )
} with approve

// Signature of the read tool
read(filename: string, dir: string): Result

• We lock the dir argument to "demo"
• Now the agent can only read files in that directory.

No human input needed, but still safe.

Aditya Bhargava  What if the harness mattered more than the model?

Interface text:
slides
-bash  slides  -bash

### Agent proposes a median fix

The agent reads the file without asking permission and proposes an empty-list check, but asks whether to update the file instead of applying the fix.

[17:22](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1042s) [Terminal output proposes handling an empty list in a median function and asks whether to update the file.](https://assets.ai.engineer/sites/aiecode2025/sha256/8aeeddcfa3d9b4ba5a784db250cac85cba0b15e8fa265ffd04d77395fedee4b8.webp)

ai-talk (main) $ npx agency examples/04-pfa.agency
examples/04-pfa.agency → examples/04-pfa.js (in 80.69ms)
Running examples/04-pfa.js...
---
The `median` function you provided appears to be mostly correct. However, it may raise an error if the input list `numbers` is empty, as attempting to calculate the median of an empty list does not make sense.

To fix this issue, we can add a check at the beginning of the function that returns `None` or raises a `ValueError` if the input list is empty. Here's the corrected version:

```python
def median(numbers):
    """Return the median value of a list of numbers."""
    if not numbers:
        raise ValueError("The list is empty. Cannot compute median.")
    ordered = sorted(numbers)
    middle = len(ordered) // 2
    if len(ordered) % 2 == 0:
        return (ordered[middle - 1] + ordered[middle]) / 2
    return ordered[middle]
```

Would you like me to update the file with this corrected function?
ai-talk (main) $

Interface text:
-bash
-bash    slides    -bash

### The feedback loop

The prompt directs the coding agent to run tests, inspect failures, and fix the code in a loop, then run the tests once more to confirm success.

[18:04](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1084s) [Slide describes reading files, running tests, fixing failures, and repeating until tests pass, alongside a reason-act-observe loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/51e9d28ec533960f899d892954b5c1d80c72cef8e087549130f761b4cea65f56.webp)

05 - The feedback loop

const sysMessage = """
  You are a coding agent fixing a Python project. Steps:
  (1) read median.py and test_median.py;
  (2) run the tests with the exec tool using command 'python'
      and args ['test_median.py'];
  (3) if the tests fail, read the error, fix median.py with the write tool, then run the tests again;
  (4) repeat until the tests pass.

  Always finish by running the tests one final time to confirm they pass.
"""

• Test-driven development
• The agent thinks, acts, observes the consequences, and decides what to do next

Reason -> Act -> Observe -> Reason

Aditya Bhargava  What if the harness mattered more than the model?

Interface text:
slides
-bash
slides
-bash

### Running the feedback-loop agent

The agent reads the median implementation and its tests, then runs the test file. The tests pass, so the agent reports that there are no bugs to fix.

[19:30](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1170s) [Terminal shows an agent reading the median implementation and tests, then reporting that the tests passed.](https://assets.ai.engineer/sites/aiecode2025/sha256/835f29e6ac1bebcd4d31f3e9c85bba5e05bc5fab8cfc4c1dc4f1b3b515c1e53b.webp)

ai-talk (main) $ npx agency examples/05-feedback.agency
examples/05-feedback.agency → examples/05-feedback.js (in 141.25ms)
Running examples/05-feedback.js...
---
● read({"filename":"median.py","offset":0,"limit":0,"useAgentCwd":false})
"def median(numbers):\n    \"\"\"Return the median value of a list of numbers.\"\"\"\n    ordered = sorted(numbers)\n    middle = len(ordered) // 2\n    if len(ordered) % 2 == 0:\n        return (ordered[middle - 1] + ordered[middle]) / 2\n    return ordered[middle]\n"
● read({"filename":"test_median.py","offset":0,"limit":0,"useAgentCwd":false})
"from median import median\n\n\ndef ch[illegible](actual, expected):\n    if actual != expected:\n        raise AssertionError(f\"median() returned [illegible], expected {expected}\")\n\n\ncheck(median([3, 1, 2]), 2)  # odd length -> middle value\ncheck(median([7]), 7)  # single element\ncheck(median([1, 2, 3, 4]), 2.5)  # even length -> average of 2 and 3\ncheck(median([5, 2, 8, 1]), 3.5)  # even length -> average of 2 and 5\n\nprint(\"All tests passed ✅\")\n"
● exec({"command":"python","args":["test_median.py"],"timeout":30000,"stdin":null,"blockedCommands":null,"allowedPaths":null,"useAgentCwd":false})
{"stdout":"All tests passed ✅\n","stderr":"","exitCode":0}
The tests for the `median` function passed successfully, which means the function is working as intended. There are no bugs to fix in `median.py`. If you have any other requests or need further assistance, feel free to ask!
ai-talk (main) $

Interface text:
-bash
-bash
slides
-bash

## Full transcript

[0:01](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1s) Hi everyone. My name is Aadit, pronounced Aadit like a tax audit, um, and I'm here to talk to you about what if the harness mattered more than the model?

[0:11](https://www.youtube.com/watch?v=2e9ANoOEn28&t=11s) A little bit about me first. I'm a staff engineer at Etsy. I'm also Etsy's IC initiative lead for agentic commerce. I wrote Grokking Algorithms, which is an illustrated book on algorithms, uh, and I also do illustrated posts on ducktyped.org.

[0:27](https://www.youtube.com/watch?v=2e9ANoOEn28&t=27s) Um, and I wanna talk to you about

[0:32](https://www.youtube.com/watch?v=2e9ANoOEn28&t=32s) what if the harness mattered more than the model, by which I mean what if we should be paying more attention to building expertise and harnesses? Um, and my controversial take here is I hear a lot of people say, "Oh, the models are so good that you can just keep the harness simple.

[0:54](https://www.youtube.com/watch?v=2e9ANoOEn28&t=54s) Just give the model a few tools and it'll do the rest." And I've been hearing this so often that it has now, uh, started becoming the wisdom of the tech industry.

[1:06](https://www.youtube.com/watch?v=2e9ANoOEn28&t=66s) Uh, to which my answer is yes, but that's moving in the wrong direction because that is making us reliant on fancy proprietary models that can't be run locally. So what if we instead talked about the other direction?

[1:23](https://www.youtube.com/watch?v=2e9ANoOEn28&t=83s) What if we focused on open-source models that can be run locally? What if we focused on building a harness that is so good that we can get the performance of a cutting-edge model through a local open-source model?

[1:37](https://www.youtube.com/watch?v=2e9ANoOEn28&t=97s) So what if the harness mattered more than the model? What if our focus should be on the harness?

[1:45](https://www.youtube.com/watch?v=2e9ANoOEn28&t=105s) Um, how much does the harness matter, right? This is something I have been researching a lot in my spare time. Um, there's a really interesting paper that talks about HarnessBench, which is a benchmark for harnesses, which is something we haven't seen a lot of yet.

[2:02](https://www.youtube.com/watch?v=2e9ANoOEn28&t=122s) Um, it's got 106 tasks. Um, it's testing these different models and different harnesses. Uh, but the point is that it's running the same setup, it's doing the same evaluation, same model, but testing different harnesses.

[2:22](https://www.youtube.com/watch?v=2e9ANoOEn28&t=142s) Um, and the results are pretty interesting. So scores range from fifty-two point four percent to seventy-six point two percent, so more than a twenty-point difference, and only the harness changed.

[2:36](https://www.youtube.com/watch?v=2e9ANoOEn28&t=156s) Um, and the really interesting thing is that for weaker models, the harness matters more.

[2:44](https://www.youtube.com/watch?v=2e9ANoOEn28&t=164s) Um, if the harness matters more than the model, that's great news, uh, because if the model matters more, then we're dependent on a handful of companies who are able to build and train these models.

[2:58](https://www.youtube.com/watch?v=2e9ANoOEn28&t=178s) But if the, if a good harness can compensate and can make a weaker model perform better, then we can build our own harnesses, and that's something that any of us can do, and we don't have to depend on paid models.

[3:13](https://www.youtube.com/watch?v=2e9ANoOEn28&t=193s) Um, so I have been spending a lot of time and doing a lot of research specifically into what would it take to build a really good harness. Um, and

[3:29](https://www.youtube.com/watch?v=2e9ANoOEn28&t=209s) I realized early on that none of the existing tools or frameworks, uh, were able to do what I wanted, and I realized that what was actually needed was a language-level solution.

[3:43](https://www.youtube.com/watch?v=2e9ANoOEn28&t=223s) So this is my other controversial take in this talk is I think building a good harness actually requires language-level support. Um, and this is kind of where my research has gone in the last six months, and I've spent a good part of that six months building a new language called Agency, which is a language for

[4:07](https://www.youtube.com/watch?v=2e9ANoOEn28&t=247s) building agents. Um, and just to talk about the terminology there real quick, because the terms agent and harness aren't super crisp in the industry yet. They're not super well defined.

[4:24](https://www.youtube.com/watch?v=2e9ANoOEn28&t=264s) Um, but in this talk I'll kind of use them interchangeably. But when I talk about the agent, I really mean the harness and the model. So if you take Claude Code, for example, Claude Code is an agent.

[4:38](https://www.youtube.com/watch?v=2e9ANoOEn28&t=278s) It uses Opus, which is the model, and then everything else is the harness. Um, but you know, when we talk about Agency, it's really for building agents, but the thing it's providing is the harness.

[4:55](https://www.youtube.com/watch?v=2e9ANoOEn28&t=295s) Um, Agency is really good. Uh, you know, it's still new. Building a language is a lot of work, and it's only six months in. Uh, but I love this quote by Alan Kay, "Simple things should be simple.

[5:10](https://www.youtube.com/watch?v=2e9ANoOEn28&t=310s) Complex things should be possible." And that's something I love about Agency is that I think, you know, when people build agents, there are a lot of primitives, a lot of things that every agent

[5:26](https://www.youtube.com/watch?v=2e9ANoOEn28&t=326s) will need to be able to do, and those things should be simple to do. Um, and there's a lot of complex things that we want to be able to do with agents, and it should be possible to do those things in the framework of your choice, and with Agency it is possible.

[5:46](https://www.youtube.com/watch?v=2e9ANoOEn28&t=346s) Um, so now we have a clear goal, which is try to build the best possible harness, and we have a great tool to do it, which is Agency. Uh, so the rest of this talk I'm going to talk about building a coding agent in Agency.

[6:02](https://www.youtube.com/watch?v=2e9ANoOEn28&t=362s) And we'll see the same agent evolve over seven different examples. We'll see the same model, the same task, but the harness will improve each time, and we'll see the harness improve the agent's performance each time.

[6:19](https://www.youtube.com/watch?v=2e9ANoOEn28&t=379s) Um, so you know, like I said, I've been doing a ton of research on this. My goals with this talk are to give you, like, the basics of building a harness.

[6:29](https://www.youtube.com/watch?v=2e9ANoOEn28&t=389s) When we talk about building a good harness, what does that mean? You know, if you're starting from zero, what are the basics? Um, I'll cover a good bit about agent safety because, you know, when we talk about building agents, what we really want them to be able to do is take action on our behalf.

[6:50](https://www.youtube.com/watch?v=2e9ANoOEn28&t=410s) Um, but the big problem with that is how do you get-- how do you give agent capabilities, so they can take actions,

[7:01](https://www.youtube.com/watch?v=2e9ANoOEn28&t=421s) but not so much capability that they do something unsafe? So when we talk about agent quality and capability, agent safety goes hand in hand with that.

[7:14](https://www.youtube.com/watch?v=2e9ANoOEn28&t=434s) Um, at the end, I'll do an introduction to some of the more advanced concepts like subagents and self-optimization. I think that's where, you know, more interesting things come in, and we really start to talk about, like, what are all the possibilities, you know, because when you talk about building a good harness, there are so many things to

[7:34](https://www.youtube.com/watch?v=2e9ANoOEn28&t=454s) explore. And so I hope you stick with me till the end there, because that's where we get into some really interesting stuff.

[7:45](https://www.youtube.com/watch?v=2e9ANoOEn28&t=465s) Um, so this is the code we're going to ask our coding agent to edit. This is a code to calculate the median of an ordered

[7:57](https://www.youtube.com/watch?v=2e9ANoOEn28&t=477s) list of numbers. Um, and if it's an odd length list, you just take the middle element. If it's an even length list, you need to take the middle two elements and take the mean of those two elements.

[8:13](https://www.youtube.com/watch?v=2e9ANoOEn28&t=493s) But right now, this function doesn't do that. So there's a bug in this function. There's a test that shows the failure, um, and we're going to ask our agent to fix this code.

[8:30](https://www.youtube.com/watch?v=2e9ANoOEn28&t=510s) So let's go ahead and start. Let's just dive in. Um, this is the first example. This is Agency code now, and you'll notice that Agency looks a lot like TypeScript.

[8:42](https://www.youtube.com/watch?v=2e9ANoOEn28&t=522s) It's heavily inspired with T-- by TypeScript with some Python syntax thrown in. Um, so the entry point is this node main. I have a simple prompt, the Python function median in demo/median.py has a bug.

[8:58](https://www.youtube.com/watch?v=2e9ANoOEn28&t=538s) Please fix the bug. And I just pass that prompt into the LLM function, and I get the result, and I print the result. Um, and of course, this isn't going to work because it can't read or write to this file.

[9:13](https://www.youtube.com/watch?v=2e9ANoOEn28&t=553s) So let's run this example real quick. Great. So of course, as expected, it says, you know, "In order to help you fix the bug, I'll need to see the existing code."

[9:32](https://www.youtube.com/watch?v=2e9ANoOEn28&t=572s) Um, so this is just a model, very little harness, and of course, it can't do anything. So the most obvious improvement to any harness is give it some tools.

[9:45](https://www.youtube.com/watch?v=2e9ANoOEn28&t=585s) So real quick, let's talk about how tools work in Agency, because it's very simple. Um, I'll define a function. In this case, just for an example, I'm going to define a greet function that greets a user, and then I'll make an LLM call that says, "Use your greet tool to greet Aadit," and then I pass in that

[10:08](https://www.youtube.com/watch?v=2e9ANoOEn28&t=608s) tool. Um, and every function can automatically be used as a tool in Agency, so there is nothing else to do. Uh, Agency creates a JSON Schema out of this function and sends it to the LLM call.

[10:24](https://www.youtube.com/watch?v=2e9ANoOEn28&t=624s) The docstring here will get used as a tool description. So let's go ahead and run the second example.

[10:32](https://www.youtube.com/watch?v=2e9ANoOEn28&t=632s) So npx agency... Oops. Actually, npx agency example, and it's actually zero one B, because we're gonna talk about tools first.

[10:47](https://www.youtube.com/watch?v=2e9ANoOEn28&t=647s) So howdy, Aadit. And how can we know that it ran that tool and didn't just generate an example, right? So I'll just also show real quick,

[11:00](https://www.youtube.com/watch?v=2e9ANoOEn28&t=660s) um, Agency writes out logs, and it has this great log viewer built in. So I'll just expand this last run, and you can see here it's making this tool call to the greet function with these parameters.

[11:16](https://www.youtube.com/watch?v=2e9ANoOEn28&t=676s) The tool responds by returning howdy, Aadit, and the assistant just passes that straight back to us. Um,

[11:26](https://www.youtube.com/watch?v=2e9ANoOEn28&t=686s) so that's tools. Now let's look at using the tools in Agency. So again, this is, you know, the next step of our agent, which is give it tools. Um, here is an example where we're giving the agent read and write tools.

[11:48](https://www.youtube.com/watch?v=2e9ANoOEn28&t=708s) Again, read and write are just functions that are built into Agency, and because every function can be used as a tool, I can just pass this straight to the LLM.

[11:58](https://www.youtube.com/watch?v=2e9ANoOEn28&t=718s) Uh, except giving the agent the ability to read and write arbitrary files on our file system is really unsafe. So by default, Agency won't allow you to do this.

[12:12](https://www.youtube.com/watch?v=2e9ANoOEn28&t=732s) Agency has a human-in-the-loop feature that will require some sort of approval. We're not providing it anywhere, so this code will crash and print an error. So let me show you what that looks like.

[12:33](https://www.youtube.com/watch?v=2e9ANoOEn28&t=753s) Great. So interrupt standard read was not handled. Here's the interrupt message. "Are you sure you want to read this file?" You can see the file name it's trying to read, and it's basically telling us that we didn't approve this interrupt, and it's giving us a pointer to the guide where it tells us how to use handlers.

[12:54](https://www.youtube.com/watch?v=2e9ANoOEn28&t=774s) Um, so we started with just the model. We tried tools. Now the next step is to make the tools safe. So we're going to do exactly what that message asks us to do and use a handler.

[13:12](https://www.youtube.com/watch?v=2e9ANoOEn28&t=792s) So now here's the same code, you know, passing through that read and write, uh, those read and write functions as tools, um, but now it's wrapped in a handle block.

[13:25](https://www.youtube.com/watch?v=2e9ANoOEn28&t=805s) Uh, and it has a handler function down here. So now when those tools call or raise an interrupt,

[13:34](https://www.youtube.com/watch?v=2e9ANoOEn28&t=814s) this code is going to execute. So it'll print the information about that interrupt. Um, it'll ask the user for input using the built-in input function, and if the user approves, the tool call is approved.

[13:52](https://www.youtube.com/watch?v=2e9ANoOEn28&t=832s) And I know I'm saying interrupt, and I'm not really defining what that means, um, and I can get to that a little bit later, but it's essentially a way to pause execution at some point, um, and ask for human input.

[14:08](https://www.youtube.com/watch?v=2e9ANoOEn28&t=848s) And so interrupts are a great feature in Agency, and all the functions in the Agency standard library that mutate code or mutate something or have some sort of destructive action or read sensitive data, data all throw an interrupt first.

[14:29](https://www.youtube.com/watch?v=2e9ANoOEn28&t=869s) So here is the safe version of our agent, where the tools

[14:38](https://www.youtube.com/watch?v=2e9ANoOEn28&t=878s) will raise an interrupt by default, and now this code is going to ask the user, "Do you approve?" So let's run this example.

[14:53](https://www.youtube.com/watch?v=2e9ANoOEn28&t=893s) Great. "Are you sure you want to read this file?" I'll just say yes.

[14:58](https://www.youtube.com/watch?v=2e9ANoOEn28&t=898s) You can see it's reading. Now it's trying to write to that file. So, you know, that's an example of us

[15:08](https://www.youtube.com/watch?v=2e9ANoOEn28&t=908s) making the agent safer by asking for user input. And this is better, and it is safe, but it's also very slow. So the next thing we want to do is make the agent autonomous

[15:22](https://www.youtube.com/watch?v=2e9ANoOEn28&t=922s) while still keeping it safe, and that's the tricky part, right? How do you give it just enough capability so that you don't have to manually approve every single action without giving it, without giving it so much capability that it's, like, able to do something destructive with your file system?

[15:44](https://www.youtube.com/watch?v=2e9ANoOEn28&t=944s) Um, so Agency has a really cool feature to handle this. It, that's called partial function application. Um, it's a concept borrowed from functional programming. It's a really fancy term for a pretty simple concept.

[16:02](https://www.youtube.com/watch?v=2e9ANoOEn28&t=962s) So here's a signature of the read tool, right? It takes a file name and a directory, um, and when you read a file, it's going to look inside that directory for that file name.

[16:15](https://www.youtube.com/watch?v=2e9ANoOEn28&t=975s) And with partial function application, what we do is we take that read function and call .partial on it

[16:22](https://www.youtube.com/watch?v=2e9ANoOEn28&t=982s) and give it a directory. And what we're doing here is we're locking the directory argument to demo. Um, the LLM isn't going to be able to change that argument.

[16:34](https://www.youtube.com/watch?v=2e9ANoOEn28&t=994s) It's not even going to know that that argument exists. When we pass in the read tool now, it's just going to see one parameter, which is file name. It's going to pass in the file name, and it will read that file from this directory.

[16:49](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1009s) So we've locked the directory parameter. Now the agent can only read files from and write files to this directory. Uh, partial function application is a really great way to constrain the capabilities of your agent.

[17:07](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1027s) Now, no human input is needed, but it's still safe. So let's see that in action. So again, we run npx agency examples. PFA is partial function application. So here

[17:23](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1043s) it's thinking, and you can see it read the file, didn't need to ask us for permission, and now it said, you know, to fix this issue, it's kind of giving us a new code, but it's not updating the code.

[17:38](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1058s) It asks, "Would you like me to update the file with this corrected function?" But it didn't do that itself. So, you know, this version of this agent is the best one we've seen so far because it does give us the correct solution without any human input needed, but it hasn't actually fixed the code.

[18:01](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1081s) So the next step is to get it to actually fix the code. Um, and that's where the feedback loop comes in. Um, so there's a very popular agent pattern called ReAct, and ReAct stands for reason and act.

[18:18](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1098s) And essentially, you're asking the agent to work in this loop where you reason, act, observe the consequences of your action, and then decide what to do next. So in this case, reason, read the two files, act, run the tests,

[18:40](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1120s) observe if the tests fail, read the error, and then reason again. So look at that test failure and think about what you want to do next, and just do this in a loop until the tests pass.

[19:00](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1140s) So this is a really key pattern to making the agent better. And so now let's see the same agent. The only thing we're changing is the prompt and giving it the ability to run these tests, and now the agent should work better.

[19:24](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1164s) In this case, I'm gonna also print all the tool calls just so you can see what's happening. Um...

[19:47](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1187s) Well, shoot. Maybe... [laughs] All right. Hang on. There's always where... Uh-huh.

[20:02](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1202s) There we go. Okay. Now let's see it in action. Um, so now you can see it's calling the tools. It's reading those two files, like we said. It's running the tests.

[20:16](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1216s) The tests fail. It's writing to the file. It's running the tests again and confirming success. So this is a really important

[20:27](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1227s) stage in the harness development because now we finally have an agent that successfully

[20:37](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1237s) figures out the task, checks that the code is failing,

[20:43](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1243s) modifies the code, and checks that the code now passes. Um, so let's talk about how we have improved the harness so far. So we started with just the model, couldn't do anything.

[20:57](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1257s) Then we added tools, but the agent wasn't safe. Then we added handlers, which added safety but needed human input. Then we added partial function application, which was safety without requiring human input.

[21:14](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1274s) And finally, the last one that we just looked at was the feedback loop, which added reasoning. So the agent made the change and confirmed that the change worked.

[21:29](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1289s) So I have two more examples. This one, we're going to go in a slightly different direction, and this is where we're getting into some of the more advanced concepts.

[21:38](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1298s) So this one is subagents. Um, everything so far we've done has made the same task better. But with subagents, we're asking, how do you make the agent do more things?

[21:49](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1309s) Um, and let me show you what this code looks like real quick.

[22:02](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1322s) Okay, so this program, I'll start by just showing you the prompt we're giving because it's slightly different. So fix the failing test in test_median.py, then research Jenso-Jensen's inequality for medians with the Wikipedia agent and explain it to me in limerick form.

[22:22](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1342s) So here is a case where we're actually asking the agent to do a little more, right? And this one is interesting because this is our main agent, but for tools, we're not giving it r-the read and write functions.

[22:38](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1358s) Instead, we're giving it two subagents as tools. And I really like this pattern in Agency because I've seen a lot of frameworks make subagents their own concept and make them kind of hard to grasp.

[22:55](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1375s) Um, in Agency, a subagent is just another function, and you just call it, use it like a tool, just like you would anywhere else. And that consistency has a lot of benefits, makes it really easy to write and reason about.

[23:09](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1389s) So here's the coding agent subagent. Um, same one as before, you know, there's a system message.

[23:19](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1399s) This subagent has its own LLM call, gets its own set of tools. Similarly, you have the Wikipedia agent. That has its own tool, which is the search tool, uh, which is a tool built into this Agency standard library that lets you search Wikipedia.

[23:36](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1416s) Uh, so Agency has a big standard library that lets you do all kinds of things, including this Wikipedia search tool. I'm also using the callback feature to print the tool call information.

[23:49](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1429s) And now let's see this example in action, and this is an example where we're actually asking the agent to do a little bit more, and we're giving it two subagents to accomplish its task.

[24:04](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1444s) So here you can see it's calling those two subagents. Uh, it's calling them in parallel, so the Wikipedia agent is doing the search, the coding agent is modifying the file, and That fixed the error, and now here is a limerick.

[24:21](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1461s) So subagents are cool because they let you add new capabilities without bloating context. Um, and this is a good pattern because it allows your agent, when it has a lot of different tools at its disposal, it allows your agent to do the right

[24:46](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1486s) thing more consistently. Often when agents get confused and fail, it's because they have too much context bloat, but also because they have too many unrelated concepts in their context, and they have too many tools that are unrelated, and so they have a hard time picking the right tool to use.

[25:06](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1506s) In this case, we make that problem a lot cleaner. For the top-level agent, we just say, "Pick the subagent you want to call," and then the subagents have those tools that are kind of grouped in a way that makes sense.

[25:21](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1521s) Um, so now let's look at the last example, which is self-optimization.

[25:28](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1528s) Uh, Agency has a GEPA optimizer built in. It actually has a few optimizers built in, but the GEPA one, uh, is a really interesting one. And the way the optimization feature works is you mark any variables that you want the optimizer to optimize.

[25:49](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1549s) You can just use this optimize modifier, and then you run the optimizer and give it a goal. So, you know, in this case, you can see that the two initial prompts I have are very basic.

[26:03](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1563s) I'm giving it hardly any information. And then the goal I give it is fix the bug in median.py using TDD. And the reason this is such a great step for our harness is we're no longer doing trial and error.

[26:22](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1582s) We're no longer just trying stuff to see what works. Instead, now we have a way to systematically measure and improve performance. So I'm just gonna go ahead and run this code, um, and I'm actually gonna also reset.

[26:42](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1602s) Here, I'm gonna modify this code. Cool. So now... [clears throat]

[26:59](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1619s) And I won't show the full run here because the optimizer takes a while, but you can see first it's, like,

[27:09](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1629s) checking what prompts it has that it needs to optimize. It's going to run the agent to establish a baseline. So the baseline objective is zero point two, and it's going to try to improve upon that.

[27:25](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1645s) So in this case, I did run the optimizer earlier just so I could show you what a full run looked like. Um, and so here is one where you can see

[27:37](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1657s) it achieved the objective in the first iterations. It was actually quite quick in this case, um, and you can see that it's rewritten this prompt. Um,

[27:51](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1671s) so the reason this self-optimization is so great is because we're not guessing and checking. We're systematically measuring and improving, which is a big leap forward.

[28:04](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1684s) So this is the ladder we just went through for improving the harness. Um, you know, first step was nothing, just the model, can't even read the file. Then we added tools, but in a way that was unsafe and not allowed in Agency.

[28:19](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1699s) Next, we added handlers, which added safety but made things slower because they required human input. Then we added partial function application, which was safe and fast. Then we added reasoning, which gave us higher confidence in the output of the agent.

[28:39](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1719s) Uh, and then we switched tracks to subagents, added more capabilities, and finally, we did self-optimization, so we got measured improvement and not guess and check. So, you know, I hope this gives you an idea of how you can build harnesses and then slowly improve those harnesses.

[29:03](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1743s) And I hope, you know, you try doing that and come along on this, you know, experimentation with me to see if we can build a better harness, um, because what if the harness matters more than the model?

[29:17](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1757s) You know, what if we can build a harness that's good enough that we can now start using local models and still get the performance we need for the tasks we wanna do.

[29:30](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1770s) So what language level support does Agency provide? You know, we used Agency for all these examples. How did it help?

[29:36](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1776s) Um, simple syntax for defining tools. Again, you know, the simple things should be simple. Every agent is going to need tools. Let's make it easy to use them. Tools for safety, very important for agents, so interrupts, handlers, PFA.

[29:52](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1792s) True pause and resume execution. I didn't cover this a ton 'cause I didn't wanna make this talk too long, um, but one of the really killer features of Agency is that when you raise that interrupt, it's actually going to pause execution and return control back to the user, but in a way where you can resume execution later.

[30:15](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1815s) Uh, and as far as I know, very few languages allow you to do something like this. And so typically, most frameworks you'll use, if they have a human-in-the-loop feature, it has a lot of restrictions.

[30:29](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1829s) There's a lot of things you need to do to work around it. Agency is completely different. If you wanted, you could, you know, raise interrupts inside of a for loop, inside of a tool call, inside of a subagent, um, and it would correctly pause execution, go back to the user, and then resume execution at that exact point

[30:52](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1852s) inside that subagent, inside that tool call, at that exact point in that for loop, um, which is really cool. You can serialize the execution. You can come back to it a week later, and you don't need to change anything about all the code we have been looking at.

[31:08](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1868s) It just works, which is really cool. Uh, it also has these built-in optimizers, which are great, you know, which kind of get you to a more analytical, mathematical way of doing the improvements.

[31:20](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1880s) So if you wanna try it out, please do. pnpm install agency lang. All you need is the package and a API key, and you're set. Um, again, takeaways, you know, an agent is a model plus a harness for this talk, at least.

[31:37](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1897s) Uh, better harness equals better performance, especially for weaker models. And then how do you build a better harness? Give it tools, make it safe, make it autonomous, make it reason, give it subagents, and have it self-optimize.

[31:52](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1912s) Thank you so much. Again, my name is Aadit Bhargava. Check out Agency, agencylang.com, uh, and then please follow me on Bluesky if you wish. Thank you. Take care.
