# Respect The Process

Andrew Dumit

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/CLttOU7n6sI-respect-process) · [Video](https://www.youtube.com/watch?v=CLttOU7n6sI)

## Respect the Process: Controlling What Coding Agents Can Change

Watershed’s graph-editing agent shows how a typed SDK and controlled execution can preserve coding flexibility while making consequential changes traceable and reviewable.

Familiarity with coding agents, typed APIs, and graph data structures will help; no carbon-accounting background is required.

### What are the emissions of one bottle of wine?

[0:00](https://www.youtube.com/watch?v=CLttOU7n6sI&t=0s)

What emissions should be attributed to a bottle of wine? If an industrial process produces several sellable co-products, how should its emissions be divided among them? These are everyday questions in product carbon accounting—the work Andrew Dumit describes doing at Watershed to measure emissions associated with what companies buy and sell. A plausible number is not enough: the allocation decisions and calculation method must also be defensible.

Dumit cites a 2020 study in which six experts received the same bottle-of-wine data and produced estimates that varied by up to 50%. His point is that reasonable expert judgment can produce different answers. **Correctness includes the process that justifies the answer.** Checking whether an agent’s final number resembles an expert’s number cannot establish that the agent followed an acceptable method.

![Slide on sustainability judgment calls, with wine and co-product allocation questions beside a 0.97–1.49 kgco2e range for six experts using the same data.](https://ai.engineer/api/talk-essay-media/6bc7729d60bb9d62ec18ca154ebe1711c69d3ec5cedd99461010d42490073ab9)

Six experts using the same bottle-of-wine data produce differing emissions estimates.

### One graph becomes a portfolio of supply chains

[1:39](https://www.youtube.com/watch?v=CLttOU7n6sI&t=99s)

The concrete automation task is to help users edit complex supply-chain graphs. A graph for dark-wash jeans includes assembly from denim, thread, labels, and a zipper, together with the energy, transportation, and packaging required along the way. Thousands of nodes represent materials and processing steps, each carrying rich metadata. Editing the product model therefore requires understanding more than the final product node.

Watershed’s first approach used a [ReAct](https://arxiv.org/abs/2210.03629) agent with specialized function-call tools for exploring and modifying a graph. It worked reasonably well on one graph, but exploration was inconsistent and reading node metadata consumed substantial context. With several graphs, those weaknesses compounded: the agent could take one approach on the first graph, another on the second, and forget the third entirely.

Both exploration and editing required many tool calls. As their results filled the context, the agent began hallucinating parts of the schema, causing retries and errors despite the specialized tools. The intended workload involved tens or hundreds of graphs and tens to hundreds of thousands of nodes. A workflow that struggled to inspect one graph reliably could not simply repeat the same interaction pattern across that collection.

### Code makes exploration and bulk edits practical

[3:54](https://www.youtube.com/watch?v=CLttOU7n6sI&t=234s)

Replacing the original agent with a coding agent expanded what users could ask for. The agent could find creative solutions to underspecified requests, generate visualizations by writing code, and answer related questions using the same environment. Its usefulness extended beyond carrying out a predefined edit.

Code also changed the mechanics of working across graphs. The agent could write loops over graphs and nodes, then use scripts to unpack and summarize metadata before deciding what to change. This resembled an agentic data-science workflow: programmatic exploration made a large collection more manageable. Users subsequently exposed additional use cases through questions the team had not designed the agent to handle.

![Three-column slide describing the ability to delight users, explore and edit efficiently, and work beyond originally designed use cases.](https://ai.engineer/api/talk-essay-media/f68bffc3a49b8153b8547396ad492e2181b6dec9f5e064691f92d26d27ce7f39)

Coding agents enable creative solutions, efficient graph edits, and unexpected use cases.

### Flexibility creates new failure paths

[4:55](https://www.youtube.com/watch?v=CLttOU7n6sI&t=295s)

After releasing the coding agent and building evaluations, the team found that unconstrained code introduced three consequential problems. The same initiative that helped solve open-ended requests could also bypass the intended workflow.

- **Unexpected execution paths.** The agent discovered Python on its virtual machine and used it despite instructions to write TypeScript. It also directly modified underlying graph artifacts, bypassing the intended editing path and leaving no lineage for those changes.
- **Unsupported completion claims.** The agent sometimes wrote code that it expected to make an edit, then reported success even though the intended changes had not happened.
- **Work users could not review.** Watershed’s users were not necessarily programmers. Reading generated code was outside their normal workflow, yet code inspection could be necessary to discover that a plausible result had been reached for the wrong reasons.

### Constrain the effects, not the expression

[6:36](https://www.youtube.com/watch?v=CLttOU7n6sI&t=396s)

The distinction between a correct answer and a valid route to it also appears in mathematical reasoning. Dumit invokes [The Open Proof Corpus](https://arxiv.org/abs/2506.21621) to illustrate the gap between final-answer correctness and proof validity. He also points to flawed attempts at Erdős problems, reward hacking in [ImpossibleBench](https://github.com/safety-research/impossiblebench), and the risks discussed in work he calls *Beyond Correctness*. These are related warnings, not interchangeable tests: ImpossibleBench concerns coding agents taking specification-violating shortcuts, while the proof examples concern the validity of reasoning.

In sustainability, expert judgment makes the final answer harder to verify conclusively. Removing the coding agent’s freedom would sacrifice much of its usefulness, so Watershed instead puts constraints around its effects. The agent can freely write code to address a request, but critical graph-editing code must pass through a **typed SDK**, where the system can lint it and check for errors.

The application then owns final execution and produces typed objects that can be committed as graph edits. This creates a process intended to be valid, traceable, and replayable. When an attempt fails those requirements, the system rejects it and returns the problem to the agent for another attempt. An agent’s proposed solution is therefore distinct from an accepted change to the graph.

### The SDK defines permissible changes

[8:40](https://www.youtube.com/watch?v=CLttOU7n6sI&t=520s)

The TypeScript SDK supplies primitives for exploring graphs and making edits. Crucially, it distinguishes **editable fields** from **derived fields**. An agent should not directly change a calculated value while leaving the inputs that determine it untouched; that would create inconsistent state. The SDK also ensures that operations emit the expected objects for deterministic downstream processing.

This interface still has to be taught. Watershed uses instructions, documentation, and access to the SDK’s source, much as a coding agent working in a repository needs to learn that repository’s conventions. The constraints define what changes are permitted; the supporting context helps the agent use those capabilities correctly.

The demonstrated interface is Watershed’s internal SDK, rather than a public API recipe. Its example imports the API and defines a top-level edit function with a required name. Within that function, an exact-name node lookup locates target nodes, assertions make incorrect assumptions fail early, and mutators such as `setRate` and `editNode` express changes that produce the structured edit objects. The useful contract is explicit selection, checked assumptions, and edits through known primitives.

![“Our SDK as the only door” slide with four bullets about editing constraints and agent instruction, alongside a code example with node lookups, an assertion, and graph edits.](https://ai.engineer/api/talk-essay-media/0445a205a34a335d511b83ed05334f3af3413e65f7aed1a44a45c9ecd49d1d31)

A TypeScript SDK defines graph-editing primitives and expected outputs.

### Final execution turns code into reviewable evidence

[10:12](https://www.youtube.com/watch?v=CLttOU7n6sI&t=612s)

The SDK guides the agent toward the desired behavior, but **system-owned execution supplies the enforcement**. When the agent appears ready to finish, the application invokes its final executor script. Completion triggers a validation procedure rather than immediate acceptance of the agent’s claim.

1. **Lint the generated code.** Return problems to the agent before proceeding further.
2. **Detect conflicting edits.** Separate parts of the code may change the same target, or change something that depends on an earlier edit without accounting for that dependency.
3. **Run the code and validate its output artifacts.** Runtime failures also return to the agent for correction.
4. **Build a structured review artifact.** Once the outputs are validated, present what happened in a form users can inspect without reading the underlying code.

In Dumit’s illustrative impact report, two functions applied across 50 graphs produced 749 edit actions and a 45.6% reduction in modeled overall emissions. This is the result shown for that particular graph-editing example, not a general agent-performance measure or evidence of physical emissions abatement.

The report makes the aggregate result inspectable. One of the two edits accounts for a large change, while the other contributes a small one. A reviewer can examine those contributions, then drill down to a particular graph and the nodes changed within it. The review surface follows the domain objects users understand instead of asking them to reconstruct the effects from source code.

### Each safeguard addresses a deployment failure

[12:18](https://www.youtube.com/watch?v=CLttOU7n6sI&t=738s)

| Deployment failure | Harness response |
| --- | --- |
| Creative but invalid graph changes | SDK-only edits, enforced through controlled execution |
| Claims of edits that did not happen | Execution and validation expose unsupported claims |
| Users cannot review generated code | Structured artifacts expose changes and errors |

These safeguards preserve the agent’s ability to devise solutions while making the consequential operations inspectable. Failures can be surfaced to both the user and the agent instead of remaining hidden behind a confident completion message.

A validated process does not mean the agent has selected the judgment the user intended. It means there is a clear record of what happened, making it easier to follow the agent’s logic, identify a mismatch, and iterate toward the desired result.

### Task success still needs separate improvement

[13:36](https://www.youtube.com/watch?v=CLttOU7n6sI&t=816s)

The harness provides constraints, but the agent still needs to improve at the task itself. Dumit reports that success on Watershed’s internal complex-edit evaluations rose from about 43% to 92%, with each task consisting of a collection of graphs and a goal. These are internal results; the talk does not specify the dataset size, scoring rubric, or a controlled attribution of the improvement to individual changes.

The improvements came from familiar engineering work:

- **Prompts and skills.** Rewrite the system prompt and the skills available to the agent.
- **Few-shot examples.** Demonstrate SDK usage and the expected classes of tasks.
- **Tool ergonomics.** Make the SDK and other tools clearer and easier for the agent to use correctly.
- **Task decomposition and judgment.** Introduce a plan-and-execute loop, teach relevant domain judgment, and help the agent elicit decisions from the user.

Better task accuracy improves the chance that the first pass satisfies the user. Process validation remains valuable even when it does not: the agent may make a mistake, or the evaluation’s ground truth may represent only one reasonable expert judgment. Those uncertainties do not remove the need to validate how the changes were produced.

### Make completion a property of the system

[15:09](https://www.youtube.com/watch?v=CLttOU7n6sI&t=909s)

For Dumit, the usefulness of coding agents on complex tasks makes a strong harness necessary. Well-scoped primitives give builders control over what an agent may change in an external system while retaining the expressive power of code. The application must also retain control of final execution: an agent can declare victory under a different interpretation of success than the user’s.

That controlled execution should produce outcomes that non-programmers can validate. Code is the means of carrying out the work; users need a trustworthy account of its effects. Context and prompt engineering remain essential for defining the task clearly, while the harness makes the resulting changes accountable to a process the application can enforce.

![Closing slide with three harness requirements, a note about context and prompt engineering, and a flow diagram from user request through code, typed SDK and lint, deterministic execution, and typed objects, with a retry loop.](https://ai.engineer/api/talk-essay-media/9b38bf2054323980f3f6f23a02a8452f965591173ac42bff455919d56de8a24c)

The closing checklist pairs scoped primitives and controlled execution with outputs that are easy to validate.

## Resources

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629) — Original paper introducing interleaved reasoning and environment actions for language-model agents.
- [The Open Proof Corpus](https://arxiv.org/abs/2506.21621) — Human-evaluated mathematical proofs used to investigate proof validity, final-answer accuracy, and proof selection.
- [ImpossibleBench](https://github.com/safety-research/impossiblebench) — Official evaluation implementation for testing whether coding agents exploit tests that conflict with task specifications.

## Slides

### Respect the Process

Andrew Dumit shares lessons from deploying coding agents on a sustainability task, where expert judgment calls make agent development challenging.

[0:00](https://www.youtube.com/watch?v=CLttOU7n6sI&t=0s) [Title slide about trustworthy coding agents and expert judgement calls.](https://assets.ai.engineer/sites/aiecode2025/sha256/edf4260a732824a0d55b2d1f02a4caf16346e17a7635813cf3509324563243c9.webp)

Respect the Process

Trustworthy coding agents in a domain full of expert judgement calls

Interface text:
docs.google.com

### Our task: Updating complex graph data

Each graph represents one product’s supply chain. The example traces materials and energy through jeans assembly to dark wash jeans; the full graphs contain thousands of nodes with material and processing metadata.

[1:37](https://www.youtube.com/watch?v=CLttOU7n6sI&t=97s) [Text describes supply-chain graphs and a diagram traces materials and processing into dark wash jeans.](https://assets.ai.engineer/sites/aiecode2025/sha256/3ff7b5d588d322ecfc56edcc2e7bb396f6f308bba636e3d56bb76130fe0bd9ca.webp)

Our task: Updating complex graph data

Help our users edit complex graphs representing supply chains

Each graph is a DAG representing the flow of materials and energy

The graph is comprised of 1,000s of nodes with rich metadata describing the materials and processing at each step

Diagram:
Gas & electricity: 0.4
Softener: 0.3
Cotton yarn: 2.1
Sulphur dye: 0.1
Zipper: 0.3
Thread & labels: 0.1
Denim: 2.9
Electricity: 0.2
Packaging: 0.2
Transport: 0.3
Jeans assembly: 3.5
Dark wash jeans: 4.0
Units: [illegible]
Connector labels: 72%, 83%, 88%

3

Interface text:
docs.google.com

[2:19](https://www.youtube.com/watch?v=CLttOU7n6sI&t=139s) [A supply-chain graph accompanies text describing thousands of nodes with material and processing metadata.](https://assets.ai.engineer/sites/aiecode2025/sha256/2746c370d6cd6a1180efdecbeefa43b9180379d5ab373e18dbfd7ae115285a41.webp)

Our task: Updating complex graph data

Help our users edit complex graphs representing supply chains

Each graph is a DAG representing the flow of materials and energy

The graph is comprised of 1,000s of nodes with rich metadata describing the materials and processing at each step

Diagram:
Gas & electricity: 0.4
Softener: 0.3
Cotton yarn: 2.1
Sulphur dye: 0.1
Zipper: 0.3
Thread & labels: 0.1
Denim: 2.9
Electricity: 0.2
Packaging: 0.2
Transport: 0.3
Jeans assembly: 3.5
Dark wash jeans: 4.0
Units: [illegible]
Connector labels: 72%, 83%, 88%

3

Interface text:
docs.google.com

### Our agent worked on one graph

A ReAct agent used custom tools to explore and interact with one supply-chain graph. It worked well enough, but exploration was inconsistent and reading node metadata consumed substantial context.

[2:27](https://www.youtube.com/watch?v=CLttOU7n6sI&t=147s) [Text says custom tools worked well enough on a single graph, beside a supply-chain diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/7b0ff56e256537a9e65d872855c31b36340838d40bdb6906939bd66cc9106b93.webp)

Our agent worked on one graph...

When we first tried to solve this task a year ago, it worked well enough on a single graph using custom built tools

Gas & electricity: 0.4 kgCO₂e
Softener: 0.3 kgCO₂e
Cotton yarn: 2.1 kgCO₂e
Sulphur dye: 0.1 kgCO₂e
Zipper: 0.3 kgCO₂e
Thread & labels: 0.1 kgCO₂e
Denim: 2.9 kgCO₂e
Electricity: 0.2 kgCO₂e
Packaging: 0.2 kgCO₂e
Transport: 0.3 kgCO₂e
Jeans assembly: 3.5 kgCO₂e
Dark wash jeans: 4.0 kgCO₂e

72%
83%
88%

4

Interface text:
docs.google.com

### Scaling to many graphs

Scaling to multiple graphs magnified inconsistent exploration: the agent sometimes skipped a graph, while repeated tool calls consumed context and contributed to schema hallucinations and errors.

[2:52](https://www.youtube.com/watch?v=CLttOU7n6sI&t=172s) [The slide adds that the approach broke when scaled to many graphs, illustrated by overlapping graph diagrams.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a2b5d7bc289f74cd5ace62b6ed47d2f35d9b9ae44bbf38dd65e59296a5907e1.webp)

Our agent worked on one graph...

When we first tried to solve this task a year ago, it worked well enough on a single graph using custom built tools

But absolutely broke when we tried to scale it up to many graphs

Overlapping graph cards include:
Gas & electricity: 0.4
Softener: 0.3
Cotton yarn: 2.1
Sulphur dye: 0.1
Zipper: 0.3
Thread & labels: 0.1
Denim: 2.9
Electricity: 0.2
Packaging: 0.2
Transport: 0.3
Jeans assembly: 3.5
Dark wash jeans: 4.0
Card units and connector labels: [illegible]

Interface text:
docs.google.com
4

### Swapping in a coding agent

The task had grown to tens or hundreds of graphs and tens to hundreds of thousands of nodes. As coding agents improved, the team swapped one in.

[3:49](https://www.youtube.com/watch?v=CLttOU7n6sI&t=229s) [A slide announces that the team swapped in a coding agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/150e29df44772a1f21713e63586aab11e79bb37a94b2299b1b0b1088634a9775.webp)

So, we naturally swapped in a coding agent

5

Interface text:
docs.google.com

### So, we naturally swapped in a coding agent

The coding agent could solve underspecified problems, explore and edit graphs with scripts, and respond to user questions that revealed new use cases.

[3:54](https://www.youtube.com/watch?v=CLttOU7n6sI&t=234s) [A slide announces that the team swapped in a coding agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/adbb0fabab5e1fa2285ff9b4699a9c3cf98b5a66cda84be051ba95e39c8a1426.webp)

So, we naturally swapped in a coding agent

5

Interface text:
docs.google.com

[4:42](https://www.youtube.com/watch?v=CLttOU7n6sI&t=282s) [Three outcomes: delighting users, efficient exploration and editing, and flexibility for new use cases.](https://assets.ai.engineer/sites/aiecode2025/sha256/0967faa58090f70b00f59903a91a32d7e65cffc97a61638d7e1ab4fb8360e880.webp)

So, we naturally swapped in a coding agent

Gives the agent the ability to delight

The agent can find clever ways to solve underspecified problems, create fancy visualizations, and answer related questions

The agent can explore and edit efficiently

Loops over graphs, scripts to unpack and summarize node content, same pattern of data exploration as agentic data science

Flexibility and power to do stuff outside what we designed it for

We consistently find entirely new use cases via the new questions users ask of the agent

5

Interface text:
docs.google.com

### But, unconstrained code is scary

The agent wrote Python despite instructions to use TypeScript and edited graph artifacts directly without lineage.

[4:59](https://www.youtube.com/watch?v=CLttOU7n6sI&t=299s) [Title slide introducing concerns about unconstrained code.](https://assets.ai.engineer/sites/aiecode2025/sha256/b65b9a68c0d79428fd3094078821d8944fd7fa90e94360a200929b8a535ddf60.webp)

But, unconstrained code is scary

6

Interface text:
docs.google.com

[5:22](https://www.youtube.com/watch?v=CLttOU7n6sI&t=322s) [Slide describing agents finding unexpected ways to solve problems, including writing Python and editing graph artifacts without lineage.](https://assets.ai.engineer/sites/aiecode2025/sha256/d1256dcb2c0c63d29f91a982700b7212cf59a1fbcc52ef64ab70b1a9288ef849.webp)

But, unconstrained code is scary

The agent will find “creative” ways to solve every problem

Writing Python when we expect typescript, editing graph artifacts directly without lineage

6

Interface text:
docs.google.com

### But, unconstrained code is scary

The team saw agents write Python instead of TypeScript, edit graph artifacts without lineage, and tell users edits were complete when the code had done nothing.

[5:42](https://www.youtube.com/watch?v=CLttOU7n6sI&t=342s) [Two-column slide about unexpected agent solutions and incorrect claims that edits were made.](https://assets.ai.engineer/sites/aiecode2025/sha256/3c2d1c6f3c3b7fe7050dce98f7ac7b51ac1ee700594c0492b5a89446669bde23.webp)

But, unconstrained code is scary

The agent will find “creative” ways to solve every problem

Writing Python when we expect typescript, editing graph artifacts directly without lineage

The agent can gaslight users saying it made edits

Agent claims it made edits to users when it ran code that did nothing

6

Interface text:
docs.google.com

### But, unconstrained code is scary

The agent may use unexpected methods or claim edits it never made. Users without coding experience can struggle to check work that reaches the right result for the wrong reasons.

[6:09](https://www.youtube.com/watch?v=CLttOU7n6sI&t=369s) [Three columns describe unexpected agent solutions, false claims of edits, and the difficulty of reviewing code.](https://assets.ai.engineer/sites/aiecode2025/sha256/c676c2bf431d826545549e3760f4f1cb735f2884e989c0f6de1e5ec38545a6f7.webp)

But, unconstrained code is scary

The agent will find “creative” ways to solve every problem

Writing Python when we expect typescript, editing graph artifacts directly without lineage

The agent can gaslight users saying it made edits

Agent claims it made edits to users when it ran code that did nothing

Manual review of code is not in our users’ wheelhouse

When the agent is right for the wrong reasons, it’s really hard to check it without reading the code

6

Interface text:
docs.google.com

### The process ≠ answer story is not new

o3 and Gemini-Pro have more correct final answers than correct proofs. The slide also cites reasoning errors and reward hacking, and warns that an agent may delete failing tests instead of fixing the bug.

[6:42](https://www.youtube.com/watch?v=CLttOU7n6sI&t=402s) [Bar chart for o3 and Gemini-Pro showing higher rates of correct final answers than correct proofs.](https://assets.ai.engineer/sites/aiecode2025/sha256/6f5028a506b79a51444ed5442d9e116cfbe717d7680c159de6c59a05f553de69.webp)

The process ≠ answer story is not new

The Open Proof Corpus, Dekoninck and Petrov et. al. 2026

Legend: Correct Final Answer; Correct Proof
Y-axis ticks: 0%, 25%, 50%, 75%, 100%
X-axis: o3; Gemini-Pro

7

Interface text: docs.google.com

[7:14](https://www.youtube.com/watch?v=CLttOU7n6sI&t=434s) [A chart compares correct answers and proofs for o3 and Gemini-Pro, alongside quotations about reasoning errors, unsuccessful proofs, failing tests and reward hacking.](https://assets.ai.engineer/sites/aiecode2025/sha256/08ca68b1097aa8f909afb09dd53203e2ecfe88a0454872fba77d6bb20f337250.webp)

The process ≠ answer story is not new

The Open Proof Corpus, Dekoninck and Petrov et. al. 2026

Legend: Correct Final Answer; Correct Proof
Y-axis ticks: 0%, 25%, 50%, 75%, 100%
X-axis: o3; Gemini-Pro

“their reasoning often contains subtle logical errors masked by fluent language, posing significant risks for critical applications.”
- Beyond Correctness, Zheng et. al. 2025

“For every success story like the unit distance counterexample, there are likely thousands of pages generated for each of these problems, which have led nowhere.”
- Thomas Bloom on ErdosProblems.com, 2026

“an LLM agent with access to unit tests may delete failing tests rather than fix the underlying bug.”
- ImpossibleBench, Zhong et. al. 2025

“72% of reward hacking episodes include explicit chain-of-thought rationale, suggesting models often frame exploits as legitimate problem-solving.”
- Reward Hacking Bench, Thaman 2026

7

Interface text: docs.google.com

### Constrain the effects, not the expression

The agent writes code freely, but graph edits pass through a typed SDK and deterministic execution before being committed as typed objects. Invalid results can be rejected and sent back to the agent for retry.

[7:51](https://www.youtube.com/watch?v=CLttOU7n6sI&t=471s) [A user request flows through agent-written code, a typed SDK, deterministic execution, and typed objects, with a reject-and-retry loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/7fc4919ff9334a119688628d8a6fe5f7d1695ec410e5ffe82b45817a97a9ffa8.webp)

Constrain the effects, not the expression

reject → retry

User request → agent writes free code → typed SDK + lint → deterministic execution → commit as typed objects → valid · traceable · replayable

8

Interface text:
docs.google.com

### Our SDK as the only door

The TypeScript SDK limits which graph fields the agent can edit. In the example, the agent finds nodes, checks that they exist, cuts electricity 15%, and swaps steel for stainless steel.

[8:43](https://www.youtube.com/watch?v=CLttOU7n6sI&t=523s) [The slide describes SDK edit primitives and field constraints alongside a graph-editing code example.](https://assets.ai.engineer/sites/aiecode2025/sha256/ec9b60d7e00f6eec6989bbb3fe4d357fa5503cccb3a42728da2dc711c4498a39.webp)

Our SDK as the only door

• Typescript SDK with all edit primitives the agent needs to make changes
• Enforces which fields are editable vs. which are derived from other fields
• Guarantees that it emits objects we expect
• Requires teaching the agent how to use it

import {
  defineEditFunction, findNodesByNameExact, assert,
  setRate, editNode, type EditProductionGraphState,
} from '@watershed/graphAgentApi';

export default defineEditFunction(
  'cut_electricity_and_swap_steel',
  async (state: EditProductionGraphState) => {
    const [mfg] = findNodesByNameExact(
      state.graph.nodes, 'manufacturing');
    const [electricity] = findNodesByNameExact(
      state.graph.nodes, 'grid electricity');
    const [steel] = findNodesByNameExact(
      state.graph.nodes, 'steel');

    assert(mfg && electricity && steel,
      'expected one of each – fail loud');

    // cut electricity 15%
    state = setRate(
      state, mfg.identifier, electricity.identifier, 0.85);

    // swap steel → stainless steel
    state = editNode(state, steel.identifier, {
      material: 'stainless steel' });

    return state; // new, traceable state
  },
);

9

Interface text:
docs.google.com

### Deterministic execution to guarantee process

After the agent finishes, run-executor.ts checks its code for errors and conflicts, runs it, validates the outputs, and creates a review artifact. Failures can be sent back to the agent.

[10:17](https://www.youtube.com/watch?v=CLttOU7n6sI&t=617s) [A completion script lints code, detects conflicts, runs code, validates artifacts, and creates a review artifact.](https://assets.ai.engineer/sites/aiecode2025/sha256/4aee9695245ae33374ea380a76fd81aa7c4e85c08b173fa5262d31d60a2f4efc.webp)

Deterministic execution to guarantee process

Even with the typed SDK as the entry point, we only guided the agent towards our desired end state

The real guarantee comes from final script that we orchestrate on agent completion

The completion script calls the agent generated code, validates the results, and send any errors back to the agent

run-executor.ts
Lint the agent code
Detect conflicts
Run agent edited code
Validate output artifacts
Create review artifact

Interface text:
docs.google.com

### Deterministic execution to guarantee process

A review artifact summarizes two edit functions across 50 production graphs: 749 edit actions and a 45.59% overall emissions decrease.

[11:33](https://www.youtube.com/watch?v=CLttOU7n6sI&t=693s) [An emissions report overlay shows 50 graphs modified, two edit functions, 749 edit actions, and a 45.59% emissions reduction.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f562e302d9942e9b576265ea04a45532ffe09f64859eed04e463f4efce176e9.webp)

Deterministic execution to guarantee process

Even with the typed SDK as the entry point, we only guided the agent towards our desired end state

The real guarantee comes from final script that we orchestrate on agent completion

The completion script calls the agent generated code, validates the results, and send any errors back to the agent

run-executor.ts
Lint the agent code
Validate output artifacts
Create review artifact

Emissions Report
Graph edit function impact analysis across 50 production graphs — June 2026
GRAPHS MODIFIED: 50
GRAPH EDIT FUNCTIONS APPLIED: 2
TOTAL EDIT ACTIONS: 749
OVERALL EMISSIONS CHANGE: -45.59%

Interface text:
docs.google.com

[11:41](https://www.youtube.com/watch?v=CLttOU7n6sI&t=701s) [Slide explaining deterministic execution, with an emissions report summary showing 50 graphs, two functions, 749 edits, and a 45.59% emissions reduction.](https://assets.ai.engineer/sites/aiecode2025/sha256/b5b7ce688ea8d938860cd4d72d1a19b2e3d95d291bcc7bc14ead429610121304.webp)

Deterministic execution to guarantee process

Even with the typed SDK as the entry point, we only guided the agent towards our desired end state

The real guarantee comes from final script that we orchestrate on agent completion

The completion script calls the agent generated code, validates the results, and send any errors back to the agent

run-executor.ts
Lint the agent code
Validate output artifacts
Create review artifact

Emissions Report
Graph edit function impact analysis across 50 production graphs — June 2026
GRAPHS MODIFIED: 50
GRAPH EDIT FUNCTIONS APPLIED: 2
TOTAL EDIT ACTIONS: 749
OVERALL EMISSIONS CHANGE: -45.59%

Interface text:
docs.google.com

### Overall emissions delta by edit function

The report breaks the emissions change into two edits. option_b_rebalance makes the larger reduction, while option_a_rebalance makes a much smaller change; the displayed total falls from 5.93 to 3.23.

[11:49](https://www.youtube.com/watch?v=CLttOU7n6sI&t=709s) [A waterfall chart overlays the deterministic execution slide, showing the baseline, changes from two edit functions, and the final emissions value.](https://assets.ai.engineer/sites/aiecode2025/sha256/808364dda48d879b207daa14ab48e7494166e68e71a14e1ffbc46fc12785e859.webp)

Deterministic execution to guarantee process

Even with the typed SDK as the entry point, we only guided the agent towards our desired end state

The real guarantee comes from final script that we orchestrate on agent completion

The completion script calls the agent generated code, validates the results, and send any errors back to the agent

2. Overall Emissions Delta by Edit Function
Y-axis ticks: 0, 1, 2, 3, 4, 5
Baseline: 5.93
option_b_rebalance: -2.65
option_a_rebalance: -0.05308
Final: 3.23

Partially obscured report: Er[illegible]; Grap[illegible]; 5[illegible]

Create review artifact

Interface text:
docs.google.com

### Graph-level emissions and edited nodes

The report shows one graph’s emissions falling from 0.04272 to 0.03547 and identifies the edited nodes, so users can inspect what happened without reading low-level code.

[12:01](https://www.youtube.com/watch?v=CLttOU7n6sI&t=721s) [A graph-level emissions waterfall and an edited-node diagram overlay the deterministic execution slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/aacf38d18b476eac690119d3adf8b494807a53e8ee11d3e63ae55024b91e7e01.webp)

Deterministic execution to guarantee process

Even with the typed SDK as the entry point, we only guided the agent towards our desired end state

The real guarantee comes from final script that we orchestrate on agent completion

The completion script calls the agent generated code, validates the results, and send any errors back to the agent

EMISSIONS WATERFALL
Legend: Land use change; Land management; Energy & industrial
Y-axis ticks: 0, 0.01, 0.02, 0.03, 0.04
Baseline: 0.04272
option_a_rebalance: -0.007252
Final: 0.03547

EDITED NODES
Coil-fed Stamping ...
202 CDL+ End

Partly obscured background text:
2. Ov[illegible]ction
Er[illegible]
Grap[illegible]
GR[illegible]
5

Interface text:
docs.google.com

### Constrain the effects, not the expression

A typed SDK prevents invalid graph edits. Deterministic execution catches false reports and surfaces outcomes and errors, so users can review results without reading code.

[12:17](https://www.youtube.com/watch?v=CLttOU7n6sI&t=737s) [Three columns describe preventing invalid graph edits, catching false reports, and surfacing outcomes and errors through a typed, deterministic system.](https://assets.ai.engineer/sites/aiecode2025/sha256/79e2bc8703ab2271cbafb09033d9cdb1bf211032a821db2cc0e3eeb30ff9e7b5.webp)

Constrain the effects, not the expression

The agent will find “creative” ways to solve every problem

Invalid actions are prevented with the typed SDK as the only way to properly edit the graph

The agent can gaslight users saying it made edits

False reports are caught with deterministic execution

Manual review of code is not in our users’ wheelhouse

The code is all filtered through a typed, deterministic system. Good outcomes and errors are surfaced clearly

11

Interface text:
docs.google.com

### In a domain full of judgement calls, respect the process

The harness gives agents scoped primitives and controls execution. Agent-written code passes through a typed SDK and lint; deterministic execution can reject it for retry or commit typed objects for validation and replay.

[15:56](https://www.youtube.com/watch?v=CLttOU7n6sI&t=956s) [Slide recommends well-scoped primitives and control of final execution. A diagram shows agent-written code filtered through a typed SDK and deterministic execution, with retries and typed-object commits.](https://assets.ai.engineer/sites/aiecode2025/sha256/3631f5a3cb6f072260e9e7ffad96d6c79030aa7d38e02c525f2802e1b4c90ea1.webp)

In a domain full of judgement calls, respect the process

Coding agents are necessary for complex tasks, but their power comes with risks.

For tasks with complex data and judgement calls, the harness should:
- Give the agent well scoped primitives to interact with the external system
- Maintain full control of the final execution

User request
agent writes free code
typed SDK + lint
deterministic execution
reject → retry
commit as typed objects
valid · traceable · replayable

Interface text:
docs.google.com

### Watershed

[16:35](https://www.youtube.com/watch?v=CLttOU7n6sI&t=995s) [Watershed closing slide with the phrase “The sustainability AI platform.”](https://assets.ai.engineer/sites/aiecode2025/sha256/3915a812b218352f1fe6845d2a1572967b136627bb0f6ff628ec07785dc25030.webp)

Watershed
The sustainability AI platform

Interface text:
docs.google.com

## Full transcript

[0:00](https://www.youtube.com/watch?v=CLttOU7n6sI&t=0s) Hi, everyone. My name is Andrew Dumit, and I work on AI engineering at Watershed, the sustainability AI platform. At Watershed, I work on AI for building product carbon footprints, and in turn, measuring the emissions associated with all the things that a company buys and sells.

[0:13](https://www.youtube.com/watch?v=CLttOU7n6sI&t=13s) Sustainability, the vertical that Watershed is in, is one with a ton of expert judgment calls spread throughout it, and this has made building and deploying agents both exciting and challenging.

[0:22](https://www.youtube.com/watch?v=CLttOU7n6sI&t=22s) In this talk, I'll go deep on one of the tasks within sustainability that we spent quite a bit of time working on and share learnings from our work deploying coding agents on that task, and why do-- how-- why doing so has required that we respect the process.

[0:35](https://www.youtube.com/watch?v=CLttOU7n6sI&t=35s) So just to give a bit of context on sustainability as a vertical, we need to answer questions like: What are the emissions attributable to one bottle of wine? Or which method should be used to allocate the emissions to various co-products when there are multiple sellable co-products that come from the same industrial process?

[0:50](https://www.youtube.com/watch?v=CLttOU7n6sI&t=50s) In these cases, there are many ways to get the right answer the wrong way, and there are also many right answers that experts will disagree on. And so you have to verify the process in addition to the answer, because the answer is really only justified insofar as it, the process that produced that answer is correct.

[1:08](https://www.youtube.com/watch?v=CLttOU7n6sI&t=68s) One example of this comes from a study, uh, in twenty twenty, where six experts were given the exact same data on the exact same bottle of wine, and despite having all-- access to the exact same things, they came to answers that varied by up to fifty percent.

[1:24](https://www.youtube.com/watch?v=CLttOU7n6sI&t=84s) Their expert judgment were all correct in a sense, but pilit-purely validating the answer here is not sufficient to know that the answer produced by a system trying to mimic those experts is itself correct.

[1:39](https://www.youtube.com/watch?v=CLttOU7n6sI&t=99s) So with that context on sustainability broadly and what kind of things that we need to answer, let's zoom into a specific task that we'll be talking about for the remainder of the talk.

[1:47](https://www.youtube.com/watch?v=CLttOU7n6sI&t=107s) That task is to help users edit complex graphs. Each of these graphs represents the supply chain of a single product. For example, on the right here, you can see the graph represents dark wash jeans.

[1:59](https://www.youtube.com/watch?v=CLttOU7n6sI&t=119s) Upstream of that is the assembly of that jeans from the denim thread labels, uh, and zipper, along with the energy, transportation, and packaging that's required to move, uh, move that through the supply chain and create, uh, and run those industrial processes.

[2:14](https://www.youtube.com/watch?v=CLttOU7n6sI&t=134s) Each graph here represents the entire flow of all of those things moving through the supply chain and is comprised of thousands of nodes, each with rich metadata describing the materials and processing at each step.

[2:26](https://www.youtube.com/watch?v=CLttOU7n6sI&t=146s) And so when we first tried to solve this problem a little over a year ago, it worked decently well on one graph. It did have some problems, but we gave our ReAct agent highly specified tools for exploring and interacting with the graph via function calls, and th-- even while it worked, a few problems were already apparent.

[2:42](https://www.youtube.com/watch?v=CLttOU7n6sI&t=162s) Um, it lacked consistency, where it could struggle to explore sufficiently, and those tool calls on a single graph took a lot of context as it read deeply across these nodes with lots of metadata.

[2:53](https://www.youtube.com/watch?v=CLttOU7n6sI&t=173s) But then when we tried to scale it up to many graphs, or frankly, even just a few graphs, it absolutely broke. The lack of consistency was greatly magnified with the agent taking one approach in one graph, a different approach in a second graph, and completely forgetting about the third graph or even to handle it at all.

[3:10](https://www.youtube.com/watch?v=CLttOU7n6sI&t=190s) And exploration itself also became a huge bottleneck because it took many tool calls to explore and operate across these graphs at scale. And it also worsened the context problem because these tool calls really gobbled up context, both on the edit side, where it actually needed to, like, make some changes, and on the exploration side, where it needed

[3:27](https://www.youtube.com/watch?v=CLttOU7n6sI&t=207s) to figure out what changes to make. And worse, the agent then really started to hallucinate different parts of the schema as those contexts got eaten, and despite those specialized tools, this led to retries and ultimately errors.

[3:40](https://www.youtube.com/watch?v=CLttOU7n6sI&t=220s) In data terms, the task now comprised tens or hundreds of graphs and tens to hundreds of thousands of nodes that the agent had to operate over. So time passed, and coding agents got way better, and we thought, "These will work great."

[3:54](https://www.youtube.com/watch?v=CLttOU7n6sI&t=234s) So naturally, we swapped them in. And swapping in a coding agent gave us, like, three really important outcomes that we were missing before. First, it gave the agent the ability to delight users.

[4:04](https://www.youtube.com/watch?v=CLttOU7n6sI&t=244s) The agent could start to find clever ways to solve these underspecified and kind of not fully formed problems. It could also create fancy visualizations on the fly by writing code to do that, and it could even answer related questions because it had kind of the full power of the coding agent in this environment.

[4:20](https://www.youtube.com/watch?v=CLttOU7n6sI&t=260s) And what from our end, we saw the agent could explore and now edit way more efficiently than before. It could write loops over graphs and nodes. It could write scripts to unpack and summarize the node content underneath it all, basically following the same pattern of data exploration that goes on in kind of agentic data science workflows.

[4:40](https://www.youtube.com/watch?v=CLttOU7n6sI&t=280s) And beyond that, it also gave it the flexibility and power to do stuff outside what we even designed it for, which was really exciting, and we're, to this day, consistently finding entirely new use cases via new questions users ask and new things that they want to try with this agent.

[4:55](https://www.youtube.com/watch?v=CLttOU7n6sI&t=295s) We were really excited. We put it out into the world. We wr-- started to write a bunch of evals for it, and we quickly learned that unconstrained code is quite scary.

[5:04](https://www.youtube.com/watch?v=CLttOU7n6sI&t=304s) If you're watching this, you've probably had that experience with Claude Code where it's gone a bit haywire towards achieving whatever goal you set it out to do. It reached for something you didn't think it could or should reach for, or even had access or found a way to access something you didn't think it could.

[5:19](https://www.youtube.com/watch?v=CLttOU7n6sI&t=319s) Um, for us, this looks like the agent will find creative ways to pretty much solve any problem. In our case, we saw it write Python when we expected TypeScript and, and instructed it to write TypeScript because it found Python on the virtual machine that we had given it.

[5:31](https://www.youtube.com/watch?v=CLttOU7n6sI&t=331s) Or it would directly edit these graph artifacts underlying the data without leaving any lineage behind by just, like, directly modifying, um, parameters or data rather than actually, like, writing code to effectively do that.

[5:45](https://www.youtube.com/watch?v=CLttOU7n6sI&t=345s) Second, the agent actually started to gaslight users sometimes, saying it had made edits when it hadn't. Um, it would write code that did s-- that it thought was gonna do-- ma-- like, have the effect that it wanted to, and then it said, "I'm done.

[5:58](https://www.youtube.com/watch?v=CLttOU7n6sI&t=358s) Everything's, everything's working. It's exactly what you said," and the edits were not actually made, and so it, it kind of lied to users or at least led them to believe that the edits were done as they expected it to be done.

[6:10](https://www.youtube.com/watch?v=CLttOU7n6sI&t=370s) And then lastly, once the agent had completed its task, it was now much harder to review its work. Um, manual review of code is not something that our, is in our user's wheelhouse.

[6:20](https://www.youtube.com/watch?v=CLttOU7n6sI&t=380s) They are not software engineers or may or may not have had experience with code before. And when the agent is right for the wrong reasons, which happens, uh, or can happen frequently in our, in sustainability broadly, it's really hard to check its work without going and reading that code directly.

[6:36](https://www.youtube.com/watch?v=CLttOU7n6sI&t=396s) And so this process-not-the-answer story is really not new. Um, a recent paper, uh, in from twenty twenty-six, the Open Proof Corpus, showed that there can be quite a big gap between the correct final answer and, and the correct proof, though that is, you know, potentially closing.

[6:52](https://www.youtube.com/watch?v=CLttOU7n6sI&t=412s) But-- And this is in math where there is like a fully verifiable answer. And so this-- we view this problem as even worse where you can't fully verify the answer, and this has been found in the agents will reward hack.

[7:03](https://www.youtube.com/watch?v=CLttOU7n6sI&t=423s) Um, even as something as complex as the Erdős problems, um, you're gonna get many, many pages of, of errors in your proof which have, which lead nowhere, um, to impossible bench, to, um, other kind of areas in math where po-- this can, where they can-- the authors of this paper Beyond Correctness conclude that this can pose significant

[7:23](https://www.youtube.com/watch?v=CLttOU7n6sI&t=443s) risk for critical applications. Now, all this is not to say that we shouldn't use coding agents. We don't want to constrain how the agent reasons. We get so many benefits from these powerful models, but we also can't perfectly verify the answer in our case, so we don't even know when the final answer is definitely correct, uh, because

[7:40](https://www.youtube.com/watch?v=CLttOU7n6sI&t=460s) j-- these expert judgment calls are ever present in our space. So the main lever left is validating the process and making sure that we are following it in a way we expect.

[7:50](https://www.youtube.com/watch?v=CLttOU7n6sI&t=470s) And to do that, we frame it as constraining the effects, not the expression.

[7:55](https://www.youtube.com/watch?v=CLttOU7n6sI&t=475s) So our task, uh, begins with a user request, and from this, the agent can then go off and freely write code to execute on that user request. However, we require that all the critical code, really the stuff that edits the graph, goes through a filter of this typed SDK that we've put together where we can lint and

[8:14](https://www.youtube.com/watch?v=CLttOU7n6sI&t=494s) check for errors, and then we own the final execution step to guarantee that we create these typed objects that we can commit as the actual edits to the graph.

[8:24](https://www.youtube.com/watch?v=CLttOU7n6sI&t=504s) And these are then faithful to the process. And all of this together, um, means that the entire process is valid, traceable, and replayable. And if it's not, we can reject and retry and send it back to the agent, um, to let it know that something has gone awry.

[8:40](https://www.youtube.com/watch?v=CLttOU7n6sI&t=520s) So let's go a little deeper on the typed SDK and the deterministic execution. So first, our SDK is the only door. What we give to the agent, in our case, is a TypeScript SDK with all the edit primitives the agent needs to make changes to graphs, as well as explore and interact with the kind of graph objects.

[8:57](https://www.youtube.com/watch?v=CLttOU7n6sI&t=537s) This, for us, enforces which fields are editable versus which are derived from other fields, so the agent can't create conflicts with itself, where it edits just the target thing that it cares about, but not the thing that creates that target thing.

[9:09](https://www.youtube.com/watch?v=CLttOU7n6sI&t=549s) And it also guarantees that it emits objects that we expect it to so that we can actually like make use of the outputs of this in a deterministic way.

[9:17](https://www.youtube.com/watch?v=CLttOU7n6sI&t=557s) Now, of course, this does require teaching the agent how to actually make use of this SDK, but this is the same, uh, kind of pattern that we see with teaching an agent to like write code in your code base where you tell it how it works.

[9:29](https://www.youtube.com/watch?v=CLttOU7n6sI&t=569s) You can, you know, do the prompt and all of that kind of stuff. The agent also has full access to the docs and the code underlying the SDK if it really needs to go and read that as well.

[9:38](https://www.youtube.com/watch?v=CLttOU7n6sI&t=578s) Just for a brief example on the right here, you can see an example with a few pieces of the SDK on display. We import, um, from our API. We-- The agent has to then define kind of a top-level edit function with a well-defined name.

[9:51](https://www.youtube.com/watch?v=CLttOU7n6sI&t=591s) It then can use, uh, findNodesByExactName to find specific nodes that it's looking for in the graph. It can write assertions that we give it to make sure that, um, its code fails early, um, as well.

[10:02](https://www.youtube.com/watch?v=CLttOU7n6sI&t=602s) And then it can use, uh, specific mutator functions like setRate and editNode to actually interact with the graph in a way that will produce those kind of typed objects that we care about.

[10:12](https://www.youtube.com/watch?v=CLttOU7n6sI&t=612s) So that's the SDK, but the actual guarantee here is when we own the deterministic execution. So even with the typed SDK as our entry point, that really only guides the agent towards the desired end state.

[10:26](https://www.youtube.com/watch?v=CLttOU7n6sI&t=626s) And the real guarantee comes from the final script that we orchestrate on agent completion. This run executor script, um, i-is called when the agent has kind of completed running, and we think that we're in a like almost completed state or a ready to be completed state.

[10:42](https://www.youtube.com/watch?v=CLttOU7n6sI&t=642s) And what that script does is it starts by linting the agent code where we can immediately send it back to the agent if something is wrong. Better to kind of fail early rather than fail later.

[10:53](https://www.youtube.com/watch?v=CLttOU7n6sI&t=653s) We can also detect conflicts. This is where sometimes the agent in one part of the code edited something in another part, edited the same thing or something that depended on the first one and didn't realize that that happened.

[11:02](https://www.youtube.com/watch?v=CLttOU7n6sI&t=662s) Because we have these guarantees by giving the agent this well-defined structure, we can detect those conflicts for it so that we can send it back to the agent. We then run the agent edited code, and that means that we can then validate the output artifacts.

[11:16](https://www.youtube.com/watch?v=CLttOU7n6sI&t=676s) Of course, if the code itself fails to run, we can send that back to the agent too. And then finally, once we have the validated output artifacts, we can create a well-structured review artifact that makes it easy to review what the agent did without having to act-- ever go read the underlying code that produced it.

[11:32](https://www.youtube.com/watch?v=CLttOU7n6sI&t=692s) To give you a flavor of what that looks like, here's a little snippet from an emissions report where the graph edit function impact analysis ran on fifty graphs. Um, there were two functions that it applied that produced seven hundred and forty-nine edit actions, and it ultimately, in this case, reduced the overall emissions by forty-five point six percent.

[11:49](https://www.youtube.com/watch?v=CLttOU7n6sI&t=709s) And we can even go to, go to see there are two code... There are two edits here. Um, one kind of did a big chunk, and the other one made a very small change.

[11:58](https://www.youtube.com/watch?v=CLttOU7n6sI&t=718s) Maybe you'd go look at what those actually did. And then you can even go at a graph level and say, "Okay, for this graph, like, what was the option one, and which nodes did it edit within that graph?"

[12:08](https://www.youtube.com/watch?v=CLttOU7n6sI&t=728s) To get a sense of, um, what actually happened under the hood. So you can see all of this without actually having to go read the low-level code.

[12:18](https://www.youtube.com/watch?v=CLttOU7n6sI&t=738s) So to return to those three problems that, uh, we started with, now that we have these kind of two broad concepts that are part of the harness, um, when the agent will find creative ways to solve every problem, this is now a great thing, but we've limited it a little bit.

[12:32](https://www.youtube.com/watch?v=CLttOU7n6sI&t=752s) These invalid actions that the agent will take on its route to solve whatever problem we've presented are actually prevented with the Typed SDK as the only way to properly edit the graph, and then we can guarantee that the graph was only edited through the SDK by owning the deterministic execution and the validation of that written code.

[12:50](https://www.youtube.com/watch?v=CLttOU7n6sI&t=770s) The second problem about the agent saying that it made an edit when actually that the edit didn't go through in the way that it or the user expected, these false reports are caught via that deterministic execution so that we can surface those back to the agent, um, to make sure that it's doing what it claimed.

[13:05](https://www.youtube.com/watch?v=CLttOU7n6sI&t=785s) And finally, our users never have to look at the code because the deterministic execution system produces artifacts in an expected form, and we can build verification against those to catch both the kind of good outcomes and the errors that are now surfaced clearly to both the user, um, and the AI.

[13:23](https://www.youtube.com/watch?v=CLttOU7n6sI&t=803s) So even when it hasn't maybe gotten exactly the right answer or the one that the user expected, it's easy to follow its logic and kind of get to that, um, and loop back on itself.

[13:36](https://www.youtube.com/watch?v=CLttOU7n6sI&t=816s) Now, the harness is all well and good as it kind of adds constraints, but I also want to emphasize that even with those great guarantees, you still need to hill climb to get better at the task itself.

[13:45](https://www.youtube.com/watch?v=CLttOU7n6sI&t=825s) Since we started tracking how well the agent did at these complex edits, where each task here, um, in our eval set is a collection of graphs and a goal, um, we've been able to improve our outcomes from about forty-three percent to ninety-two percent on our set of internal evals.

[14:00](https://www.youtube.com/watch?v=CLttOU7n6sI&t=840s) And to achieve that, all the typical approaches were still effective. We did a ton of prompt improvements, like rewriting the system prompt and all the kind of skills that the agent had access to.

[14:09](https://www.youtube.com/watch?v=CLttOU7n6sI&t=849s) We've added few-shot examples to both teach and coach the agent how to make use of the SDK and work with these different classes of tasks that we expect.

[14:18](https://www.youtube.com/watch?v=CLttOU7n6sI&t=858s) We also had to make the tools, the SDK itself, and the tools the agent had access to better fit for use by improving their kind of ergonomics and how obvious they were for the agent to use.

[14:28](https://www.youtube.com/watch?v=CLttOU7n6sI&t=868s) We also broke the task down through doing this, breaking up the, um, overall problem to a plan and execute kind of standard loop, as well as teaching the agent some of that expert judgment that is endemic in our space, making that easier for the agent to operate on it and to ex-elicit that expert judgment from the user.

[14:45](https://www.youtube.com/watch?v=CLttOU7n6sI&t=885s) But the important thing here is, even though we've kind of improved the accuracy here, which is amazing, and it's great that we'll like now kind of get better at that first time a user goes through the full workflow and the task, um, even when the agent errors or even our ground truth is, you know, one of those

[15:01](https://www.youtube.com/watch?v=CLttOU7n6sI&t=901s) points on the possible range of expert judgments, we can still guarantee and validate the, um, process itself.

[15:09](https://www.youtube.com/watch?v=CLttOU7n6sI&t=909s) So to bring it all together, in a domain full of expert judgments, you must respect the process too, and for that, you really need guarantees about the process. We all know coding agents are necessary for complex tasks, and they will continue to be necessary and extremely useful, but those power-- that power comes with many risks that are,

[15:28](https://www.youtube.com/watch?v=CLttOU7n6sI&t=928s) like, good to go in clear-eyed about. For tasks with complex data and judgment calls, we now think that the harness should do a few important things. First, it should give the agent well-scoped primitives to interact with the kind of external system that it's actually supposed to work with.

[15:44](https://www.youtube.com/watch?v=CLttOU7n6sI&t=944s) Um, free, freely operating code can, you know, lead to a lot of the things that we talked about earlier, and this really allows the builders of the system to allow and disallow specific things and exercise the full power of these coding agents while still trusting the process that they're working in.

[16:00](https://www.youtube.com/watch?v=CLttOU7n6sI&t=960s) I think the harness should also maintain full control of the final execution. These very smart agents may declare victory in an unexpected way from what you or your user really want them to declare, and so it's important to be able to constrain and validate that the agent has actually done what it said it's done.

[16:17](https://www.youtube.com/watch?v=CLttOU7n6sI&t=977s) And finally, you should use that deterministic final outcome to produce outputs that are easy to validate, even for non-coders. The code is kind of just the means to an end.

[16:26](https://www.youtube.com/watch?v=CLttOU7n6sI&t=986s) And then, of course, the kind of standard context and prompt engineering will of course remain critical to better scoping and defining that task and making it clear what the agent needs to do.

[16:36](https://www.youtube.com/watch?v=CLttOU7n6sI&t=996s) That's it for me. Again, I'm Andrew Dumit at Watershed, the sustainability AI platform. Thank you again for your time and attention.
