# How We Took an AI Agent From 41% to 97% Accurate - Milind Sunil Shah, Brainbase Labs

Milind Sunil Shah, Brainbase Labs

[Talk page](https://ai.engineer/talks/4VmugsE-iP0-we-took-ai-agent-from-41-97) · [Video](https://www.youtube.com/watch?v=4VmugsE-iP0)

## How Brainbase took Domino’s competitor-pricing agent from 41% to 97% accuracy

Milind Sunil Shah follows a competitor’s pizza price from a local listing to a cell an analyst can sign off on, showing how Brainbase used failed runs to improve accuracy, halve cost per source and cut runtime.

### What does a pizza cost down the street?

[0:12](https://www.youtube.com/watch?v=4VmugsE-iP0&t=12s)

Milind Sunil Shah, a member of the technical staff at Brainbase Labs, builds and manages internal agents for enterprises. His example begins in the CMO’s office for one Domino’s region. Domino’s had more than 20,000 stores across more than 90 markets, but the question this office wanted answered was local. What was the lowest price a customer could pay for a particular item at a nearby competitor, given the store, ordering channel, fulfillment method and available promotions?

An analyst would open the competitor’s website or delivery app, select the local store, compare delivery and takeaway prices, look for promotions and enter the cheapest qualifying price in a spreadsheet. The pricing team used the finished sheet for analysis. An error early in that sequence could survive every subsequent step. A price from the wrong store looked just as plausible in a completed cell as the right one.

![Six cards trace the manual task from opening a competitor’s site to entering prices in a spreadsheet. A warning identifies two ways a cell can be wrong, and a bottom strip shows where the agent flags uncertainty for a human.](https://ai.engineer/api/talk-essay-media/98de4230f4ace4b4b76483a25c09b7a060d140a7f36ca9026dbc5b7e3f605d5c)

Six steps for collecting competitor prices

Brainbase’s first agent achieved 41% accuracy. By version eleven, accuracy had reached 97%, cost per source had fallen from $2 to $1, and elapsed time per run had fallen from 129 to 71 minutes. A source was an item–competitor–website or app combination, so checking a competitor’s own site and checking a delivery app were separate work. Shah attributed the gains to changes in instructions, skills and playbooks. The team used the same model throughout.

### 1. Agree on what the analyst signs off on

[2:23](https://www.youtube.com/watch?v=4VmugsE-iP0&t=143s)

Brainbase began its six-step process by asking the people doing the job to describe a successful day. What did they deliver, who judged it, how much work did it take, and which decisions required a person? An analyst spent about two hours of active work each day preparing the spreadsheet, and every cell was checked. That baseline measured human attention. The 129 and 71 minutes measured elapsed time for an agent run.

The regional office wanted a faster, repeatable collection process that retained the analyst’s judgment on edge cases. The agent would read the list of items, find and interpret prices, populate the existing report and flag anything it could not verify. The analyst would sign off before the pricing team used it. Accuracy, cost per source and runtime all mattered. Even the feasibility of discount search was an open question at kickoff, so the team needed a way to tell when the agent had found a price but missed the offer a customer could actually use.

![A four-row table applies the kickoff questions to the pricing team’s spreadsheet. It gives the analyst’s time per run, the cell-by-cell accuracy check, and the human sign-off requirement. A footer lists the measures of success.](https://ai.engineer/api/talk-essay-media/6f3c3208f657ec063539452289386837a9688b85e2710b8c7ffe347785ff27a5)

Domino’s brief for collecting local competitor prices

### 2. Connect each target to a source, a pricing rule and a cell

[3:35](https://www.youtube.com/watch?v=4VmugsE-iP0&t=215s)

Shah uses *ontology* for the definitions that make the analyst’s task executable. The team wrote down what to collect, where to find it, where to put it and what to record when the evidence ran out. Brainbase gave the agent a memory primitive resembling a Postgres database. A targets table held the requested items, a rules file described how to interpret an offer, and a price observations table held what the agent found.

Consider Shah’s hypothetical target 14. The request was for a large Brand B pizza, for delivery, on app B. Suppose the agent opened the listing and saw 24 struck through beside a sale price of 19. Both numbers belonged to the correct product. A previous failure had prompted an explicit rule to enter what the customer would pay at checkout. The agent would therefore record 19 as the observed price, retain 24 as the full price and save the source alongside the target ID.

The report already contained placeholder item IDs. To fill the cell marked 14, the agent could look up the corresponding observation and substitute its price. That arrangement also worked in reverse during review. An analyst questioning 19 could trace the cell to the page the agent read and the rule that made 19 the answer.

![Four linked panels follow Brand B’s large delivery pizza from target ID 14 to a spreadsheet cell. The agent records the checkout price of 19, the full price of 24, and a source link, then replaces ID 14 with 19.](https://ai.engineer/api/talk-essay-media/7418f3fbf76aba05c3c39f02539eea69bebd98f71c4d92857a9d6f36b0f699bf)

The agent looks it up instead of guessing.

Several definitions came from live traces. A large pizza could qualify as an individual item or as part of a bundle. A takeaway listing could not satisfy a delivery target, a distinction the agent had confused. An offer of 30% off required the agent to calculate the discount before entering the price. When it could not verify an answer, the prescribed output was a gap.

### 3. Reconstruct the input, then choose how to judge the answer

[5:43](https://www.youtube.com/watch?v=4VmugsE-iP0&t=343s)

Shah separated two evaluation questions. Could the agent reproduce an analyst’s answer on a past case, and what could the team check during a live run? For the offline benchmark, past spreadsheets supplied the expected prices. To replay a case, Brainbase also needed the offers an analyst could see that day. The team looked for dated snapshots of competitor websites on archive.org, but the same approach failed for delivery apps, which served their data differently. Starting with version one, Brainbase saved browser screenshots during each run so its internal benchmark included the offers the agent had actually seen.

Shah distinguished three ways to score the work. First, deterministic checks in code could inspect the spreadsheet and its arithmetic. Did the agent replace each placeholder ID with a price? Did it calculate the discount correctly? If a price moved 20% up or down relative to the previous run, did it raise a flag? The last check tested whether the agent had called attention to a change. A substantial price change could still be genuine.

Second, an LLM as a judge could read the trace to decide whether the selected item matched the target and whether the agent applied the right promotion. Third, an agent as a judge could revisit each flagged price. It could take multiple turns and make tool calls to determine whether the evidence justified the flag. The human analyst then received the final flags and investigated the reason when something had gone wrong.

![Two columns show how Domino’s scores replayed pages against corrected spreadsheets and checks live runs. The live checks progress from code to an LLM, a page-checking agent, and an analyst. The LLM’s questions are highlighted.](https://ai.engineer/api/talk-essay-media/82afe0fec0e1bcfb55205c11336567d479c3136e4187d5bf30af6e10abb4a7bb)

A benchmark before shipping, evals on every run.

### 4. Measure one item on one app in one city

[7:42](https://www.youtube.com/watch?v=4VmugsE-iP0&t=462s)

Brainbase chose a slice small enough to score from day one. One item on one app in one city still required the agent to select the store, interpret an offer and deliver a completed report. At 41% accuracy, more than two hours a run and about $2 per source, version one gave the engineers specific misses to investigate and a baseline for what each attempt cost.

The team made fixes before widening the assignment. It added burgers and takeaway options, then another app, then other websites. That order matters when reading the version-one and version-eleven figures. The later agent covered a broader task, and the engineers had a record of failures from the smaller task to test as they expanded.

![Two bars show 100% completion and 41% of cells correct without an analyst’s fix. Five mistake types point to specific fixes. A July timeline tracks expansion from two medium pizzas on one app in one city to competitors’ own sites.](https://ai.engineer/api/talk-essay-media/a7b86c656d21a50002ceb2928561786fea6fbe6f6e72dfc81a0503e262825df2)

Version one: done, and 41% right.

### 5. Score a failure, change one thing and measure the cost

[8:49](https://www.youtube.com/watch?v=4VmugsE-iP0&t=529s)

Iteration meant returning to the definitions, benchmark and running agent. Shah advised prioritizing failures by frequency and expense. A recurring miss might warrant a new pricing rule, a benchmark case that preserved the offer, or a change in how the agent used the browser. He recommended changing one thing at a time where possible and fixing instructions before considering a model change. Additional verification had a price of its own, so the team checked what each fix did to runtime and cost.

### The $21 deal below the linked section

[9:44](https://www.youtube.com/watch?v=4VmugsE-iP0&t=584s)

For one Brand B target, the agent received a link to a large pizza for delivery on the competitor’s own website. The link highlighted the relevant section. The agent clicked through, found the large pizza, read its price and finished. During review, an analyst spotted a cheaper $21 deal farther down the same page.

The item selection was correct. The mistake was treating the first matching offer as sufficient evidence of the lowest price. Brainbase changed the instruction to require the agent to examine the entire page, including offers outside the section the link opened.

![A delivery menu contrasts the agent’s pick, a large pizza priced at 26, with the analyst’s find in Deals, a large pizza and drink priced at 21. Beside it are the resulting instruction and the check applied on every run.](https://ai.engineer/api/talk-essay-media/c477459ade135f8f9bc684a8b180b30fd13dcb315353cc2457a4870ff61768e4)

One correction became a rule and a check.

### The promotion card the agent found at minute 13

[10:28](https://www.youtube.com/watch?v=4VmugsE-iP0&t=628s)

On Brand D’s promotions page, prices appeared inside images. The agent inspected all 26 promotional images, then examined them again at four, five and six times magnification. It found the right card at minute 13 and continued zooming to double-check it. At minute 15, the team stopped the run manually. Here the question was how to narrow the search and give verification an endpoint.

An engineer reviewed the sequence of tool calls alongside the website. The HTML DOM offered information the agent could inspect before reading pixels, including image titles and alt text. Brainbase changed the browser instructions to examine those first, then inspect the relevant images. It also set a limit on zooming.

For this case, the revised agent used one-third as many image reads, cost over 50% less and produced a finished report. The trace showed something the runtime figure alone could not reveal. By minute 13 the agent had already found the card, and its remaining calls were spent rechecking it.

![Two panels compare runs on Brand D’s promotions page. The first used 61 image reads, cost $6.93, and stopped at minute 15. The second used HTML titles to narrow the search and finished the report with 21 image reads for $3.17.](https://ai.engineer/api/talk-essay-media/0f80b95dff8a37ea12d576eff22b5220557647ba0d2e5cd0718b1960808ba96c)

Same answer, half the cost.

### 6. Give the pricing team ownership of the next correction

[11:56](https://www.youtube.com/watch?v=4VmugsE-iP0&t=716s)

At handoff, Shah asks teams to write down who runs the agent, who reviews the output, who converts a correction into a rule or benchmark case, and who decides when a version is ready to ship. Fixing a cell gives the pricing team the right number today. Preserving the case lets the owner of the next version check whether it can find that number unaided.

In the Domino’s deployment, the agent could run daily on the team’s schedule or on demand through an email trigger. Analysts who had checked every cell now reviewed exceptions and could add corrections to the benchmark themselves, with Brainbase available to help. At the time Shah described the project, the teams were working on expanding it to additional markets and areas.

## Key takeaways

- Give each price a target ID, fulfillment method, source and interpretation rule so an analyst can trace a disputed cell back to the offer.
- Save what the browser showed during collection. Historical spreadsheets supply expected answers, but replaying a pricing case also requires the offers available at the time.
- Use code for arithmetic and completeness, a trace-reading LLM for item and promotion selection, and a tool-using judge for flags that need investigation.
- Inspect the tool calls around a failure before adding more verification. Brand B needed a wider search, while Brand D needed a cheaper way to select images and a stopping rule.

## Slides

### Domino’s local competitor pricing

[0:25](https://www.youtube.com/watch?v=4VmugsE-iP0&t=25s) [Domino’s store and market counts appear on the left. On the right, a white diamond emblem and the Domino’s logo overlay a dark group portrait above a question about local pizza prices.](https://assets.ai.engineer/sites/aiecode2025/sha256/d0848052c3bb8f2a5447d76819c90317bf52ac76377c905810159c67dab77511.webp)

Hill-climbing back-office agents
The job   03 / 22

THE CUSTOMER
Domino’s Pizza
Domino's

22,500+
stores

90+
markets

Domino’s

CMO OFFICE · LOCAL COMPETITOR PRICING
What does a pizza cost at the competitor down the street — in
this market, through this channel, with this promotion, today?

### Six steps for collecting competitor prices

[0:58](https://www.youtube.com/watch?v=4VmugsE-iP0&t=58s) [Six cards trace the manual task from opening a competitor’s site to entering prices in a spreadsheet. A warning identifies two ways a cell can be wrong, and a bottom strip shows where the agent flags uncertainty for a human.](https://assets.ai.engineer/sites/aiecode2025/sha256/98de4230f4ace4b4b76483a25c09b7a060d140a7f36ca9026dbc5b7e3f605d5c.webp)

Hill-climbing back-office agents
The job   04 / 22

THE MANUAL WORKFLOW
Easy steps. Each needs accurate info from a messy source.

01  Open a competitor's site or app
02  Pick the right local store
03  Compare delivery with takeaway
04  Find the cheapest item that qualifies
05  Account for promos and missing items
06  Type it all into the spreadsheet

• Pick the wrong store or miss one banner promo, and you've got a cell that looks finished and is wrong.

THE AGENT  read targets → browse each site → interpret prices → fill the spreadsheet → flag what it's unsure of → a human

### Price accuracy, cost per source and runtime across eleven versions

[1:34](https://www.youtube.com/watch?v=4VmugsE-iP0&t=94s) [A comparison table shows correct prices rising from 41% to 97%, cost per source falling from $2.11 to $1.01, and runtime falling from 129 to 71 minutes against an analyst’s 120 minutes.](https://assets.ai.engineer/sites/aiecode2025/sha256/e7b90405c7f42a4ff836c743a5bca539710e120304f08102a817ff4927bc2780.webp)

Hill-climbing back-office agents
The job   05 / 22

WHERE IT ENDED UP · ELEVEN VERSIONS
Hill-climbing took it from worse than an analyst to 97% right.

VERSION 1                    VERSION 11
Prices correct               41% → 97%

Cost per source              $2.11 → $1.01
each site or app checked

Minutes per run              129 → 71
an analyst takes 120

How we did it: six steps.

### The six-step method from kickoff to handoff

[2:02](https://www.youtube.com/watch?v=4VmugsE-iP0&t=122s) [Six numbered milestones run left to right along a dotted line. Each milestone pairs a stage of agent development with a short description.](https://assets.ai.engineer/sites/aiecode2025/sha256/29658eaf2f3c743421e5f686f8e8614e4355e436f2305b178b33ce3af3cf338b.webp)

Hill-climbing back-office agents
Six steps   06 / 22

THE METHOD
Six steps. For each one: the questions to answer,
then how we answered them at Domino’s.

01  Kickoff
Agree on the win before you build

02  Ontology
Write down what the agent needs to know

03  Benchmarks
A fixed test before each version ships, plus evals of live runs

04  First agent
Ship v1, scored from its first run

05  Iterate
Score, fix, ship: every change gets scored

06  Handoff
The team owns it; you stay on call

### Kickoff questions for the one-page brief

[2:22](https://www.youtube.com/watch?v=4VmugsE-iP0&t=142s) [The Kickoff heading sits beside a four-question brief covering human sign-off, current time and cost, required accuracy, and actions the agent must leave to a human.](https://assets.ai.engineer/sites/aiecode2025/sha256/79f14478ddbe2fd4b0e43a59626e229b6ef222b013ce9dbf0a8ea28aa0016316.webp)

Hill-climbing back-office agents
1 · Kickoff   07 / 22

STEP 01
Kickoff
Agree on the win before you build.

THE ONE-PAGE BRIEF ANSWERS
▪ What does a human sign off on today?
▪ How long does it take now, and what does it cost?
▪ How accurate does it need to be, and who judges?
▪ What must the agent never do alone?

### Domino’s brief for collecting local competitor prices

[2:44](https://www.youtube.com/watch?v=4VmugsE-iP0&t=164s) [A four-row table applies the kickoff questions to the pricing team’s spreadsheet. It gives the analyst’s time per run, the cell-by-cell accuracy check, and the human sign-off requirement. A footer lists the measures of success.](https://assets.ai.engineer/sites/aiecode2025/sha256/6f3c3208f657ec063539452289386837a9688b85e2710b8c7ffe347785ff27a5.webp)

Hill-climbing back-office agents
1 · Kickoff   08 / 22

AT DOMINO’S
The brief.
A faster, repeatable way to collect local competitor pricing, without losing the judgment
edge cases need.

What does a human sign off on today?
The pricing team’s spreadsheet

How long does it take now, and what does it cost?
An analyst, at least two hours a run

How accurate does it need to be, and who judges?
Every cell checked against the analyst-corrected spreadsheet. Like a real
analyst, not a demo.

What must the agent never do alone?
Sign off. It flags what it isn’t sure of.

WIN, IN NUMBERS   accuracy · cost per source (each site or app checked) · runtime · can discount search be automated at all?

### Ontology — what the agent needs to know

[3:33](https://www.youtube.com/watch?v=4VmugsE-iP0&t=213s) [Four questions ask what the agent collects, where answers come from and go, how experts define terms, and how to handle uncertainty. A rule below calls for one definition per term.](https://assets.ai.engineer/sites/aiecode2025/sha256/27c722bdf265ff268001c662b9676a5067362d158ac4711eb10ca8e955b90335.webp)

Hill-climbing back-office agents
2 · Ontology    09 / 22

STEP 02
Ontology
Write down what the agent needs to know
about your world, in a form it can look up.

ANSWER THESE
▪ What exactly is the agent collecting?
▪ Where does each answer come from, and where does it go?
▪ What does each term mean to your experts?
▪ What should the agent do when it can’t tell?

RULE
Give every term one definition. What you don’t write down,
the agent guesses.

### The agent looks it up instead of guessing.

[4:06](https://www.youtube.com/watch?v=4VmugsE-iP0&t=246s) [Four linked panels follow Brand B’s large delivery pizza from target ID 14 to a spreadsheet cell. The agent records the checkout price of 19, the full price of 24, and a source link, then replaces ID 14 with 19.](https://assets.ai.engineer/sites/aiecode2025/sha256/7418f3fbf76aba05c3c39f02539eea69bebd98f71c4d92857a9d6f36b0f699bf.webp)

Hill-climbing back-office agents
2 · Ontology    10 / 22

AT DOMINO’S · WHAT WE BUILT
The agent looks it up instead of guessing.

1 · WHAT TO CHECK
targets
ID   PRICE TO CHECK
13   Brand A · large · takeaway
14   Brand B · large · delivery
     on App B
15   Brand B · large · takeaway
FOR 14, THE APP SHOWS
2̶4̶   19
We numbered every price they track. Its number is its ID.

2 · WHAT “PRICE” MEANS
rules file
What you pay at checkout.
So the price is 19, not 24.
Written once, looked up on every run.

3 · WHAT IT SAVES
price_observations
ID          14
price       19
full price  24
source      page link
Kept so a reviewer can check it against the page.

4 · WHERE IT GOES
1 LARGE PIZZA
         DELIVERY       TAKEAWAY
Brand A  ID 12          ID 13
Brand B  19             ID 15
         was ID 14
Brand C  ID 16          ID 17
Each cell starts with an ID in it. The agent swaps ID 14 for its price.

THE OTHER RULES WE WROTE
A large pizza
anything with one in it, bundles included
Delivery ≠ takeaway
never fill one from the other
A promotion
do the basket math; it isn’t a price
Can’t verify it?
write a gap, never a guess

When a cell is wrong, it points to the page and the rule behind it.

### Benchmarks and evals

[5:42](https://www.youtube.com/watch?v=4VmugsE-iP0&t=342s) [A checklist separates benchmark questions about checked answers, replayable inputs, and field scores from live evaluation questions about code checks, judges, and review.](https://assets.ai.engineer/sites/aiecode2025/sha256/b746413271220e3053c1a97bd5996e6c872a75bac0acbc023dc12a2034044616.webp)

Hill-climbing back-office agents
3 · Benchmarks    11 / 22

STEP 03
Benchmarks
and evals
Score the agent two ways: a fixed test
before it ships, and evals on every live run.

ANSWER THESE
BENCHMARK · BEFORE EVERY VERSION SHIPS
▪ Which past cases have answers someone already checked?
▪ Can you rerun each case on the exact inputs it saw?
▪ How is each field scored against the answer key?

EVALS · ON EVERY LIVE RUN’S TRACE
▪ What can code check, every time?
▪ What needs a judge: an LLM, or an agent that re-checks?
▪ Who reviews what’s flagged, and where do fixes go?

RULE
Score every field. Use code/deterministic checks where you can, a judge where you can’t.

### A benchmark before shipping, evals on every run.

[6:41](https://www.youtube.com/watch?v=4VmugsE-iP0&t=401s) [Two columns show how Domino’s scores replayed pages against corrected spreadsheets and checks live runs. The live checks progress from code to an LLM, a page-checking agent, and an analyst. The LLM’s questions are highlighted.](https://assets.ai.engineer/sites/aiecode2025/sha256/82afe0fec0e1bcfb55205c11336567d479c3136e4187d5bf30af6e10abb4a7bb.webp)

Hill-climbing back-office agents
3 · Benchmarks    12 / 22

AT DOMINO’S
A benchmark before shipping, evals on every run.

Benchmark
ANSWER KEY
Old spreadsheets the pricing team had already corrected
REPLAYED PAGES
archive.org snapshots of competitors’ own sites
DELIVERY APPS
Our own capture of every page the agent visited, from v1 on
SCORED BY
Code, cell by cell, against the answer key

Evals    CHEAPEST CHECK FIRST
Code
every run
One price or one gap per ID
Delivery never filled from takeaway
Discount math recomputed
Moved over 20% since last run? Flagged

LLM judge
reads the trace
Right item? Does the promo actually apply?

Agent judge
opens the page
Re-checks every flagged price on the live page

Analyst
final say
Fixes what’s still wrong, and tags the reason

### First agent — scoring a narrow first version

[7:41](https://www.youtube.com/watch?v=4VmugsE-iP0&t=461s) [Five questions cover the first agent’s scope, field-level accuracy, runtime and unit cost, mistakes, and the score needed to expand.](https://assets.ai.engineer/sites/aiecode2025/sha256/bb42ad775f0e81b026f3d083a734d9c76b4d59000672ea135ec005757552394e.webp)

Hill-climbing back-office agents
4 · First agent    13 / 22

STEP 04
First agent
Ship a narrow first version, in your stack,
scored from run one.

ANSWER THESE
▪ What’s the smallest slice worth scoring?
▪ How accurate is it, field by field?
▪ How long does a run take, and what does each unit cost?
▪ What did it get wrong, and where does each fix belong?
▪ What score earns the next slice?

RULE
Version one’s job is to be measured, not to be good. Widen the scope when the score holds.

### Version one: done, and 41% right.

[8:14](https://www.youtube.com/watch?v=4VmugsE-iP0&t=494s) [Two bars show 100% completion and 41% of cells correct without an analyst’s fix. Five mistake types point to specific fixes. A July timeline tracks expansion from two medium pizzas on one app in one city to competitors’ own sites.](https://assets.ai.engineer/sites/aiecode2025/sha256/a7b86c656d21a50002ceb2928561786fea6fbe6f6e72dfc81a0503e262825df2.webp)

Hill-climbing back-office agents
4 · First agent    14 / 22

AT DOMINO’S
Version one:
done, and 41% right.

Completed    100%
Correct      41%
share of cells an analyst didn’t have to fix
129 min · $2.11 per source · Claude Opus 4.6

V1’S MISTAKES, BY REASON → WHERE THE FIX GOES
Wrong location → targets + glossary
Wrong channel → glossary
Missed promotion → that source’s instructions
Missing item → glossary: unavailable ≠ zero
Messy output → output format

STARTED NARROW · WIDENED AS THE SCORE HELD
JUL 15 · FIRST INSTRUCTIONS
App A, one city, 2 medium pizzas
JUL 16
→ + burgers, + takeaway
JUL 24
→ + App B
JUL 30
→ + competitors’ own sites

### Step 05 · Iterate

[8:48](https://www.youtube.com/watch?v=4VmugsE-iP0&t=528s) [A four-question checklist asks which failures to fix, how to measure a change, and which correction to keep as a benchmark. A dark panel highlights the rule to change one thing at a time.](https://assets.ai.engineer/sites/aiecode2025/sha256/50543e26de96e51cc121d9d6336e74e5098864e3669890f9271fa027fc1aa4e2.webp)

Hill-climbing back-office agents
5 · Iterate    15 / 22

STEP 05
Iterate
Score, fix, ship. Every change is scored before it reaches the people who use the output.

EACH ROUND, ANSWER
▪ Which failures are most frequent and most expensive?
▪ What one change should fix them?
▪ Did the score go up, and did cost or runtime go down?
▪ Which correction becomes the next benchmark case?

RULE
Change one thing at a time. Fix the instructions before you swap the model.

### Eleven versions, one scoring rule.

[9:25](https://www.youtube.com/watch?v=4VmugsE-iP0&t=565s) [A stepped chart plots Domino’s agent accuracy against cost per source, with cheaper runs to the right. It rises from v1 at about 41% to v11 at 97% and $1.01 per source. Gray dots mark setbacks at v2 and v10.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d600ecc0e752a3e89b61f0e5025cb761a56ced5c31b64afc0710e6624529644.webp)

Hill-climbing back-office agents
5 · Iterate    16 / 22

AT DOMINO’S · ACCURACY VS. COST PER SOURCE · ONE DOT PER VERSION
Eleven versions, one scoring rule.
Accuracy = share of cells an analyst didn’t have to fix. Runs nobody corrected aren’t scored.
● better on both axes    ● fell back on one

ACCURACY →
100%
90%
80%
70%
60%
50%
40%

v1
v2    Opus 4.6 → Sonnet 5 · $2.88
v3    Improved browser runtime
v4    Corrected Burger market
v5
v6
v7
v8    Instructions adjustment
v9    Corrected Pizza market
v10 · 87%
v11 · 97% · $1.01

$3.00    $2.50    $2.00    $1.50    $1.00
COST PER SOURCE · CHEAPER →

### One correction became a rule and a check.

[9:43](https://www.youtube.com/watch?v=4VmugsE-iP0&t=583s) [A delivery menu contrasts the agent’s pick, a large pizza priced at 26, with the analyst’s find in Deals, a large pizza and drink priced at 21. Beside it are the resulting instruction and the check applied on every run.](https://assets.ai.engineer/sites/aiecode2025/sha256/c477459ade135f8f9bc684a8b180b30fd13dcb315353cc2457a4870ff61768e4.webp)

Hill-climbing back-office agents
5 · Iterate    17 / 22

AT DOMINO’S · ONE CORRECTION
One correction became a rule and a check.
Brand B · 1 large pizza · delivery, on its own website

Brand B · delivery menu
LARGE PIZZAS    ← the link opens here
Large pizza    ✗ agent’s pick    26
SIDES
DEALS
Large pizza + drink    ✓ analyst’s fix    21

WHAT WE CHANGED
1    CAUGHT
An analyst found a cheaper bundle in Deals.
2    RULE · ONE NEW LINE
“Don’t stop at the section the link opens.”
3    CHECK · EVERY RUN
A cheaper bundle must win.

An analyst caught it once. The check catches it every run.

### Same answer, half the cost.

[10:31](https://www.youtube.com/watch?v=4VmugsE-iP0&t=631s) [Two panels compare runs on Brand D’s promotions page. The first used 61 image reads, cost $6.93, and stopped at minute 15. The second used HTML titles to narrow the search and finished the report with 21 image reads for $3.17.](https://assets.ai.engineer/sites/aiecode2025/sha256/0f80b95dff8a37ea12d576eff22b5220557647ba0d2e5cd0718b1960808ba96c.webp)

Hill-climbing back-office agents
5 · Iterate    18 / 22

AT DOMINO’S · ONE BROWSER FIX
Same answer, half the cost.
Brand D’s promotions page · every price is inside an image · model cost, list price

BEFORE · AUG 4 RUN
$6.93    61 image reads
Looked at all 26 promo images
Then again at 4×, 5× and 6× zoom
Found the right card at minute 13,
kept zooming to double-check and verify
Never finished: we stopped it at minute 15

AFTER · SAME PAGE, SAME DAY
$3.17    21 image reads
26 cards, titles read from the HTML
18 clearly not a meal, dropped
8 images, looked at once each
13 more, zooming into one card’s fine print
Same answer, and a finished report

So we added: past 3× zoom, stop. Open the card’s page; the fine print is often text there.

Read text first. Spend image reads only on what text can’t rule out.

### Step 06 · Handoff

[11:48](https://www.youtube.com/watch?v=4VmugsE-iP0&t=708s) [Four handoff questions cover who runs the agent, reviews its output, turns corrections into benchmark cases, and approves releases. A dark panel says handoff is complete when the team ships a version independently.](https://assets.ai.engineer/sites/aiecode2025/sha256/28c333ddea8bbd8b70a17ada865057968a32a7b2ac5c6db1b1c1f8833b55b57e.webp)

Hill-climbing back-office agents
6 · Handoff    19 / 22

STEP 06
Handoff
The agent, the benchmark and the instructions belong to the team that uses them. You stay on call.

ANSWER THESE
▪ Who runs the agent and reviews its output?
▪ Who turns a correction into a rule and a benchmark case?
▪ Who decides a new version is ready to ship?
▪ When did you last have to change anything?

RULE
Hand off the loop, not just the agent. You’re done when they ship a version without you.

### The loop is theirs now.

[12:24](https://www.youtube.com/watch?v=4VmugsE-iP0&t=744s) [Three cards describe the Domino’s handoff. The team schedules runs, analysts review exceptions, and the team turns corrections into rules and benchmark cases before shipping the next version. The last card outlines expansion to more markets.](https://assets.ai.engineer/sites/aiecode2025/sha256/87491810f6d3ac2537115aeabc3ce1548fa7c433a6aa7b30fbdf1da0896a58a3.webp)

Hill-climbing back-office agents
6 · Handoff    20 / 22

AT DOMINO’S
The loop is theirs now.

THEY RUN IT
The agent runs on their schedule
Analysts review exceptions instead of collecting prices

THEY IMPROVE IT
Each correction becomes a rule and a benchmark case
They ship the next version

FROM HERE
More markets on a schedule, without more analyst time
We stay on call

Handoff isn’t a date. It’s when the people who correct the output can change the agent.

## Full transcript

[0:12](https://www.youtube.com/watch?v=4VmugsE-iP0&t=12s) Hey, so I'm Milind, MTS at Brainbase. We help enterprises deploy internal agents at scale and manage them. Today I'm gonna be talking about hill climbing back office agents and how we did that successfully with a customer.

[0:26](https://www.youtube.com/watch?v=4VmugsE-iP0&t=26s) So yeah, our customer in this case that I'm go-gonna be talking about throughout, and like the framework we used repeatedly is, is, is Domino's. As you all know, they have twenty-plus thousand stores, ninety-plus markets. We started off with a single region with their CMO office. We wanted to answer this question that they kept asking: What does a pizza cost at the competitor down the street? What, what does it cost in their delivery apps? What does it cost on their actual website, with promos, without promos? Things like that. They essentially wanted a spreadsheet at the end with the lowest price

[0:55](https://www.youtube.com/watch?v=4VmugsE-iP0&t=55s) po-possible for a bunch of different items. Before we built the agent for them, they had an analyst manually doing these steps. They all look easy in isolation. You open a competitor's website, pick the right store, compare prices with delivery, takeaway, find the right promos, find the cheapest price, and write it all down into a spreadsheet. It, it all looks easy enough, but if you get one of these steps wrong, you look like you, you, you get a cell at the end that looks finished, but it's just wrong. We had to build an agent that does the same.

[1:25](https://www.youtube.com/watch?v=4VmugsE-iP0&t=85s) Reads a bunch of items, finds the website, interprets prices, fill the spreadsheets, hand it off to an agent, and flag anything that it thinks it's not sure of.

[1:36](https://www.youtube.com/watch?v=4VmugsE-iP0&t=96s) W-When we created version one of this agent for them, we had forty-one percent accuracy. By the end, at version eleven, we had ninety-seven percent accuracy. Our cost per source, per source means, um, per item, per, per br- competitor, per website, uh, went from two dollars per source to one dollar per source, and the amount of minutes it took per run went from one twenty-nine to seventy-one. All of this when just was-- just, just through instructions and, and skills and playbooks changing. Here is like the six steps that we use,

[2:06](https://www.youtube.com/watch?v=4VmugsE-iP0&t=126s) not just with Domino's, but with other customers as well. We start off with kickoff, followed by ontology, benchmarks, first agent, iterating, and handing off. I'll talk about what these steps mean in detail and what we did for Domino's in each of these, and hopefully it's a repeatable framework that you all can use internally as well. So step one is kickoff. You should ideally talk to your customer or end user and agree on what the win looks like. As of today, what, what is the job? What does a human sign off on at the end of it? How long does it take? How much does

[2:36](https://www.youtube.com/watch?v=4VmugsE-iP0&t=156s) it cost? How accurate does your agent need to be? Who judges the output? And what must the agent never do alone? What assumptions should it never make? For Domino's, we ended up creating this brief that worked really well for us. Our outcome at the end had to be a faster, repeatable way to collect local competitor pricing without losing the judgment edge cases need. So we asked them a- all of these questions. As of today, the analysts create a spreadsheet and the pricing team, uh, fin-finally analyzes it. An analyst

[3:06](https://www.youtube.com/watch?v=4VmugsE-iP0&t=186s) spent two hours daily manually, actively working on creating the spreadsheet. Every cell was then checked and, and just confirmed, and the agent at the end just had to ask for the analyst to sign off and flag anything else it had to. Our win in numbers looked like just having the highest accuracy, the lowest cost per source as possible, runtime to be as low as possible, and before we even started, they weren't even sure if we can do dis-discount search at all. Step two is ontology. What does that exactly mean?

[3:36](https://www.youtube.com/watch?v=4VmugsE-iP0&t=216s) We, we try to write down all the terms that, that the agent needs to know to never make assumptions, which means we need to tell the agent exactly what it's supposed to collect, where each answer comes from, where does it go, what does each term mean to the analysts that, that actually do the term, uh, do the work manually, and what should the agent do when it can't figure something out. If, if there's one rule here, it is that every term should ideally have a definition. If you don't write something down, the agent's gonna guess and probably get something wrong.

[4:07](https://www.youtube.com/watch?v=4VmugsE-iP0&t=247s) For Domino's, here's what it looked like. We at Brainbase have a memory primitive of our own that has memory that looks like a Postgres database that the agent can do whatever it wants with. We gave it a targets table where we gave it all the items. In this case, let's look at ID number fourteen. ID fourteen is brand B, it's a large pizza, uh, takeaway method-- sorry, it's delivery, and on app B. The agent hypothetically does browser use, finds the price on the website, finds that it's twenty-four and nineteen. Twenty-four is

[4:37](https://www.youtube.com/watch?v=4VmugsE-iP0&t=277s) the striked out price, nineteen is the sale price. We also now have a rules file where we actually tell the agent to specifically mark down the price that it, that a user pays at checkout, not twenty-four. This came from like a valid failure. What it saves at the end, we created a table again called price observations, where it, with the corresponding ID, stores what price it saw, what the full price was, and the actual source. And at the end, we give it a spreadsheet that has all placeholder item IDs, and all it has to

[5:07](https://www.youtube.com/watch?v=4VmugsE-iP0&t=307s) do is look up the ID and replace the price. When a cell is wrong, it points to the exact page and the rule behind why it chose the selected price. Other rules that we mentioned that came through actual live traces were definition of what a large pizza looks like. It's anything that's actually the item itself or a bundle. Delivery is not equal to takeaway. It, it got confused with that. It's supposed to do promotion math. If it's thirty percent off the price, it should do that before entering it in the spreadsheet.

[5:37](https://www.youtube.com/watch?v=4VmugsE-iP0&t=337s) And if it can't verify it, write it down as a gap and do not guess. Step three is figuring out benchmarks and evals. Now, now these steps can be interchangeable slightly, but essentially you're trying to figure out what you can run offline evals on and what you can run online evals on. For offline evals, you should ideally start off by asking them if they have past cases that they already have answers for. In this case, we had analysts from Domino's who had spreadsheets that they worked on. Can we actually rerun those Um, input

[6:07](https://www.youtube.com/watch?v=4VmugsE-iP0&t=367s) cases to get the same final output. And then at the end, how do you figure out how to score each thing that the agent's doing? For evals, you try to figure out what you can check deterministically, what you can check via an LLM as a judge, and what you can check as an agent as a judge. That's a concept that we think is slightly more unique, where if there's an output artifact, you probably want an agent to take multiple turns, multiple tool calls to figure out how correct that, that artifact is. And at the end, what happens when something goes wrong and what happens when it's flagged?

[6:37](https://www.youtube.com/watch?v=4VmugsE-iP0&t=397s) You should try to score as many things as you can with these three things. So yeah, at Domino's, our benchmark essentially started off with old spreadsheets that they had. We tried to create the input environment by going to archive.org and having snapshots of comparative websites at those given dates. What we saw that-- was that it didn't work with delivery apps because they didn't sl-serve data the same way. So we ended up storing browser screenshots for everything the agent was doing, starting from version one of the agent, and that's how we created our, our environment for, for the internal

[7:07](https://www.youtube.com/watch?v=4VmugsE-iP0&t=427s) benchmark. For evals, we used code for things like checking if every ID cell was replaced with the price, if the discount map was computed correctly, if the price was twenty percent plus or minus compared to the previous run, was it flagged or not? LLM as a judge just checks the trace to see if the right item was selected, if the right promo was applied. Agent as a judge actually goes to every flagged price and actually confirms if the flag was valid or not. And the analyst at the end will go look at the final flags and

[7:38](https://www.youtube.com/watch?v=4VmugsE-iP0&t=458s) try to figure out what the reason for it was if something went wrong. Step four is actually shipping some version of a first agent. You should ideally ship a narrow first agent in your stack that you can score from ver- from, from day one. You should probably figure out what the smallest slice worth scoring is. You should figure out how accurate things should be, how long... And, and you should use this agent to figure out how long does the run take, what, what does it cost, what does it get wrong, and things like that. Version one's job is to be measured, it's not to be

[8:07](https://www.youtube.com/watch?v=4VmugsE-iP0&t=487s) good. You should widen the score-- uh, widen the scope after you get some baseline score that you're trying to meet. For Domino's, uh, like I mentioned earlier, version one for us was just forty-one percent correct. It took more than two hours. It was two dollars per source, and, and it just wasn't working well. We found a bunch of changes, made fixes, and then improved and, and increased the scope. So we started off with just one app, one item in one city, moved on to adding burgers and takeaway

[8:37](https://www.youtube.com/watch?v=4VmugsE-iP0&t=517s) options, moved on to adding another app, and finally other websites as well. So starting off small, narrow scope worked out really well for us to, to figure out how we can fix things. Step five is to just iterate. You should score, fix, and ship. Here is where you'll do steps two, three, and four. Keep iterating, improving your ontology, improving your evals, your benchmarks to get things that work. You should figure out which failures are most expensive, most frequent, fix them, what changes should be used to fix

[9:07](https://www.youtube.com/watch?v=4VmugsE-iP0&t=547s) them. If that change results in increased runtime or cost, if yes, are you okay with that increased runtime and cost? And how can you use those failures to create and add to your benchmark cases or eval criteria? Ideally, change one thing at a time and fix the instructions before you want to change the model entirely. At Domino's, we-- I, I highlighted two examples. Before I get to that, here is a graph of how our progression went. V1 started off at forty-one percent, um, more than two dollars a source.

[9:37](https://www.youtube.com/watch?v=4VmugsE-iP0&t=577s) V11 after a bunch of changes is one dollar a source and, and ninety-seven percent accurate. So yeah. Here's an example of a very simple rule change that we made. We gave the agent a link of, of a large pizza to find for brand B via delivery on its own website. It went to the link which had the section already highlighted. It clicked on it, found the large pizza, found the price, and ended there. The analyst caught, that is somebody manually caught at the end, that there

[10:07](https://www.youtube.com/watch?v=4VmugsE-iP0&t=607s) was actually a deal below that website, below on the same page that was just twenty-one dollars instead and, and it missed that, right? So all we had to do there was add a simple rule to not stop at the section that it gets linked to, but actually look at the entire page. And a simple change like that helped us just, like, catch this and, and make sure it doesn't happen again. Another more intricate change that we had to make was actually browser use behavior of said agent. So in one of the runs, we saw that for brand D,

[10:37](https://www.youtube.com/watch?v=4VmugsE-iP0&t=637s) this, this is a brand called brand D promotions page, where all the promotion prices were inside images, so not actually captured in text. What the agent did is it looked at all twenty-six promo images to find the item it, it was looking for. It looked at, looked at them again by zooming in f- four x, five x, six x. It found the right card at minute thirteen, but it still kept zooming to double-check and verify, and it, it was just in a loop, and we had to manually stop it at minute fifteen. We realized after

[11:07](https://www.youtube.com/watch?v=4VmugsE-iP0&t=667s) manually looking at the trace and the website, that the HTML DOM is actually super important and could have saved the agent a lot of work here. So we, we tell the agent in, in our latest rules to go look at the HTML DOM first, look at things like image titles, alt text, before looking at the actual image reads and just stopping past a certain zoom level. What that resulted in is, is one-third the number of image reads, reducing cost by over fifty percent, and just us getting an actual answer and a finished report. So while we

[11:37](https://www.youtube.com/watch?v=4VmugsE-iP0&t=697s) have automated checks and eval criteria to help us catch a lot of things, some things require our engineers to actually look at traces and, and browse around used tool calls and, and just see what went wrong. So yeah, we made a bunch of these changes, got to V11, got to ninety-seven percent accuracy, and now it's just figuring out, "Hey, like, like, what next?" So, so our, our way of this is trying to figure out how best to hand off to the customer at the end. We wanna figure out who's running these agents, who's reviewing the outputs, who's trying to correct, who's trying to take the

[12:07](https://www.youtube.com/watch?v=4VmugsE-iP0&t=727s) corrections into and converting it into rules and benchmark cases, and who decides when a new version is ready to ship. All these questions are things you should probably have written down. Um, you're trying to hand off an entire loop and not just the agent. Ideally, the user should be empowered enough to make these changes if they're smart enough themselves. So for, for Domino's, that's exactly what we did. The agent now runs on their schedule, either multi-- um, daily or even via email trigger manually. Analysts review only the exceptions at this point. Each

[12:36](https://www.youtube.com/watch?v=4VmugsE-iP0&t=756s) correction that they catch, they're automatically able to add as a benchmark case. If they can't, then they obviously-- we're always available for them. And we are now at a place where we are expanding with them, um, trying to get into multiple markets and multiple other, um, areas.

[12:54](https://www.youtube.com/watch?v=4VmugsE-iP0&t=774s) So yeah, this is basically how we, we used, um, our framework to work with Domino's to go from an agent that barely works to an agent that works almost perfectly while reducing the cost. We found this framework to be very effective for various other use cases, especially where you have messy sources of data. We're trying to figure out the right answer and, um, thanks.
