# Engineering Agency out of the Happy Path — Matthew Jewkes, Standard Cybernetics

Matthew Jewkes, Standard Cybernetics

[Talk page](https://ai.engineer/talks/wWisOHEcMPE-engineering-agency-out-happy-path) · [Video](https://www.youtube.com/watch?v=wWisOHEcMPE)

## Engineering Agency out of the Happy Path

Clarivate’s drug-document indexing system reached 99.95% accuracy on a defined set of business-critical tasks. Deciding which cases belonged on that path also gave experts a way to see customer needs they had been encountering one document at a time.

### What an agent notification costs an expert

[0:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=14s)

When sales needs a feature to close an account, or an urgent customer ticket comes in, the same domain expert often gets called. The expensive part starts before the decision. Someone who already owns a team or a critical process has to reconstruct a complicated situation, only to discover that the answer looks routine in retrospect. As execution gets cheaper, a growth-stage company can do more things. Its busiest people still have to decide which ones are worth doing. That is the constraint that interests [Matthew Jewkes](https://linkedin.com/in/mjewkes).

In Jewkes’s definition of full automation, a routine input goes all the way to a finished output without human approval. People remain available for exceptions. An agent that sends a notification to an expert’s phone may have done considerable work, yet the expert still has to interrupt theirs to decide what happens next. Changing agent skills and harnesses also give engineers more to maintain, and unusual failures can be hard for the rest of the business to handle. A familiar automation, like one in marketing or DevOps, lets other teams rely on its output without following every internal decision.

Consider a vendor who signs up and starts receiving a whole sequence of marketing emails. Every subsequent step can work as designed once the wrong person enters the sequence. AI can provide an escape hatch at intake by checking whether the signup belongs in the customer workflow. The same approach lets a system reject an adversarial input or escalate something unusually consequential. Giving the model discretion over admission reduces the variety of cases the routine workflow has to handle.

### A research agent takes on drug-document indexing

[4:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=258s)

Clarivate maintained an annotated history of drug development spanning tens of millions of documents, with hundreds of thousands of edits a year. Its ontology gave indexers tens of thousands of ways to classify material. Inserting or editing a document could trigger work on both its index entries and its relationships to other documents. Those changes generated tens of thousands of new tasks a month. At that volume, the economics turned on what happened after the initial engineering investment. The finished automation was expensive to build, but Jewkes found it stable and cheap to run.

The first implementation resembled a research and assignment agent. After a few sprints it reached roughly 80–83% on the indexing work. For a team accustomed to spending months fine-tuning a **BERT** model for a single component of a language-processing pipeline, that was an exciting start. To understand the remaining failures, the engineers assembled 1,000 evaluation examples and sat down with domain experts to examine actual assignments.

On earlier projects, asking a busy expert to fill a blank spreadsheet with 100 evals had produced little momentum. Here an indexer could look at a decision the system had already made, recognize what it could do, and spend the meeting on a case that looked wrong. The engineers could then ask a question the indexer was particularly well placed to answer. What should have happened to this document?

### A wrong answer, an unsettled definition, and new science

[6:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=399s)

The review turned up a large group of contentious cases, a small group of errors everyone agreed could threaten important accounts, and perhaps 3–5% of cases that needed escalation. A single performance score had put three different problems in the same pile.

- **A known answer that the model missed.** In Jewkes’s experience, even strong models could fail around 2–3% of the time on multistep workflows. Rerunning the same case could make the failure disappear. When experts agreed on the required output and a mistake put an account at risk, the engineers had a clear target for reliability work.

- **A relationship the organization had yet to define.** In a simplified gout example, a drug binds to tubulin and interrupts a chain of events, reducing a downstream inflammatory molecule. Should the index list that downstream molecule as a drug target? A direct-binding definition yields one answer. A definition that includes indirect effects allows another. Models went back and forth, and two experts could confidently disagree. A reliable label required a decision about what the index meant by *target*.

- **A decidable case that required new scientific judgment.** What made the heart-disease drug candidate Verve 102 an escalation? The desired output was clear. The team wanted to index six mechanisms. Jewkes singled out four of the six as new to science. An expert had to work out how the ontology should represent those mechanisms before the system could treat them as familiar assignments.

![The largest category contains cases the organization has yet to settle. Two smaller categories cover critical errors, where some labels need nearly perfect recall, and unusual cases that need human attention.](https://ai.engineer/api/talk-essay-media/161831c49d06dbf5b119afb818047d392576cdd3cd8abee3eaf1641478d010e2)

The experts separate cases that need an organizational decision from critical errors and unusual exceptions.

### Where an indexing task can leave the pipeline

[8:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=532s)

Which tasks warranted the most heavily tested pipeline, and where could that pipeline send a case it still couldn’t settle? By Jewkes’s estimate, the team spent at least as much time building case routing as it spent improving accuracy on the final critical set. Routing kept unsettled definitions and unfamiliar inputs out of routine processing. It also concentrated testing on decisions where a wrong answer would be costly.

Most tasks could ship as usual. If a case looked unusual or the model was confused, the system could stop, have the AI describe the difficulty, and file the case under a category. Undecided cases also left the normal path. A plausibly business-critical task went through a pipeline tested against thousands of evals. That pipeline could usually ship an answer itself. An expert received the exception if uncertainty persisted. This second chance to escalate let the team handle high-stakes work without asking a person to approve every answer.

Two opportunities to catch an exception
An unfamiliar input can leave at intake. A critical task that fits the workflow can still reach a human later if the tested pipeline cannot resolve it.
- Route the indexing task — Check whether the case is familiar, settled, and plausibly business-critical.
- Categorized exception — Describe what is unusual and take the case out of routine processing.
- Undecided cases — Set aside work for which the organization lacks an agreed answer.
- Tested critical pipeline — Apply the workflow evaluated on thousands of examples.
- Ship the answer — Routine work and resolved critical work can both finish automatically.
- Expert review — Resolve the outstanding critical exception.
- Route the indexing task → Ship the answer (Routine)
- Route the indexing task → Categorized exception (Confusing or unfamiliar)
- Route the indexing task → Undecided cases (Unsettled definition)
- Route the indexing task → Tested critical pipeline (Plausibly critical)
- Tested critical pipeline → Ship the answer (Resolved)
- Tested critical pipeline → Expert review (Still uncertain)

### What the 99.95% measured

[9:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=598s)

The initial agent’s 80–83% described the broader indexing task. Once the team removed ambiguous cases and unusual escalations, standard approaches reached about 95% accuracy on the small set of business-critical tasks. The engineers then examined failures across many evals and refined the workflow for those structured inputs. On that critical set, they reached 99.95%. Going from about 5% errors to 0.05% meant reducing the remaining error rate by roughly a factor of 100.

The team waited to deploy until a random sample showed better performance than the historical indexing work. Consider the conditions under which experts had produced that work. A PhD-trained indexer might process hundreds of cases a week, with fatigue, pressure from other work, and the occasional misclick. Jewkes would still expect an expert given one case and time to examine every detail to outperform the system. The historical comparison answered the production question because that was the workload the automation would inherit.

![The slide reports full automation of decidable cases and greater than 99.95% accuracy on business-critical tasks. It says a PhD expert could outperform the system on an individual case and attributes the system’s advantage at scale to fatigue, misclicks and random human error.](https://ai.engineer/api/talk-essay-media/cc247042a7e0204e4ef76d0966737cd61cbf3ae77cb83f4a16ce94f5e97f003c)

Jewkes reports more than 99.95% accuracy on business-critical tasks. The slide distinguishes that metric from performance on a single case.

### What an expert can see in a pile of exceptions

[11:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=684s)

Routing changed the expert’s workday twice. Routine decisions disappeared from it, and the remaining work arrived grouped by the kind of judgment it required. Experts could quickly review and ship the few unresolved critical cases. A pile of disputed assignments raised a different question. Was settling the definition worth anyone’s time?

A large backlog might motivate the company to write a specification. It might also show why nobody had written one. Some disputed assignments were only marginally valuable. Seeing them together gave the expert grounds to propose freezing that whole class of work until someone could make a business case for it. The company could make that decision once and spare its indexers the cost of reopening the question document by document.

The novel cases had a different value in aggregate. An indexer might encounter something like Verve 102 once a week or every other week, scattered among drugs with familiar mechanisms. One such document tells the expert little about how often the problem occurs, what it would cost to support, or who might pay. With two or three hours to consider similar exceptions together, the expert could look for a recurring need and propose a specific investment.

That preparation paid off on an eight-figure account. By the time the customer raised a problem, an expert who had studied the grouped cases already had an opinion about where to invest and what to build. A change in market demand can show up first as a series of awkward indexing jobs, before it arrives as a feature request.

![An expert icon sits above three groups labeled Undecided, Exceptions and Critical Review. The bullets explain that automation saves the expert time and directs attention to ambiguous cases, unusual cases and critical cases the system was unsure about.](https://ai.engineer/api/talk-essay-media/39a530a76d9bfc3e0b3fa171c4dcd2a3e762d8b8182ea6a83cef0cf94c87f83b)

The queue groups exceptions by the type of attention they need from the expert.

### Which workflows justify the investment?

[14:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=891s)

Reading a spreadsheet of evals for hours is expensive when the person reading it is an AI engineer. Moving a non-engineering process into software also changes who owns it and whose work the business has to trust. For a project of that size, Jewkes would involve sales and marketing early enough to work out whether the investment could improve customer satisfaction, earn revenue, or open a neighboring market.

A promising candidate is already running at scale and earning a return. Jewkes’s example for a CFO is a category of tasks associated with seven figures of revenue and five customer tickets asking for a change. The revenue establishes what depends on the process. The tickets identify a problem customers actually feel. For a startup still discovering what customers want, he would preserve the flexibility of having people do the work.

Even an imperfect written specification is encouraging. Somebody close to the process thought its decisions could be written down and invested effort in trying. Detail-heavy work is particularly attractive when the burden comes from remembering and applying many rules reliably at volume. Work that depends on subjective judgment calls for more discretion. If that judgment is what makes the business distinctive, Jewkes would look elsewhere for a full automation.

### The time it takes to read 1,000 evals

[17:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1037s)

What did Jewkes put at the top and bottom of his list of techniques for going from 95% to 99.95%? At the top, read and write evals. At the bottom, build 1,000 of them and expect to spend time reading them. He sees clean inputs and a specified process as promising conditions for that level of accuracy. The cases that still defeat a model under those conditions interest him because they can suggest what to build next.

Everyone on the team had to spend time with individual failures. Reading a model’s agreeable prose and finding the point where it departed from the expected decision was exhausting. Yet that was how the team could tell whether an assignment needed an engineering fix, a definition from the business, or a scientist’s attention. A score alone couldn’t tell them whom to ask.

Jewkes closes with a paraphrase of advice he attributes to Rockefeller. Once you can specify your job well enough to hire someone to do it, sit down and think about the next opportunity. At Clarivate, the grouped exceptions gave the expert both that time and something concrete to think about. Jewkes bets the market will sort out competing orchestration tools and harnesses. He expects the greater advantage to come from how companies direct their collective attention, for as long as human judgment commands a premium.

## Key takeaways

- Bring domain experts completed examples. An actual indexing decision gives them a way to distinguish a model mistake from a rule the organization has yet to settle.
- Budget for routing as well as accuracy. Jewkes estimates that the team spent at least as much time deciding where cases belonged as it spent reducing errors on the critical set.
- Evaluate the workload the automation will inherit. The 99.95% result applied to defined critical tasks, and deployment depended on a comparison with historical indexing work.
- Group exceptions by the decision they require. Experts can clear unresolved critical cases, freeze marginal work, and examine recurring novel cases for evidence of customer demand.

## Slides

### Focus on domain experts

When an urgent customer ticket comes in, who does your team tag first? That expert may lead a small team or run a critical business process. Ask how they spend their week.

[0:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=33s) [Four bullets ask viewers to identify their strongest domain expert, consider who gets contacted first for priority work, identify a team or process lead, and ask how that person spends the week.](https://assets.ai.engineer/sites/aiecode2025/sha256/501e6d39457b8f81d2deec219542838d6386893fe45a939a812524c51ea16c47.webp)

Focus on domain experts

• Strongest domain expert
• First @ for priority GTDs
• Team or Process lead
• How do they spend their week?

STANDARD
CYBERNETICS

AI Engineer
World’s Fair

### Initial approach: do everything

The research and assignment agent reached 83% performance after a few sprints. The team used to spend months fine-tuning BERT for one pipeline step. It compiled about 1,000 evals, then hit a wall when it brought examples to domain experts.

[5:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=316s) [The research and assignment agent reached 83% performance after a few sprints. The team compiled about 1,000 evaluations and encountered difficulties when it met with domain experts.](https://assets.ai.engineer/sites/aiecode2025/sha256/f47f19e827a1aa6a73233355292453499d7c33efbc01ca84be9fae257e22b910.webp)

Initial approach: do everything

• Research & assignment agent
• 83% performance after a few sprints
  • Shockingly effective compared to pre - '23 BERT fine tuning
• Our team compiled ~1000 evals

• Met with domain expert team, quickly hit a wall

STANDARD
CYBERNETICS

### Experts say: a heterogeneous mix of problems

The largest group contained cases where experts could disagree or the organization had yet to decide. Two smaller groups posed different problems. Certain labels needed near-perfect recall, and exceptions needed a person to resolve them.

[6:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=400s) [The largest category contains cases the organization has yet to settle. Two smaller categories cover critical errors, where some labels need nearly perfect recall, and unusual cases that need human attention.](https://assets.ai.engineer/sites/aiecode2025/sha256/161831c49d06dbf5b119afb818047d392576cdd3cd8abee3eaf1641478d010e2.webp)

Experts say: a heterogeneous mix of problems

• Large bucket of organizationally undecided/contentious cases
• Small bucket of critical errors
  • In our case, certain labels needed to be applied with ~perfect recall.
• Small bucket of unusual "exceptions" or "escalations"
  • Cases that had an ideal resolution, but needed human attention.

STANDARD
CYBERNETICS

### Undecided: the models don't agree

Does lowering IL-1β indirectly count as targeting it? In this gout example, GPT 5.5 answers N and Opus 4.8 answers Y to the same question about colchicine. Jewkes says both models can give either answer.

[7:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=479s) [Two screenshots ask whether colchicine targets IL1 for gout. GPT 5.5 - High answers N on the left. Opus 4.8 - High answers Y on the right.](https://assets.ai.engineer/sites/aiecode2025/sha256/edbafda8934b7af6834abade49ab64bfac3a1e5e5125a544ab42def777501701.webp)

Undecided: the models don't agree

Does colchicine target IL1 for Gout? Answer only with Y or N.
N
GPT 5.5 - High

Does colchicine target IL1 for Gout? Answer only with Y or N.
Y
Opus 4.8 - High

Production cases more complex, but often of this flavor

STANDARD CYBERNETICS

Interface text:
Thought for 4s

### Overall outcome

Jewkes reports >99.95% accuracy on business-critical tasks. He says a PhD could do better on a single case, while fatigue and misclicks give the system an edge when experts handle hundreds of cases a week.

[10:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=645s) [The slide reports full automation of decidable cases and greater than 99.95% accuracy on business-critical tasks. It says a PhD expert could outperform the system on an individual case and attributes the system’s advantage at scale to fatigue, misclicks and random human error.](https://assets.ai.engineer/sites/aiecode2025/sha256/cc247042a7e0204e4ef76d0966737cd61cbf3ae77cb83f4a16ce94f5e97f003c.webp)

Overall outcome
Defined by the shape of the customer need & business

• Decidable cases fully automated.
• Routing + Hill-climbing very rarely misses a critical label
  • >99.95% accuracy on business critical tasks

• Better than PhD human's performance at scale
  • Plausibly a PhD can out perform the system on a singular case
  • But not at scale (fatigue, misclicks, random error)

STANDARD
CYBERNETICS

### Time efficiency + attention routing

Automation takes some routine work off the expert’s plate. It sorts the remaining cases by why they need attention, so the expert can review ambiguous cases, unusual exceptions, and critical cases the system was unsure about.

[11:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=691s) [An expert icon sits above three groups labeled Undecided, Exceptions and Critical Review. The bullets explain that automation saves the expert time and directs attention to ambiguous cases, unusual cases and critical cases the system was unsure about.](https://assets.ai.engineer/sites/aiecode2025/sha256/39a530a76d9bfc3e0b3fa171c4dcd2a3e762d8b8182ea6a83cef0cf94c87f83b.webp)

Time efficiency + attention routing

• Back to the expert from the beginning
• Automation gives them back some time and attention
• But it also organizes their attention in a structured way on the remainder.
• They get organized buckets of:
  • Undecided / Ambiguous cases
  • Exceptions / Unusual cases
  • Critical cases the automation wasn't sure of

Undecided   Exceptions   Critical Review

STANDARD
CYBERNETICS

### Where to go building

When choosing work to fully automate, look for tasks already tied to significant revenue. Seven figures of proven revenue makes the case concrete for the CFO. Can you point to a customer ticket about those same tasks, or even ten?

[15:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=939s) [Under “Valuable,” the slide recommends choosing work at scale with a material return on investment. It cites seven figures of proven revenue, a rationale the CFO can understand, and customer tickets tied to the tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a3fd5d39e36895afa51358b54590cbe90b102b86dff8c4b12e9e794a86de9d9.webp)

Where to go building
Valuable

• You want to pick a domain that is scaled, and has material RoI.
  • Think a task associated with 7 figures of proven revenue.
  • Something your CFO can understand
• Is there a customer ticket, or 10, associated with this bundle of tasks?

STANDARD
CYBERNETICS

## Full transcript

[0:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1s) [music]

[0:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=14s) Uh afternoon everybody. Uh I'm going to

[0:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=16s) talk today about engineering agency out

[0:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=19s) of the happy path. Um this is going to

[0:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=21s) be focused on AI leadership, engineering

[0:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=25s) leadership at growth stage companies.

[0:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=28s) Um, and I'd like to start by asking

[0:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=30s) everybody to think about the strongest

[0:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=32s) domain experts on your team. So these

[0:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=35s) are people who uh, you know, if sales

[0:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=37s) wants to build a new feature or deliver

[0:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=39s) something new in order to close an

[0:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=40s) account or an urgent customer ticket

[0:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=42s) appears, uh, these are the people who

[0:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=44s) get at paint. Uh, they're the people

[0:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=46s) maybe who are responsible for a small

[0:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=48s) team or for a critical business process.

[0:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=51s) And I want to think about how these

[0:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=52s) people actually spend most of their work

[0:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=54s) weeks.

[0:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=56s) Uh my experience is that most experts

[0:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=58s) I've worked with have their attention

[1:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=60s) sort of smeared over multitasking,

[1:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=62s) loading up on complicated situations,

[1:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=65s) making decisions that could have been

[1:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=66s) routine or obvious in retrospect and

[1:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=69s) only a small fraction of their attention

[1:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=71s) is left over to sort of move the

[1:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=73s) business forward.

[1:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=76s) Um now like the cost of like raw white

[1:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=78s) collar output is dropping. It's

[1:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=81s) cratering. I've talked to a bunch of

[1:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=82s) people over the conference and

[1:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=84s) regardless of what you think about

[1:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=85s) frontier models and where the future's

[1:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=86s) going to go in terms of new capacity.

[1:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=88s) Everybody I've talked to has agreed that

[1:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=90s) existing capacity is going to get very

[1:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=92s) very cheap.

[1:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=94s) Cheap execution increases the load and

[1:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=96s) the on on judgment on deciding what to

[1:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=99s) do. Uh organizational judgment was

[1:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=102s) already in short supply. Uh and this is

[1:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=104s) going to make it even worse. I think

[1:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=105s) this is currently underpriced.

[1:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=108s) So with that in mind, I'm going to talk

[1:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=109s) today about using AI for full

[1:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=110s) automations. By full automations, I mean

[1:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=113s) taking humans out of the happy path

[1:55](https://www.youtube.com/watch?v=wWisOHEcMPE&t=115s) completely. Using humans maybe for

[1:57](https://www.youtube.com/watch?v=wWisOHEcMPE&t=117s) exceptions, but not not human in the

[1:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=118s) loop um style automations. I think that

[2:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=121s) these types of processes can improve

[2:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=123s) margins kind of obvious. They can reduce

[2:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=126s) complexity of a business by

[2:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=127s) encapsulating whole chunks of work. Uh

[2:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=129s) and then you can recover judgment from

[2:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=131s) your experts and hopefully route it

[2:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=132s) towards more important um tasks or

[2:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=135s) questions. I'm going to talk about that

[2:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=136s) specifically a bit later.

[2:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=138s) Um, now the key to full automations and

[2:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=141s) my experience has been engineering

[2:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=142s) agency out of the happy path out of

[2:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=145s) these automations. This is a bit of a

[2:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=147s) reversal. You know, agents have

[2:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=148s) permanently changed and expanded my idea

[2:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=150s) of what I think is possible with

[2:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=152s) software. But in enterprise, agents have

[2:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=154s) problems. They're a pain to maintain.

[2:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=157s) Uh, especially if skills and harnesses

[2:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=158s) are changing. Uh, they fail in unusually

[2:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=162s) shaped ways that enterprises find hard

[2:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=163s) to work with. And then they increase the

[2:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=166s) burden of decisions. When I see an

[2:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=168s) expert or a leader responding to agent

[2:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=170s) notifications, you know, maybe on their

[2:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=171s) phone or or on their laptop, maybe they

[2:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=173s) have an app for it. I'm often

[2:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=174s) suspicious. Is that actually the best

[2:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=176s) use of their, you know, valuable time

[2:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=178s) and attention?

[3:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=180s) Full automations, on the other hand,

[3:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=181s) they free up attention. They're legible.

[3:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=184s) So this means that, you know, they take

[3:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=185s) an input, they produce an output, they

[3:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=187s) encapsulate the whole thing, and they

[3:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=189s) hide the complexity. And they're

[3:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=191s) organizationally familiar. Like large

[3:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=193s) human organizations have been building

[3:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=194s) automations in software for decades.

[3:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=197s) Think marketing, think DevOps. And

[3:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=199s) they've been building uh automations in

[3:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=201s) industry for centuries. Full automations

[3:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=203s) are one of the key tools by which human

[3:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=206s) organizations can do bigger and bigger

[3:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=207s) things.

[3:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=214s) And that is giving the whole system an

[3:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=215s) escape hatch. So, one of the problems

[3:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=217s) with full automations is, and I'm sure

[3:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=219s) people have seen this, when you get uh

[3:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=221s) you sign up for something and maybe you

[3:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=223s) accidentally get a whole marketing drip

[3:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=224s) email when you were signing up as a

[3:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=226s) vendor. Um, one of the interesting

[3:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=228s) things you can do with AI and agency

[3:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=230s) with automations is you can pre-filter

[3:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=231s) inputs. You can check does this task

[3:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=234s) actually match the happy path of the

[3:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=236s) automation. You can reject adversarial

[3:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=238s) situations. If something looks like it's

[4:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=240s) business critical, sort of not a normal

[4:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=242s) course of business operation, you can

[4:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=244s) escalate that to a human. I think this

[4:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=246s) is kind of neat. This is the use of AI,

[4:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=248s) AI agency to reduce the variance of an

[4:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=251s) automation to make things more smooth,

[4:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=253s) which thereby lets us expand the scope

[4:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=255s) of what we can automate. So I'm going to

[4:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=258s) give a case study today. My background

[4:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=259s) is working in bioarma data at scale. I

[4:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=262s) had a company called deal enablement for

[4:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=264s) bioharmaceutical mergers, licensing and

[4:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=266s) acquisitions. In 23 we sold to clarvate.

[4:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=269s) Clarvate is a really interesting

[4:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=270s) company. They have an annotated history

[4:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=272s) of the entire you know drug development

[4:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=274s) industry. hundreds of millions of

[4:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=275s) document or tens of millions of

[4:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=277s) documents, hundreds of thousands of

[4:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=278s) edits every year and a rich ontology for

[4:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=280s) working of those sort of tens of

[4:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=282s) thousands of ways of indexing it.

[4:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=284s) [snorts] Um the automation we built was

[4:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=286s) automating the indexing and the

[4:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=288s) relationships between those documents.

[4:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=290s) So this is processing tens of thousands

[4:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=292s) of new tasks a month when a document's

[4:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=294s) edited or inserted or something changes.

[4:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=296s) We were able to achieve three and a half

[4:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=298s) n of accuracy on the tasks that the

[5:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=300s) business most valued. And I'll talk

[5:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=301s) about what that means in a few minutes.

[5:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=303s) Um, this was expensive to build. This

[5:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=305s) was a capex cost. AI engineers aren't

[5:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=307s) cheap. But once it was built, it was

[5:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=309s) cheap to operate. It mostly stayed

[5:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=311s) within its encapsulation. Um, and it was

[5:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=315s) stable.

[5:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=317s) So building this out, our initial

[5:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=319s) approach, I think in 2024, we were

[5:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=322s) excited by 40, we were excited by 03. We

[5:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=325s) did something, maybe we were early for

[5:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=326s) agents, but we built something that's

[5:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=328s) sort of similar to what people would

[5:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=329s) today think of as a research and

[5:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=330s) assignment agent. After a few sprints,

[5:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=333s) we hit, you know, 80 83% performance. We

[5:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=335s) were really excited. We had been working

[5:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=336s) in NLP for a decade. We were used to

[5:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=338s) spending months fine-tuning a BERT model

[5:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=340s) to just do one sort of component of one

[5:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=342s) of these pipelines. We compiled a

[5:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=345s) thousand devalves. We went we sat down

[5:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=346s) with the domain experts.

[5:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=349s) Um, I will say one good thing that we

[5:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=351s) did is that we brought the domain

[5:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=352s) experts a whole bunch of examples. And

[5:55](https://www.youtube.com/watch?v=wWisOHEcMPE&t=355s) um I've worked on other projects where

[5:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=358s) you go to a domain expert, you give them

[5:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=359s) a spreadsheet and you say, "Hey, fill

[6:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=361s) out a 100 evals." That doesn't go well.

[6:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=364s) Uh bringing a blank spreadsheet to a

[6:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=366s) busy expert uh doesn't create momentum.

[6:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=369s) Uh by bringing them all these these

[6:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=371s) these evals, these places we'

[6:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=372s) experienced, we very quickly were able

[6:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=374s) to get a the domain experts on board

[6:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=376s) with what's possible and b get them

[6:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=378s) fixated and interested in the places in

[6:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=380s) which it failed. Um I do think that most

[6:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=383s) automations are going to require input

[6:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=384s) with domain experts working on these

[6:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=386s) cases. If you can help them get started

[6:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=389s) with evaluation,

[6:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=394s) but writing evals is an art and it can

[6:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=396s) help get them started more quickly. But

[6:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=399s) experts, we sat down with these eval.

[6:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=401s) What did we find? Well, there was a

[6:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=402s) large bucket of undecided or contentious

[6:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=406s) cases. These were tasks where actually

[6:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=409s) two experts might confidently disagree

[6:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=411s) about what the answer is or where the

[6:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=413s) organization itself hadn't quite

[6:55](https://www.youtube.com/watch?v=wWisOHEcMPE&t=415s) decided. and everybody was not sure.

[6:57](https://www.youtube.com/watch?v=wWisOHEcMPE&t=417s) Mixed in with that sort of

[6:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=418s) heterogeneously was a small bucket of

[7:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=420s) critical errors. These were errors where

[7:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=422s) everybody agreed a misstep would, you

[7:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=424s) know, threaten material accounts. We we

[7:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=426s) just couldn't have them. And then mixed

[7:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=428s) in with that was a whole sort of maybe 3

[7:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=430s) or 5% of exceptions or escalations,

[7:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=433s) places that needed more attention.

[7:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=436s) Uh do a quick example of these three. So

[7:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=439s) the errors pretty straightforward. Even

[7:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=441s) top models, if you have multi-step

[7:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=442s) workflows fail 2 3% of the time. It's

[7:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=445s) annoying when you see it at scale.

[7:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=446s) Sometimes you can't even reproduce it,

[7:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=448s) but it happens. For undecided or

[7:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=450s) ambiguous cases, quick example, um, in

[7:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=453s) drug development, uh, drug developers

[7:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=455s) care a lot about targets. These are the,

[7:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=457s) you know, key pathway or molecule that

[7:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=458s) your drug actually impacts. Gout is a

[7:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=461s) disease that's caused by overprouction

[7:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=463s) of interlucan one beta. Culticine is a

[7:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=465s) drug that reduces interlucan one beta by

[7:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=468s) preventing a chain reaction basically.

[7:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=470s) uh so colon itself binds to a protein

[7:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=474s) called tubulin. Our system wanted to

[7:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=476s) know does cultureine target interlucan

[7:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=478s) one beta. And if you ask GPT55 sometimes

[8:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=481s) it'll say no and if you ask you know

[8:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=483s) opus 48 sometimes it'll say yes. They

[8:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=485s) actually sort of both go back and forth

[8:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=487s) on this. This is a toy case. It's

[8:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=489s) actually the real situation is a little

[8:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=490s) more complicated but it's

[8:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=491s) representative. It gives a sort of

[8:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=492s) flavor of um what undecided cases look

[8:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=495s) like. Uh in terms of unusual or

[8:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=497s) exception, I've got another bio example

[8:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=499s) that I really like. Uh, this is a really

[8:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=501s) exciting new potential drug called Verve

[8:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=502s) 102, possibly cure heart disease, which

[8:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=505s) is like it's pretty cool. I hope to get

[8:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=507s) this or something like it for my little

[8:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=508s) boy when he turns 18. Um, this does have

[8:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=512s) a decidable output. We do want to index

[8:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=514s) the sort of six mechanisms of this drug,

[8:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=517s) but the four of them that I've circled

[8:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=518s) here are all actually just new to

[8:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=519s) science. So, like these didn't exist a

[8:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=521s) few years ago. So, in order to properly

[8:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=524s) solve this, we're going to require some

[8:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=525s) human attention. The humans are going to

[8:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=526s) have to think about how to index

[8:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=528s) ontologies. This is an escalation

[8:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=532s) path forward. How are we going to handle

[8:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=533s) these three buckets? We need to make

[8:55](https://www.youtube.com/watch?v=wWisOHEcMPE&t=535s) sure that errors both get routed into a

[8:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=538s) place where we can grind out high

[8:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=539s) performance and then we need to grind

[9:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=541s) out the performance for exceptions and

[9:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=543s) for undecided cases. We want to route

[9:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=544s) those into places to keep them out of

[9:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=546s) the normal system because they're going

[9:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=548s) to require different types of handling.

[9:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=551s) And I think probably in retrospect we

[9:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=553s) spent at least as much time building

[9:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=555s) case routing as we actually did on the

[9:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=557s) final error climbing for the sort of

[9:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=559s) critical set. Is the AI ever confused

[9:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=561s) about what's going on? Is it unusual a

[9:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=563s) case? Abort. Use the AI to describe

[9:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=565s) what's going on. Categorize it away.

[9:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=567s) Skip it from the pipeline. Is it

[9:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=569s) plausibly business critical? Okay, then

[9:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=571s) send that to the extremely robustly

[9:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=573s) tested, you know, thousands of evals

[9:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=575s) pipeline. Hopefully you can ship an

[9:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=577s) answer. If through that pipeline it's

[9:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=579s) still not sure, fine. and ship it back

[9:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=580s) to a human as an exception. But vast

[9:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=582s) majority of the time you can ship an

[9:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=583s) answer

[9:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=586s) is it undecided pipeline that out as

[9:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=587s) well. Um for the majority of cases

[9:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=589s) though they didn't fall into any of

[9:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=591s) these cases and we could just ship as

[9:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=592s) usual.

[9:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=598s) um in terms of the critical path. So you

[10:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=600s) know we had this this small set of tasks

[10:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=602s) that were absolutely business critical

[10:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=604s) once we took out the ambiguous cases and

[10:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=606s) once we took out the sort of um unusual

[10:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=608s) escalation cases we're able to get to

[10:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=610s) about 95% accuracy with just sort of

[10:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=612s) standard approaches going from 95 to 3

[10:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=615s) and a half 9s. This is a case of I've

[10:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=618s) got a whole bag of tricks on it. It's a

[10:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=619s) case of hill climbing. It's a case of

[10:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=620s) looking at a lot of evals using all of

[10:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=622s) the normal tricks to juice performance

[10:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=624s) out of LLMs. I basically do believe

[10:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=626s) though that if your inputs are well

[10:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=628s) structured and you're looking at a

[10:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=629s) specific set of cases that you've

[10:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=630s) articulated from your business, getting

[10:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=632s) to three and a half nines is possible

[10:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=634s) for most automations.

[10:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=636s) Um, [snorts] we deployed this only once

[10:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=638s) a random sample showed that this

[10:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=640s) performance was better than the historic

[10:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=641s) work. This is cool. So, um, I do think

[10:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=645s) specifically um, I'm going to claim that

[10:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=647s) this system performed better than PhD

[10:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=649s) humans at scale. And I'll make that

[10:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=651s) qualifier because I know that a PhD

[10:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=653s) human if they're sitting down on a

[10:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=654s) singular case and they're looking at all

[10:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=656s) the details, they will outperform this

[10:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=658s) system, but not at scale. Not when they

[11:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=660s) have to look at hundreds of cases a

[11:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=661s) week, not when they have fatigue, not

[11:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=663s) when if you know one in a thousand times

[11:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=664s) they misclick and not uh you know given

[11:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=667s) time pressure of their other work.

[11:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=671s) Um so full automation is the happy path.

[11:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=673s) Saves you time. That's great. You get

[11:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=675s) better critical business critical task

[11:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=677s) outcomes. That's also great. You know,

[11:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=678s) this is a if you can take the places

[11:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=680s) that are most valuable to the business

[11:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=682s) and grind them out until you're

[11:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=683s) performing better than humans at scale,

[11:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=684s) that's great. There's a third benefit

[11:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=686s) here, which is that it organizes the

[11:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=687s) exhaust from the automation in a way

[11:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=689s) that your experts can pay attention to.

[11:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=690s) And I think this is pretty neat. So,

[11:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=693s) back to that expert from the beginning.

[11:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=695s) Hopefully, we've cleared their plate

[11:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=696s) somewhat. We've taken some of the

[11:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=697s) routine work out of their job. In

[11:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=700s) addition, because we've invested so

[11:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=701s) heavily in routing, we've now organized

[11:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=703s) that we can organize their attention in

[11:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=704s) a structured way among the around the

[11:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=707s) remaining cases. So they're going to get

[11:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=709s) a pile organized undecided and ambiguous

[11:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=712s) cases, exceptions and unusual cases, and

[11:55](https://www.youtube.com/watch?v=wWisOHEcMPE&t=715s) then a small number of critical cases

[11:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=716s) the automation wasn't sure about. For

[11:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=719s) those last ones, the sort of ones that

[12:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=720s) are critical, they can quickly review

[12:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=722s) them, very very few in number, and ship

[12:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=724s) them. [snorts] For the ones that are

[12:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=726s) organizationally ambiguous, this is

[12:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=728s) actually kind of interesting. You know,

[12:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=730s) the first thought is maybe this is a way

[12:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=732s) to incentivize the organization to

[12:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=733s) specify. You know, you've got 20% maybe

[12:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=736s) of the total tasks that the organization

[12:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=738s) wasn't sure with

[12:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=742s) these are often one of the reasons that

[12:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=744s) they're not yet fully specified is that

[12:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=746s) they're only marginally valuable. So the

[12:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=749s) alternative here is that you can

[12:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=750s) actually if the expert has a big pile of

[12:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=753s) them in front of them they can actually

[12:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=754s) advocate that hey let's permanently

[12:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=756s) freeze this work let's prioritize it

[12:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=758s) let's not think about it again until

[12:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=760s) there's a business case too and let's

[12:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=761s) get it off our plate that is a form of

[12:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=764s) again reducing complexity in an

[12:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=765s) organization having fewer things to

[12:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=767s) think about so the focus and attention

[12:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=769s) can go in the right place

[12:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=771s) [snorts] I think this is even more

[12:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=772s) interesting so groundbreaking work like

[12:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=774s) that verve 102 drug for for heart

[12:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=776s) disease it's only like 3% % of new drug

[12:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=779s) development work. The vast majority of

[13:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=780s) drug development work sort of pattern

[13:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=781s) matches. Everybody's familiar with the

[13:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=783s) rough shape of what you're looking at.

[13:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=784s) So an expert might see this once a week,

[13:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=786s) once every other week, one of these

[13:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=788s) cases, they're scattered throughout. Uh

[13:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=790s) the the expert might think, hey, I could

[13:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=792s) build something to handle these cases.

[13:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=794s) But if you're only seeing it

[13:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=795s) sporadically, it's hard to build an

[13:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=797s) intuition for how important it is, how

[13:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=799s) expensive it would be, who would pay for

[13:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=800s) it. You start to bucket those. And now

[13:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=803s) the expert can think maybe when they sit

[13:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=804s) down with two or three hours and look at

[13:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=806s) them in aggregate, hey there's patterns

[13:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=808s) here or hey maybe this is the specific

[13:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=810s) investment I propose to start solving

[13:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=812s) these for the organization and then if

[13:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=815s) an an account you know a real customer

[13:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=817s) comes and says hey I have a problem this

[13:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=819s) expert now is ready to figure out how to

[13:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=821s) how to deal with that how to move the

[13:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=822s) business forward on it. So we saw this

[13:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=824s) happen organically on a 8 figure account

[13:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=827s) um and the expert had basically by

[13:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=830s) having the data structured had already

[13:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=832s) pre-formed an opinion about where to

[13:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=833s) invest and where to build forward. I

[13:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=836s) think this is actually the case for most

[13:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=838s) scaled business processes. If you're

[14:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=841s) working with the with with the market

[14:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=843s) with data in aggregate you're collecting

[14:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=845s) leading indicators of market demand,

[14:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=847s) right? you know, the places that are

[14:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=848s) slightly unusual or the places that um

[14:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=851s) that the market's moving now that they

[14:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=852s) weren't a few years ago, but they look

[14:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=854s) like exceptions and if they're

[14:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=855s) encountered infrequently, it's not

[14:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=856s) enough signal to have somebody think

[14:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=858s) about what to do with them. They're

[14:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=859s) difficult to prioritize if they occur

[14:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=861s) randomly. If you can organize this

[14:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=863s) signal, this exhaust these exceptional

[14:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=865s) cases that don't land in the happy path,

[14:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=867s) your best talent can actually start to

[14:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=868s) think about what to do with it. So,

[14:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=871s) quick, we built a full automation. We

[14:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=874s) can save hours. We can improve task

[14:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=876s) level outcomes. We can reduce

[14:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=878s) organizational complexity by taking a

[14:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=880s) big bucket of tasks, encapsulating them,

[14:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=882s) and having nobody have to think about

[14:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=883s) them anymore. And then we can route

[14:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=885s) talent towards higher value problems.

[14:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=887s) This is a form of organizational

[14:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=889s) refactoring.

[14:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=891s) I will say that full automations,

[14:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=894s) especially if you're trying to very high

[14:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=896s) quality output, something like doing a

[14:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=898s) PhD level expert's job, are expensive.

[15:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=901s) AI engineers are expensive. Having an AI

[15:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=903s) engineer spend hours reading a

[15:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=904s) spreadsheet of eval is expensive. Once

[15:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=907s) you get to that 95% going from 95 to

[15:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=910s) 99.95, that's an expensive process,

[15:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=912s) etc., etc., etc. Beyond that, you

[15:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=914s) actually have to think if you're going

[15:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=915s) to be refactoring a um a non-engineering

[15:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=919s) process, you're going to have

[15:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=920s) organizational change, trust building,

[15:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=922s) ownership change. If you're going to

[15:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=924s) make an investment of this size,

[15:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=926s) hopefully you're also working with your

[15:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=927s) sales and marketing team to figure out

[15:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=928s) how to juice that for revenue, how to

[15:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=930s) increase customer satisfaction, how to

[15:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=931s) expand into a neighboring market. So,

[15:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=933s) this is a major project. Um, thinking

[15:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=936s) about where to do them takes a little

[15:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=938s) bit of of um of judgment. You want to

[15:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=940s) pick a domain that is scaled and that

[15:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=942s) already has material ROI. I would

[15:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=944s) actually not propose building full

[15:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=946s) automations if you're a startup, if

[15:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=948s) you're in discovery, if you're looking

[15:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=949s) at new and new businesses. Keep the

[15:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=950s) flexibility of humans. But if you have a

[15:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=953s) bucket of tasks that's associated with

[15:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=954s) seven figures of revenue and that maybe

[15:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=956s) there's a customer ticket, hopefully

[15:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=958s) there's 10 customer tickets, the

[15:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=959s) organization knows that there's pain

[16:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=961s) here, that can justify an investment and

[16:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=963s) that is usually something that a CFO can

[16:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=965s) understand. You know, if we say, hey,

[16:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=967s) this is associated with seven figures of

[16:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=969s) revenue and there's five tickets.

[16:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=970s) Customers don't like it or they want

[16:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=972s) something else here. That's a pretty

[16:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=973s) good leading signal.

[16:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=976s) Sort of like in terms of the tasks

[16:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=978s) themselves, look for business processes

[16:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=980s) that are running. ideal if it has

[16:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=982s) written specifications. I wouldn't worry

[16:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=983s) too much about if the written

[16:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=984s) specifications are any good.

[16:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=986s) Specifications are really hard and I

[16:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=988s) think that even if there has been

[16:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=990s) momentum to build specifications that

[16:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=993s) reflects that on the inside we the team

[16:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=996s) believes that this is specifiable the

[16:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=998s) team believes that this is not just raw

[16:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1000s) judgment complicated detail orient

[16:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1005s) work now which is actually

[16:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1008s) shockingly shockingly new and very very

[16:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1010s) interesting. It's ideal if this is

[16:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1012s) detail- oriented work because you can

[16:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1014s) take this sort of expensive thought,

[16:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1016s) memory, um, wrote work off of of humans

[17:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1020s) who tend to fail at it at scale.

[17:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1022s) Subjective judgment work is not. So

[17:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1024s) that's where you have to draw the line.

[17:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1026s) Uh, AI for like if your business is

[17:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1028s) built around a process, AI to automate

[17:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1030s) that specific differentiated process is

[17:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1032s) still in my opinion um, it's not where

[17:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1035s) the action is.

[17:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1037s) So a bag of tricks here. I'm not going

[17:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1038s) to get into this, but if you are working

[17:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1040s) on something where you're going from 95

[17:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1041s) to 99.95%,

[17:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1043s) here's a bag of tricks. This talk will

[17:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1044s) be online. You can take a photo of this,

[17:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1046s) contact me. I'd love to chat. I'm very

[17:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1048s) interested in the ways that models fail.

[17:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1050s) I feel like knowing where models fail is

[17:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1051s) really interesting for what to build

[17:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1053s) next. Um, but I'd be I'd love to chat

[17:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1055s) about any places where you have clean

[17:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1057s) input, uh, a sort of specified process,

[17:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1059s) and you can't get to three and a half

[17:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1061s) nights of accuracy. I think the two that

[17:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1063s) I'll flag out of this is the top one

[17:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1064s) which is read and write evals and the

[17:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1066s) bottom one which is build a thousand

[17:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1067s) evals and expect to spend time reading

[17:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1069s) them. Um that was really one of the big

[17:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1072s) learnings from my team is is everybody

[17:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1074s) had to invest in reading exactly why

[17:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1076s) this time the AI did not do the thing we

[17:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1079s) expected on the emails and it's

[18:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1080s) exhausting work because you're reading

[18:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1082s) psychopantic text and you're trying to

[18:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1083s) figure out what specifically went wrong

[18:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1086s) or against my expectations.

[18:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1088s) I'm close with this quote. This is one

[18:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1090s) of my favorite quotes is Rockefeller uh

[18:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1092s) paraphrase as soon as you can hire

[18:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1094s) someone to do your job. Sit down, kick

[18:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1096s) up your feet, think about how to make

[18:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1098s) the company more money. What is he

[18:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1100s) saying here? He's saying if your job is

[18:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1102s) specifiable, get someone else to do it.

[18:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1105s) What we want you to be doing is

[18:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1106s) thinking, applying your judgment,

[18:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1108s) looking for new opportunity. I think

[18:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1110s) that's perennial advice. It's great

[18:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1111s) advice. My hope is that AI might be able

[18:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1114s) to help us do this even better. If AI

[18:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1116s) can help route our attention towards the

[18:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1119s) most valuable places, the most valuable

[18:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1120s) opportunities, then maybe we can sit

[18:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1122s) down and actually think about the most

[18:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1124s) valuable things first. Um, my bet is

[18:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1128s) that orchestration layer and and um the

[18:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1131s) right harnesses, the market's going to

[18:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1133s) sort that out and there's not going to

[18:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1134s) be a huge amount of differentiated

[18:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1136s) advantage in terms of picking the right

[18:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1138s) AI orchestration software in 2026.

[19:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1141s) My bet is that the companies that

[19:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1143s) succeed the most with AI are going to be

[19:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1146s) the ones who figure out how to best

[19:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1147s) manage their collective attention and

[19:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1149s) judgment because the premium on that is

[19:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1150s) going to only go up at least for so long

[19:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1153s) as human judgment holds a premium. Um

[19:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1156s) that's it for my talk. It's exciting

[19:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1158s) times everyone. Uh I hope everybody

[19:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1160s) enjoys the conference and uh really

[19:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1162s) excited about all the stuff that we're

[19:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1163s) going to be building.

[19:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1177s) [music]
