# Engineering Agency out of the Happy Path — Matthew Jewkes, Standard Cybernetics

Matthew Jewkes, Standard Cybernetics

[Talk page](https://ai.engineer/talks/wWisOHEcMPE-engineering-agency-out-happy-path) · [Video](https://www.youtube.com/watch?v=wWisOHEcMPE)

## Engineering Agency out of the Happy Path

Clarivate’s drug-document indexing system reached 99.95% accuracy on a defined set of business-critical tasks. Deciding which cases belonged on that path also gave experts a way to see customer needs they had been encountering one document at a time.

### What an agent notification costs an expert

[0:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=14s)

When sales needs a feature to close an account, or an urgent customer ticket comes in, the same domain expert often gets called. The expensive part starts before the decision. Someone who already owns a team or a critical process has to reconstruct a complicated situation, only to discover that the answer looks routine in retrospect. As execution gets cheaper, a growth-stage company can do more things. Its busiest people still have to decide which ones are worth doing. That is the constraint that interests [Matthew Jewkes](https://linkedin.com/in/mjewkes).

In Jewkes’s definition of full automation, a routine input goes all the way to a finished output without human approval. People remain available for exceptions. An agent that sends a notification to an expert’s phone may have done considerable work, yet the expert still has to interrupt theirs to decide what happens next. Changing agent skills and harnesses also give engineers more to maintain, and unusual failures can be hard for the rest of the business to handle. A familiar automation, like one in marketing or DevOps, lets other teams rely on its output without following every internal decision.

Consider a vendor who signs up and starts receiving a whole sequence of marketing emails. Every subsequent step can work as designed once the wrong person enters the sequence. AI can provide an escape hatch at intake by checking whether the signup belongs in the customer workflow. The same approach lets a system reject an adversarial input or escalate something unusually consequential. Giving the model discretion over admission reduces the variety of cases the routine workflow has to handle.

### A research agent takes on drug-document indexing

[4:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=258s)

Clarivate maintained an annotated history of drug development spanning tens of millions of documents, with hundreds of thousands of edits a year. Its ontology gave indexers tens of thousands of ways to classify material. Inserting or editing a document could trigger work on both its index entries and its relationships to other documents. Those changes generated tens of thousands of new tasks a month. At that volume, the economics turned on what happened after the initial engineering investment. The finished automation was expensive to build, but Jewkes found it stable and cheap to run.

The first implementation resembled a research and assignment agent. After a few sprints it reached roughly 80–83% on the indexing work. For a team accustomed to spending months fine-tuning a **BERT** model for a single component of a language-processing pipeline, that was an exciting start. To understand the remaining failures, the engineers assembled 1,000 evaluation examples and sat down with domain experts to examine actual assignments.

On earlier projects, asking a busy expert to fill a blank spreadsheet with 100 evals had produced little momentum. Here an indexer could look at a decision the system had already made, recognize what it could do, and spend the meeting on a case that looked wrong. The engineers could then ask a question the indexer was particularly well placed to answer. What should have happened to this document?

### A wrong answer, an unsettled definition, and new science

[6:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=399s)

The review turned up a large group of contentious cases, a small group of errors everyone agreed could threaten important accounts, and perhaps 3–5% of cases that needed escalation. A single performance score had put three different problems in the same pile.

- **A known answer that the model missed.** In Jewkes’s experience, even strong models could fail around 2–3% of the time on multistep workflows. Rerunning the same case could make the failure disappear. When experts agreed on the required output and a mistake put an account at risk, the engineers had a clear target for reliability work.

- **A relationship the organization had yet to define.** In a simplified gout example, a drug binds to tubulin and interrupts a chain of events, reducing a downstream inflammatory molecule. Should the index list that downstream molecule as a drug target? A direct-binding definition yields one answer. A definition that includes indirect effects allows another. Models went back and forth, and two experts could confidently disagree. A reliable label required a decision about what the index meant by *target*.

- **A decidable case that required new scientific judgment.** What made the heart-disease drug candidate Verve 102 an escalation? The desired output was clear. The team wanted to index six mechanisms. Jewkes singled out four of the six as new to science. An expert had to work out how the ontology should represent those mechanisms before the system could treat them as familiar assignments.

### Where an indexing task can leave the pipeline

[8:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=532s)

Which tasks warranted the most heavily tested pipeline, and where could that pipeline send a case it still couldn’t settle? By Jewkes’s estimate, the team spent at least as much time building case routing as it spent improving accuracy on the final critical set. Routing kept unsettled definitions and unfamiliar inputs out of routine processing. It also concentrated testing on decisions where a wrong answer would be costly.

Most tasks could ship as usual. If a case looked unusual or the model was confused, the system could stop, have the AI describe the difficulty, and file the case under a category. Undecided cases also left the normal path. A plausibly business-critical task went through a pipeline tested against thousands of evals. That pipeline could usually ship an answer itself. An expert received the exception if uncertainty persisted. This second chance to escalate let the team handle high-stakes work without asking a person to approve every answer.

Two opportunities to catch an exception
An unfamiliar input can leave at intake. A critical task that fits the workflow can still reach a human later if the tested pipeline cannot resolve it.
- Route the indexing task — Check whether the case is familiar, settled, and plausibly business-critical.
- Categorized exception — Describe what is unusual and take the case out of routine processing.
- Undecided cases — Set aside work for which the organization lacks an agreed answer.
- Tested critical pipeline — Apply the workflow evaluated on thousands of examples.
- Ship the answer — Routine work and resolved critical work can both finish automatically.
- Expert review — Resolve the outstanding critical exception.
- Route the indexing task → Ship the answer (Routine)
- Route the indexing task → Categorized exception (Confusing or unfamiliar)
- Route the indexing task → Undecided cases (Unsettled definition)
- Route the indexing task → Tested critical pipeline (Plausibly critical)
- Tested critical pipeline → Ship the answer (Resolved)
- Tested critical pipeline → Expert review (Still uncertain)

### What the 99.95% measured

[9:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=598s)

The initial agent’s 80–83% described the broader indexing task. Once the team removed ambiguous cases and unusual escalations, standard approaches reached about 95% accuracy on the small set of business-critical tasks. The engineers then examined failures across many evals and refined the workflow for those structured inputs. On that critical set, they reached 99.95%. Going from about 5% errors to 0.05% meant reducing the remaining error rate by roughly a factor of 100.

The team waited to deploy until a random sample showed better performance than the historical indexing work. Consider the conditions under which experts had produced that work. A PhD-trained indexer might process hundreds of cases a week, with fatigue, pressure from other work, and the occasional misclick. Jewkes would still expect an expert given one case and time to examine every detail to outperform the system. The historical comparison answered the production question because that was the workload the automation would inherit.

### What an expert can see in a pile of exceptions

[11:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=684s)

Routing changed the expert’s workday twice. Routine decisions disappeared from it, and the remaining work arrived grouped by the kind of judgment it required. Experts could quickly review and ship the few unresolved critical cases. A pile of disputed assignments raised a different question. Was settling the definition worth anyone’s time?

A large backlog might motivate the company to write a specification. It might also show why nobody had written one. Some disputed assignments were only marginally valuable. Seeing them together gave the expert grounds to propose freezing that whole class of work until someone could make a business case for it. The company could make that decision once and spare its indexers the cost of reopening the question document by document.

The novel cases had a different value in aggregate. An indexer might encounter something like Verve 102 once a week or every other week, scattered among drugs with familiar mechanisms. One such document tells the expert little about how often the problem occurs, what it would cost to support, or who might pay. With two or three hours to consider similar exceptions together, the expert could look for a recurring need and propose a specific investment.

That preparation paid off on an eight-figure account. By the time the customer raised a problem, an expert who had studied the grouped cases already had an opinion about where to invest and what to build. A change in market demand can show up first as a series of awkward indexing jobs, before it arrives as a feature request.

### Which workflows justify the investment?

[14:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=891s)

Reading a spreadsheet of evals for hours is expensive when the person reading it is an AI engineer. Moving a non-engineering process into software also changes who owns it and whose work the business has to trust. For a project of that size, Jewkes would involve sales and marketing early enough to work out whether the investment could improve customer satisfaction, earn revenue, or open a neighboring market.

A promising candidate is already running at scale and earning a return. Jewkes’s example for a CFO is a category of tasks associated with seven figures of revenue and five customer tickets asking for a change. The revenue establishes what depends on the process. The tickets identify a problem customers actually feel. For a startup still discovering what customers want, he would preserve the flexibility of having people do the work.

Even an imperfect written specification is encouraging. Somebody close to the process thought its decisions could be written down and invested effort in trying. Detail-heavy work is particularly attractive when the burden comes from remembering and applying many rules reliably at volume. Work that depends on subjective judgment calls for more discretion. If that judgment is what makes the business distinctive, Jewkes would look elsewhere for a full automation.

### The time it takes to read 1,000 evals

[17:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1037s)

What did Jewkes put at the top and bottom of his list of techniques for going from 95% to 99.95%? At the top, read and write evals. At the bottom, build 1,000 of them and expect to spend time reading them. He sees clean inputs and a specified process as promising conditions for that level of accuracy. The cases that still defeat a model under those conditions interest him because they can suggest what to build next.

Everyone on the team had to spend time with individual failures. Reading a model’s agreeable prose and finding the point where it departed from the expected decision was exhausting. Yet that was how the team could tell whether an assignment needed an engineering fix, a definition from the business, or a scientist’s attention. A score alone couldn’t tell them whom to ask.

Jewkes closes with a paraphrase of advice he attributes to Rockefeller. Once you can specify your job well enough to hire someone to do it, sit down and think about the next opportunity. At Clarivate, the grouped exceptions gave the expert both that time and something concrete to think about. Jewkes bets the market will sort out competing orchestration tools and harnesses. He expects the greater advantage to come from how companies direct their collective attention, for as long as human judgment commands a premium.

## Key takeaways

- Bring domain experts completed examples. An actual indexing decision gives them a way to distinguish a model mistake from a rule the organization has yet to settle.
- Budget for routing as well as accuracy. Jewkes estimates that the team spent at least as much time deciding where cases belonged as it spent reducing errors on the critical set.
- Evaluate the workload the automation will inherit. The 99.95% result applied to defined critical tasks, and deployment depended on a comparison with historical indexing work.
- Group exceptions by the decision they require. Experts can clear unresolved critical cases, freeze marginal work, and examine recurring novel cases for evidence of customer demand.

## Slides

### Focus on domain experts

When an urgent customer ticket comes in, who does your team tag first? That expert may lead a small team or run a critical business process. Ask how they spend their week.

[0:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=33s) [Four bullets ask viewers to identify their strongest domain expert, consider who gets contacted first for priority work, identify a team or process lead, and ask how that person spends the week.](https://assets.ai.engineer/sites/aiecode2025/sha256/501e6d39457b8f81d2deec219542838d6386893fe45a939a812524c51ea16c47.webp)

Focus on domain experts

• Strongest domain expert
• First @ for priority GTDs
• Team or Process lead
• How do they spend their week?

STANDARD
CYBERNETICS

AI Engineer
World’s Fair

### Initial approach: do everything

The research and assignment agent reached 83% performance after a few sprints. The team used to spend months fine-tuning BERT for one pipeline step. It compiled about 1,000 evals, then hit a wall when it brought examples to domain experts.

[5:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=316s) [The research and assignment agent reached 83% performance after a few sprints. The team compiled about 1,000 evaluations and encountered difficulties when it met with domain experts.](https://assets.ai.engineer/sites/aiecode2025/sha256/f47f19e827a1aa6a73233355292453499d7c33efbc01ca84be9fae257e22b910.webp)

Initial approach: do everything

• Research & assignment agent
• 83% performance after a few sprints
  • Shockingly effective compared to pre - '23 BERT fine tuning
• Our team compiled ~1000 evals

• Met with domain expert team, quickly hit a wall

STANDARD
CYBERNETICS

### Experts say: a heterogeneous mix of problems

The largest group contained cases where experts could disagree or the organization had yet to decide. Two smaller groups posed different problems. Certain labels needed near-perfect recall, and exceptions needed a person to resolve them.

[6:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=400s) [The largest category contains cases the organization has yet to settle. Two smaller categories cover critical errors, where some labels need nearly perfect recall, and unusual cases that need human attention.](https://assets.ai.engineer/sites/aiecode2025/sha256/161831c49d06dbf5b119afb818047d392576cdd3cd8abee3eaf1641478d010e2.webp)

Experts say: a heterogeneous mix of problems

• Large bucket of organizationally undecided/contentious cases
• Small bucket of critical errors
  • In our case, certain labels needed to be applied with ~perfect recall.
• Small bucket of unusual "exceptions" or "escalations"
  • Cases that had an ideal resolution, but needed human attention.

STANDARD
CYBERNETICS

### Undecided: the models don't agree

Does lowering IL-1β indirectly count as targeting it? In this gout example, GPT 5.5 answers N and Opus 4.8 answers Y to the same question about colchicine. Jewkes says both models can give either answer.

[7:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=479s) [Two screenshots ask whether colchicine targets IL1 for gout. GPT 5.5 - High answers N on the left. Opus 4.8 - High answers Y on the right.](https://assets.ai.engineer/sites/aiecode2025/sha256/edbafda8934b7af6834abade49ab64bfac3a1e5e5125a544ab42def777501701.webp)

Undecided: the models don't agree

Does colchicine target IL1 for Gout? Answer only with Y or N.
N
GPT 5.5 - High

Does colchicine target IL1 for Gout? Answer only with Y or N.
Y
Opus 4.8 - High

Production cases more complex, but often of this flavor

STANDARD CYBERNETICS

Interface text:
Thought for 4s

### Overall outcome

Jewkes reports >99.95% accuracy on business-critical tasks. He says a PhD could do better on a single case, while fatigue and misclicks give the system an edge when experts handle hundreds of cases a week.

[10:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=645s) [The slide reports full automation of decidable cases and greater than 99.95% accuracy on business-critical tasks. It says a PhD expert could outperform the system on an individual case and attributes the system’s advantage at scale to fatigue, misclicks and random human error.](https://assets.ai.engineer/sites/aiecode2025/sha256/cc247042a7e0204e4ef76d0966737cd61cbf3ae77cb83f4a16ce94f5e97f003c.webp)

Overall outcome
Defined by the shape of the customer need & business

• Decidable cases fully automated.
• Routing + Hill-climbing very rarely misses a critical label
  • >99.95% accuracy on business critical tasks

• Better than PhD human's performance at scale
  • Plausibly a PhD can out perform the system on a singular case
  • But not at scale (fatigue, misclicks, random error)

STANDARD
CYBERNETICS

### Time efficiency + attention routing

Automation takes some routine work off the expert’s plate. It sorts the remaining cases by why they need attention, so the expert can review ambiguous cases, unusual exceptions, and critical cases the system was unsure about.

[11:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=691s) [An expert icon sits above three groups labeled Undecided, Exceptions and Critical Review. The bullets explain that automation saves the expert time and directs attention to ambiguous cases, unusual cases and critical cases the system was unsure about.](https://assets.ai.engineer/sites/aiecode2025/sha256/39a530a76d9bfc3e0b3fa171c4dcd2a3e762d8b8182ea6a83cef0cf94c87f83b.webp)

Time efficiency + attention routing

• Back to the expert from the beginning
• Automation gives them back some time and attention
• But it also organizes their attention in a structured way on the remainder.
• They get organized buckets of:
  • Undecided / Ambiguous cases
  • Exceptions / Unusual cases
  • Critical cases the automation wasn't sure of

Undecided   Exceptions   Critical Review

STANDARD
CYBERNETICS

### Where to go building

When choosing work to fully automate, look for tasks already tied to significant revenue. Seven figures of proven revenue makes the case concrete for the CFO. Can you point to a customer ticket about those same tasks, or even ten?

[15:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=939s) [Under “Valuable,” the slide recommends choosing work at scale with a material return on investment. It cites seven figures of proven revenue, a rationale the CFO can understand, and customer tickets tied to the tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a3fd5d39e36895afa51358b54590cbe90b102b86dff8c4b12e9e794a86de9d9.webp)

Where to go building
Valuable

• You want to pick a domain that is scaled, and has material RoI.
  • Think a task associated with 7 figures of proven revenue.
  • Something your CFO can understand
• Is there a customer ticket, or 10, associated with this bundle of tasks?

STANDARD
CYBERNETICS

## Full transcript

[0:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1s) [music]

[0:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=14s) Uh afternoon everybody. Uh I'm going to

[0:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=16s) Uh afternoon everybody. Uh I'm going to

[0:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=16s) Uh afternoon everybody. Uh I'm going to talk today about engineering agency out

[0:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=19s) talk today about engineering agency out

[0:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=19s) talk today about engineering agency out of the happy path. Um this is going to

[0:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=21s) of the happy path. Um this is going to

[0:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=21s) of the happy path. Um this is going to be focused on AI leadership, engineering

[0:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=25s) be focused on AI leadership, engineering

[0:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=25s) be focused on AI leadership, engineering leadership at growth stage companies.

[0:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=28s) leadership at growth stage companies.

[0:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=28s) leadership at growth stage companies. Um, and I'd like to start by asking

[0:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=30s) Um, and I'd like to start by asking

[0:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=30s) Um, and I'd like to start by asking everybody to think about the strongest

[0:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=32s) everybody to think about the strongest

[0:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=32s) everybody to think about the strongest domain experts on your team. So these

[0:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=35s) domain experts on your team. So these

[0:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=35s) domain experts on your team. So these are people who uh, you know, if sales

[0:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=37s) are people who uh, you know, if sales

[0:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=37s) are people who uh, you know, if sales wants to build a new feature or deliver

[0:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=39s) wants to build a new feature or deliver

[0:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=39s) wants to build a new feature or deliver something new in order to close an

[0:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=40s) something new in order to close an

[0:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=40s) something new in order to close an account or an urgent customer ticket

[0:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=42s) account or an urgent customer ticket

[0:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=42s) account or an urgent customer ticket appears, uh, these are the people who

[0:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=44s) appears, uh, these are the people who

[0:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=44s) appears, uh, these are the people who get at paint. Uh, they're the people

[0:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=46s) get at paint. Uh, they're the people

[0:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=46s) get at paint. Uh, they're the people maybe who are responsible for a small

[0:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=48s) maybe who are responsible for a small

[0:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=48s) maybe who are responsible for a small team or for a critical business process.

[0:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=51s) team or for a critical business process.

[0:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=51s) team or for a critical business process. And I want to think about how these

[0:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=52s) And I want to think about how these

[0:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=52s) And I want to think about how these people actually spend most of their work

[0:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=54s) people actually spend most of their work

[0:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=54s) people actually spend most of their work weeks.

[0:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=56s) weeks.

[0:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=56s) weeks. Uh my experience is that most experts

[0:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=58s) Uh my experience is that most experts

[0:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=58s) Uh my experience is that most experts I've worked with have their attention

[1:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=60s) I've worked with have their attention

[1:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=60s) I've worked with have their attention sort of smeared over multitasking,

[1:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=62s) sort of smeared over multitasking,

[1:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=62s) sort of smeared over multitasking, loading up on complicated situations,

[1:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=65s) loading up on complicated situations,

[1:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=65s) loading up on complicated situations, making decisions that could have been

[1:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=66s) making decisions that could have been

[1:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=66s) making decisions that could have been routine or obvious in retrospect and

[1:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=69s) routine or obvious in retrospect and

[1:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=69s) routine or obvious in retrospect and only a small fraction of their attention

[1:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=71s) only a small fraction of their attention

[1:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=71s) only a small fraction of their attention is left over to sort of move the

[1:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=73s) is left over to sort of move the

[1:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=73s) is left over to sort of move the business forward.

[1:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=75s) business forward.

[1:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=76s) business forward. Um now like the cost of like raw white

[1:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=78s) Um now like the cost of like raw white

[1:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=78s) Um now like the cost of like raw white collar output is dropping. It's

[1:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=81s) collar output is dropping. It's

[1:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=81s) collar output is dropping. It's cratering. I've talked to a bunch of

[1:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=82s) cratering. I've talked to a bunch of

[1:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=82s) cratering. I've talked to a bunch of people over the conference and

[1:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=84s) people over the conference and

[1:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=84s) people over the conference and regardless of what you think about

[1:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=85s) regardless of what you think about

[1:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=85s) regardless of what you think about frontier models and where the future's

[1:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=86s) frontier models and where the future's

[1:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=86s) frontier models and where the future's going to go in terms of new capacity.

[1:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=88s) going to go in terms of new capacity.

[1:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=88s) going to go in terms of new capacity. Everybody I've talked to has agreed that

[1:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=90s) Everybody I've talked to has agreed that

[1:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=90s) Everybody I've talked to has agreed that existing capacity is going to get very

[1:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=92s) existing capacity is going to get very

[1:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=92s) existing capacity is going to get very very cheap.

[1:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=94s) very cheap.

[1:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=94s) very cheap. Cheap execution increases the load and

[1:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=96s) Cheap execution increases the load and

[1:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=96s) Cheap execution increases the load and the on on judgment on deciding what to

[1:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=99s) the on on judgment on deciding what to

[1:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=99s) the on on judgment on deciding what to do. Uh organizational judgment was

[1:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=102s) do. Uh organizational judgment was

[1:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=102s) do. Uh organizational judgment was already in short supply. Uh and this is

[1:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=104s) already in short supply. Uh and this is

[1:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=104s) already in short supply. Uh and this is going to make it even worse. I think

[1:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=105s) going to make it even worse. I think

[1:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=105s) going to make it even worse. I think this is currently underpriced.

[1:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=108s) this is currently underpriced.

[1:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=108s) this is currently underpriced. So with that in mind, I'm going to talk

[1:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=109s) So with that in mind, I'm going to talk

[1:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=109s) So with that in mind, I'm going to talk today about using AI for full

[1:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=110s) today about using AI for full

[1:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=110s) today about using AI for full automations. By full automations, I mean

[1:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=113s) automations. By full automations, I mean

[1:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=113s) automations. By full automations, I mean taking humans out of the happy path

[1:55](https://www.youtube.com/watch?v=wWisOHEcMPE&t=115s) taking humans out of the happy path

[1:55](https://www.youtube.com/watch?v=wWisOHEcMPE&t=115s) taking humans out of the happy path completely. Using humans maybe for

[1:57](https://www.youtube.com/watch?v=wWisOHEcMPE&t=117s) completely. Using humans maybe for

[1:57](https://www.youtube.com/watch?v=wWisOHEcMPE&t=117s) completely. Using humans maybe for exceptions, but not not human in the

[1:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=118s) exceptions, but not not human in the

[1:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=118s) exceptions, but not not human in the loop um style automations. I think that

[2:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=121s) loop um style automations. I think that

[2:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=121s) loop um style automations. I think that these types of processes can improve

[2:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=123s) these types of processes can improve

[2:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=123s) these types of processes can improve margins kind of obvious. They can reduce

[2:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=126s) margins kind of obvious. They can reduce

[2:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=126s) margins kind of obvious. They can reduce complexity of a business by

[2:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=127s) complexity of a business by

[2:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=127s) complexity of a business by encapsulating whole chunks of work. Uh

[2:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=129s) encapsulating whole chunks of work. Uh

[2:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=129s) encapsulating whole chunks of work. Uh and then you can recover judgment from

[2:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=131s) and then you can recover judgment from

[2:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=131s) and then you can recover judgment from your experts and hopefully route it

[2:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=132s) your experts and hopefully route it

[2:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=132s) your experts and hopefully route it towards more important um tasks or

[2:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=135s) towards more important um tasks or

[2:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=135s) towards more important um tasks or questions. I'm going to talk about that

[2:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=136s) questions. I'm going to talk about that

[2:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=136s) questions. I'm going to talk about that specifically a bit later.

[2:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=138s) specifically a bit later.

[2:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=138s) specifically a bit later. Um, now the key to full automations and

[2:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=141s) Um, now the key to full automations and

[2:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=141s) Um, now the key to full automations and my experience has been engineering

[2:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=142s) my experience has been engineering

[2:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=142s) my experience has been engineering agency out of the happy path out of

[2:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=145s) agency out of the happy path out of

[2:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=145s) agency out of the happy path out of these automations. This is a bit of a

[2:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=147s) these automations. This is a bit of a

[2:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=147s) these automations. This is a bit of a reversal. You know, agents have

[2:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=148s) reversal. You know, agents have

[2:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=148s) reversal. You know, agents have permanently changed and expanded my idea

[2:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=150s) permanently changed and expanded my idea

[2:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=150s) permanently changed and expanded my idea of what I think is possible with

[2:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=152s) of what I think is possible with

[2:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=152s) of what I think is possible with software. But in enterprise, agents have

[2:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=154s) software. But in enterprise, agents have

[2:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=154s) software. But in enterprise, agents have problems. They're a pain to maintain.

[2:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=157s) problems. They're a pain to maintain.

[2:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=157s) problems. They're a pain to maintain. Uh, especially if skills and harnesses

[2:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=158s) Uh, especially if skills and harnesses

[2:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=158s) Uh, especially if skills and harnesses are changing. Uh, they fail in unusually

[2:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=162s) are changing. Uh, they fail in unusually

[2:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=162s) are changing. Uh, they fail in unusually shaped ways that enterprises find hard

[2:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=163s) shaped ways that enterprises find hard

[2:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=163s) shaped ways that enterprises find hard to work with. And then they increase the

[2:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=166s) to work with. And then they increase the

[2:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=166s) to work with. And then they increase the burden of decisions. When I see an

[2:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=168s) burden of decisions. When I see an

[2:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=168s) burden of decisions. When I see an expert or a leader responding to agent

[2:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=170s) expert or a leader responding to agent

[2:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=170s) expert or a leader responding to agent notifications, you know, maybe on their

[2:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=171s) notifications, you know, maybe on their

[2:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=171s) notifications, you know, maybe on their phone or or on their laptop, maybe they

[2:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=173s) phone or or on their laptop, maybe they

[2:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=173s) phone or or on their laptop, maybe they have an app for it. I'm often

[2:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=174s) have an app for it. I'm often

[2:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=174s) have an app for it. I'm often suspicious. Is that actually the best

[2:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=176s) suspicious. Is that actually the best

[2:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=176s) suspicious. Is that actually the best use of their, you know, valuable time

[2:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=178s) use of their, you know, valuable time

[2:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=178s) use of their, you know, valuable time and attention?

[3:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=180s) and attention?

[3:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=180s) and attention? Full automations, on the other hand,

[3:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=181s) Full automations, on the other hand,

[3:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=181s) Full automations, on the other hand, they free up attention. They're legible.

[3:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=184s) they free up attention. They're legible.

[3:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=184s) they free up attention. They're legible. So this means that, you know, they take

[3:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=185s) So this means that, you know, they take

[3:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=185s) So this means that, you know, they take an input, they produce an output, they

[3:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=187s) an input, they produce an output, they

[3:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=187s) an input, they produce an output, they encapsulate the whole thing, and they

[3:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=189s) encapsulate the whole thing, and they

[3:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=189s) encapsulate the whole thing, and they hide the complexity. And they're

[3:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=191s) hide the complexity. And they're

[3:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=191s) hide the complexity. And they're organizationally familiar. Like large

[3:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=193s) organizationally familiar. Like large

[3:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=193s) organizationally familiar. Like large human organizations have been building

[3:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=194s) human organizations have been building

[3:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=194s) human organizations have been building automations in software for decades.

[3:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=197s) automations in software for decades.

[3:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=197s) automations in software for decades. Think marketing, think DevOps. And

[3:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=199s) Think marketing, think DevOps. And

[3:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=199s) Think marketing, think DevOps. And they've been building uh automations in

[3:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=201s) they've been building uh automations in

[3:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=201s) they've been building uh automations in industry for centuries. Full automations

[3:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=203s) industry for centuries. Full automations

[3:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=203s) industry for centuries. Full automations are one of the key tools by which human

[3:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=206s) are one of the key tools by which human

[3:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=206s) are one of the key tools by which human organizations can do bigger and bigger

[3:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=207s) organizations can do bigger and bigger

[3:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=207s) organizations can do bigger and bigger things.

[3:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=214s) And that is giving the whole system an

[3:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=215s) And that is giving the whole system an

[3:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=215s) And that is giving the whole system an escape hatch. So, one of the problems

[3:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=217s) escape hatch. So, one of the problems

[3:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=217s) escape hatch. So, one of the problems with full automations is, and I'm sure

[3:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=219s) with full automations is, and I'm sure

[3:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=219s) with full automations is, and I'm sure people have seen this, when you get uh

[3:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=221s) people have seen this, when you get uh

[3:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=221s) people have seen this, when you get uh you sign up for something and maybe you

[3:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=223s) you sign up for something and maybe you

[3:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=223s) you sign up for something and maybe you accidentally get a whole marketing drip

[3:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=224s) accidentally get a whole marketing drip

[3:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=224s) accidentally get a whole marketing drip email when you were signing up as a

[3:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=226s) email when you were signing up as a

[3:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=226s) email when you were signing up as a vendor. Um, one of the interesting

[3:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=228s) vendor. Um, one of the interesting

[3:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=228s) vendor. Um, one of the interesting things you can do with AI and agency

[3:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=230s) things you can do with AI and agency

[3:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=230s) things you can do with AI and agency with automations is you can pre-filter

[3:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=231s) with automations is you can pre-filter

[3:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=231s) with automations is you can pre-filter inputs. You can check does this task

[3:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=234s) inputs. You can check does this task

[3:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=234s) inputs. You can check does this task actually match the happy path of the

[3:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=236s) actually match the happy path of the

[3:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=236s) actually match the happy path of the automation. You can reject adversarial

[3:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=238s) automation. You can reject adversarial

[3:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=238s) automation. You can reject adversarial situations. If something looks like it's

[4:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=240s) situations. If something looks like it's

[4:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=240s) situations. If something looks like it's business critical, sort of not a normal

[4:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=242s) business critical, sort of not a normal

[4:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=242s) business critical, sort of not a normal course of business operation, you can

[4:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=244s) course of business operation, you can

[4:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=244s) course of business operation, you can escalate that to a human. I think this

[4:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=246s) escalate that to a human. I think this

[4:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=246s) escalate that to a human. I think this is kind of neat. This is the use of AI,

[4:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=248s) is kind of neat. This is the use of AI,

[4:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=248s) is kind of neat. This is the use of AI, AI agency to reduce the variance of an

[4:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=251s) AI agency to reduce the variance of an

[4:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=251s) AI agency to reduce the variance of an automation to make things more smooth,

[4:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=253s) automation to make things more smooth,

[4:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=253s) automation to make things more smooth, which thereby lets us expand the scope

[4:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=255s) which thereby lets us expand the scope

[4:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=255s) which thereby lets us expand the scope of what we can automate. So I'm going to

[4:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=257s) of what we can automate. So I'm going to

[4:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=258s) of what we can automate. So I'm going to give a case study today. My background

[4:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=259s) give a case study today. My background

[4:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=259s) give a case study today. My background is working in bioarma data at scale. I

[4:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=261s) is working in bioarma data at scale. I

[4:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=262s) is working in bioarma data at scale. I had a company called deal enablement for

[4:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=264s) had a company called deal enablement for

[4:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=264s) had a company called deal enablement for bioharmaceutical mergers, licensing and

[4:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=266s) bioharmaceutical mergers, licensing and

[4:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=266s) bioharmaceutical mergers, licensing and acquisitions. In 23 we sold to clarvate.

[4:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=269s) acquisitions. In 23 we sold to clarvate.

[4:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=269s) acquisitions. In 23 we sold to clarvate. Clarvate is a really interesting

[4:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=270s) Clarvate is a really interesting

[4:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=270s) Clarvate is a really interesting company. They have an annotated history

[4:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=272s) company. They have an annotated history

[4:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=272s) company. They have an annotated history of the entire you know drug development

[4:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=274s) of the entire you know drug development

[4:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=274s) of the entire you know drug development industry. hundreds of millions of

[4:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=275s) industry. hundreds of millions of

[4:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=275s) industry. hundreds of millions of document or tens of millions of

[4:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=277s) document or tens of millions of

[4:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=277s) document or tens of millions of documents, hundreds of thousands of

[4:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=278s) documents, hundreds of thousands of

[4:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=278s) documents, hundreds of thousands of edits every year and a rich ontology for

[4:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=280s) edits every year and a rich ontology for

[4:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=280s) edits every year and a rich ontology for working of those sort of tens of

[4:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=281s) working of those sort of tens of

[4:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=282s) working of those sort of tens of thousands of ways of indexing it.

[4:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=284s) thousands of ways of indexing it.

[4:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=284s) thousands of ways of indexing it. [snorts] Um the automation we built was

[4:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=286s) [snorts] Um the automation we built was

[4:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=286s) [snorts] Um the automation we built was automating the indexing and the

[4:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=288s) automating the indexing and the

[4:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=288s) automating the indexing and the relationships between those documents.

[4:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=290s) relationships between those documents.

[4:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=290s) relationships between those documents. So this is processing tens of thousands

[4:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=292s) So this is processing tens of thousands

[4:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=292s) So this is processing tens of thousands of new tasks a month when a document's

[4:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=294s) of new tasks a month when a document's

[4:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=294s) of new tasks a month when a document's edited or inserted or something changes.

[4:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=296s) edited or inserted or something changes.

[4:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=296s) edited or inserted or something changes. We were able to achieve three and a half

[4:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=298s) We were able to achieve three and a half

[4:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=298s) We were able to achieve three and a half n of accuracy on the tasks that the

[5:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=300s) n of accuracy on the tasks that the

[5:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=300s) n of accuracy on the tasks that the business most valued. And I'll talk

[5:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=301s) business most valued. And I'll talk

[5:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=301s) business most valued. And I'll talk about what that means in a few minutes.

[5:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=303s) about what that means in a few minutes.

[5:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=303s) about what that means in a few minutes. Um, this was expensive to build. This

[5:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=305s) Um, this was expensive to build. This

[5:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=305s) Um, this was expensive to build. This was a capex cost. AI engineers aren't

[5:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=307s) was a capex cost. AI engineers aren't

[5:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=307s) was a capex cost. AI engineers aren't cheap. But once it was built, it was

[5:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=309s) cheap. But once it was built, it was

[5:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=309s) cheap. But once it was built, it was cheap to operate. It mostly stayed

[5:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=311s) cheap to operate. It mostly stayed

[5:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=311s) cheap to operate. It mostly stayed within its encapsulation. Um, and it was

[5:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=315s) within its encapsulation. Um, and it was

[5:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=315s) within its encapsulation. Um, and it was stable.

[5:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=317s) stable.

[5:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=317s) stable. So building this out, our initial

[5:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=319s) So building this out, our initial

[5:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=319s) So building this out, our initial approach, I think in 2024, we were

[5:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=322s) approach, I think in 2024, we were

[5:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=322s) approach, I think in 2024, we were excited by 40, we were excited by 03. We

[5:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=325s) excited by 40, we were excited by 03. We

[5:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=325s) excited by 40, we were excited by 03. We did something, maybe we were early for

[5:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=326s) did something, maybe we were early for

[5:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=326s) did something, maybe we were early for agents, but we built something that's

[5:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=328s) agents, but we built something that's

[5:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=328s) agents, but we built something that's sort of similar to what people would

[5:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=329s) sort of similar to what people would

[5:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=329s) sort of similar to what people would today think of as a research and

[5:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=330s) today think of as a research and

[5:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=330s) today think of as a research and assignment agent. After a few sprints,

[5:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=333s) assignment agent. After a few sprints,

[5:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=333s) assignment agent. After a few sprints, we hit, you know, 80 83% performance. We

[5:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=335s) we hit, you know, 80 83% performance. We

[5:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=335s) we hit, you know, 80 83% performance. We were really excited. We had been working

[5:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=336s) were really excited. We had been working

[5:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=336s) were really excited. We had been working in NLP for a decade. We were used to

[5:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=338s) in NLP for a decade. We were used to

[5:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=338s) in NLP for a decade. We were used to spending months fine-tuning a BERT model

[5:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=340s) spending months fine-tuning a BERT model

[5:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=340s) spending months fine-tuning a BERT model to just do one sort of component of one

[5:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=342s) to just do one sort of component of one

[5:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=342s) to just do one sort of component of one of these pipelines. We compiled a

[5:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=345s) of these pipelines. We compiled a

[5:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=345s) of these pipelines. We compiled a thousand devalves. We went we sat down

[5:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=346s) thousand devalves. We went we sat down

[5:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=346s) thousand devalves. We went we sat down with the domain experts.

[5:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=349s) with the domain experts.

[5:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=349s) with the domain experts. Um, I will say one good thing that we

[5:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=351s) Um, I will say one good thing that we

[5:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=351s) Um, I will say one good thing that we did is that we brought the domain

[5:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=352s) did is that we brought the domain

[5:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=352s) did is that we brought the domain experts a whole bunch of examples. And

[5:55](https://www.youtube.com/watch?v=wWisOHEcMPE&t=355s) experts a whole bunch of examples. And

[5:55](https://www.youtube.com/watch?v=wWisOHEcMPE&t=355s) experts a whole bunch of examples. And um I've worked on other projects where

[5:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=358s) um I've worked on other projects where

[5:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=358s) um I've worked on other projects where you go to a domain expert, you give them

[5:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=359s) you go to a domain expert, you give them

[5:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=359s) you go to a domain expert, you give them a spreadsheet and you say, "Hey, fill

[6:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=361s) a spreadsheet and you say, "Hey, fill

[6:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=361s) a spreadsheet and you say, "Hey, fill out a 100 evals." That doesn't go well.

[6:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=364s) out a 100 evals." That doesn't go well.

[6:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=364s) out a 100 evals." That doesn't go well. Uh bringing a blank spreadsheet to a

[6:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=366s) Uh bringing a blank spreadsheet to a

[6:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=366s) Uh bringing a blank spreadsheet to a busy expert uh doesn't create momentum.

[6:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=369s) busy expert uh doesn't create momentum.

[6:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=369s) busy expert uh doesn't create momentum. Uh by bringing them all these these

[6:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=371s) Uh by bringing them all these these

[6:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=371s) Uh by bringing them all these these these evals, these places we'

[6:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=372s) these evals, these places we'

[6:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=372s) these evals, these places we' experienced, we very quickly were able

[6:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=374s) experienced, we very quickly were able

[6:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=374s) experienced, we very quickly were able to get a the domain experts on board

[6:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=376s) to get a the domain experts on board

[6:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=376s) to get a the domain experts on board with what's possible and b get them

[6:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=378s) with what's possible and b get them

[6:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=378s) with what's possible and b get them fixated and interested in the places in

[6:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=380s) fixated and interested in the places in

[6:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=380s) fixated and interested in the places in which it failed. Um I do think that most

[6:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=383s) which it failed. Um I do think that most

[6:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=383s) which it failed. Um I do think that most automations are going to require input

[6:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=384s) automations are going to require input

[6:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=384s) automations are going to require input with domain experts working on these

[6:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=386s) with domain experts working on these

[6:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=386s) with domain experts working on these cases. If you can help them get started

[6:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=389s) cases. If you can help them get started

[6:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=389s) cases. If you can help them get started with evaluation,

[6:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=394s) but writing evals is an art and it can

[6:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=396s) but writing evals is an art and it can

[6:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=396s) but writing evals is an art and it can help get them started more quickly. But

[6:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=399s) help get them started more quickly. But

[6:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=399s) help get them started more quickly. But experts, we sat down with these eval.

[6:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=401s) experts, we sat down with these eval.

[6:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=401s) experts, we sat down with these eval. What did we find? Well, there was a

[6:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=402s) What did we find? Well, there was a

[6:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=402s) What did we find? Well, there was a large bucket of undecided or contentious

[6:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=406s) large bucket of undecided or contentious

[6:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=406s) large bucket of undecided or contentious cases. These were tasks where actually

[6:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=409s) cases. These were tasks where actually

[6:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=409s) cases. These were tasks where actually two experts might confidently disagree

[6:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=411s) two experts might confidently disagree

[6:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=411s) two experts might confidently disagree about what the answer is or where the

[6:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=413s) about what the answer is or where the

[6:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=413s) about what the answer is or where the organization itself hadn't quite

[6:55](https://www.youtube.com/watch?v=wWisOHEcMPE&t=415s) organization itself hadn't quite

[6:55](https://www.youtube.com/watch?v=wWisOHEcMPE&t=415s) organization itself hadn't quite decided. and everybody was not sure.

[6:57](https://www.youtube.com/watch?v=wWisOHEcMPE&t=417s) decided. and everybody was not sure.

[6:57](https://www.youtube.com/watch?v=wWisOHEcMPE&t=417s) decided. and everybody was not sure. Mixed in with that sort of

[6:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=418s) Mixed in with that sort of

[6:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=418s) Mixed in with that sort of heterogeneously was a small bucket of

[7:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=420s) heterogeneously was a small bucket of

[7:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=420s) heterogeneously was a small bucket of critical errors. These were errors where

[7:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=422s) critical errors. These were errors where

[7:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=422s) critical errors. These were errors where everybody agreed a misstep would, you

[7:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=424s) everybody agreed a misstep would, you

[7:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=424s) everybody agreed a misstep would, you know, threaten material accounts. We we

[7:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=426s) know, threaten material accounts. We we

[7:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=426s) know, threaten material accounts. We we just couldn't have them. And then mixed

[7:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=428s) just couldn't have them. And then mixed

[7:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=428s) just couldn't have them. And then mixed in with that was a whole sort of maybe 3

[7:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=430s) in with that was a whole sort of maybe 3

[7:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=430s) in with that was a whole sort of maybe 3 or 5% of exceptions or escalations,

[7:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=433s) or 5% of exceptions or escalations,

[7:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=433s) or 5% of exceptions or escalations, places that needed more attention.

[7:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=436s) places that needed more attention.

[7:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=436s) places that needed more attention. Uh do a quick example of these three. So

[7:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=439s) Uh do a quick example of these three. So

[7:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=439s) Uh do a quick example of these three. So the errors pretty straightforward. Even

[7:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=441s) the errors pretty straightforward. Even

[7:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=441s) the errors pretty straightforward. Even top models, if you have multi-step

[7:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=442s) top models, if you have multi-step

[7:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=442s) top models, if you have multi-step workflows fail 2 3% of the time. It's

[7:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=445s) workflows fail 2 3% of the time. It's

[7:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=445s) workflows fail 2 3% of the time. It's annoying when you see it at scale.

[7:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=446s) annoying when you see it at scale.

[7:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=446s) annoying when you see it at scale. Sometimes you can't even reproduce it,

[7:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=448s) Sometimes you can't even reproduce it,

[7:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=448s) Sometimes you can't even reproduce it, but it happens. For undecided or

[7:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=450s) but it happens. For undecided or

[7:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=450s) but it happens. For undecided or ambiguous cases, quick example, um, in

[7:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=453s) ambiguous cases, quick example, um, in

[7:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=453s) ambiguous cases, quick example, um, in drug development, uh, drug developers

[7:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=455s) drug development, uh, drug developers

[7:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=455s) drug development, uh, drug developers care a lot about targets. These are the,

[7:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=457s) care a lot about targets. These are the,

[7:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=457s) care a lot about targets. These are the, you know, key pathway or molecule that

[7:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=458s) you know, key pathway or molecule that

[7:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=458s) you know, key pathway or molecule that your drug actually impacts. Gout is a

[7:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=461s) your drug actually impacts. Gout is a

[7:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=461s) your drug actually impacts. Gout is a disease that's caused by overprouction

[7:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=463s) disease that's caused by overprouction

[7:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=463s) disease that's caused by overprouction of interlucan one beta. Culticine is a

[7:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=465s) of interlucan one beta. Culticine is a

[7:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=465s) of interlucan one beta. Culticine is a drug that reduces interlucan one beta by

[7:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=468s) drug that reduces interlucan one beta by

[7:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=468s) drug that reduces interlucan one beta by preventing a chain reaction basically.

[7:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=470s) preventing a chain reaction basically.

[7:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=470s) preventing a chain reaction basically. uh so colon itself binds to a protein

[7:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=474s) uh so colon itself binds to a protein

[7:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=474s) uh so colon itself binds to a protein called tubulin. Our system wanted to

[7:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=476s) called tubulin. Our system wanted to

[7:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=476s) called tubulin. Our system wanted to know does cultureine target interlucan

[7:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=478s) know does cultureine target interlucan

[7:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=478s) know does cultureine target interlucan one beta. And if you ask GPT55 sometimes

[8:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=481s) one beta. And if you ask GPT55 sometimes

[8:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=481s) one beta. And if you ask GPT55 sometimes it'll say no and if you ask you know

[8:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=483s) it'll say no and if you ask you know

[8:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=483s) it'll say no and if you ask you know opus 48 sometimes it'll say yes. They

[8:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=485s) opus 48 sometimes it'll say yes. They

[8:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=485s) opus 48 sometimes it'll say yes. They actually sort of both go back and forth

[8:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=487s) actually sort of both go back and forth

[8:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=487s) actually sort of both go back and forth on this. This is a toy case. It's

[8:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=489s) on this. This is a toy case. It's

[8:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=489s) on this. This is a toy case. It's actually the real situation is a little

[8:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=490s) actually the real situation is a little

[8:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=490s) actually the real situation is a little more complicated but it's

[8:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=491s) more complicated but it's

[8:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=491s) more complicated but it's representative. It gives a sort of

[8:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=492s) representative. It gives a sort of

[8:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=492s) representative. It gives a sort of flavor of um what undecided cases look

[8:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=495s) flavor of um what undecided cases look

[8:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=495s) flavor of um what undecided cases look like. Uh in terms of unusual or

[8:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=497s) like. Uh in terms of unusual or

[8:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=497s) like. Uh in terms of unusual or exception, I've got another bio example

[8:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=499s) exception, I've got another bio example

[8:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=499s) exception, I've got another bio example that I really like. Uh, this is a really

[8:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=501s) that I really like. Uh, this is a really

[8:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=501s) that I really like. Uh, this is a really exciting new potential drug called Verve

[8:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=502s) exciting new potential drug called Verve

[8:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=502s) exciting new potential drug called Verve 102, possibly cure heart disease, which

[8:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=505s) 102, possibly cure heart disease, which

[8:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=505s) 102, possibly cure heart disease, which is like it's pretty cool. I hope to get

[8:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=507s) is like it's pretty cool. I hope to get

[8:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=507s) is like it's pretty cool. I hope to get this or something like it for my little

[8:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=508s) this or something like it for my little

[8:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=508s) this or something like it for my little boy when he turns 18. Um, this does have

[8:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=512s) boy when he turns 18. Um, this does have

[8:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=512s) boy when he turns 18. Um, this does have a decidable output. We do want to index

[8:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=514s) a decidable output. We do want to index

[8:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=514s) a decidable output. We do want to index the sort of six mechanisms of this drug,

[8:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=517s) the sort of six mechanisms of this drug,

[8:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=517s) the sort of six mechanisms of this drug, but the four of them that I've circled

[8:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=518s) but the four of them that I've circled

[8:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=518s) but the four of them that I've circled here are all actually just new to

[8:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=519s) here are all actually just new to

[8:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=519s) here are all actually just new to science. So, like these didn't exist a

[8:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=521s) science. So, like these didn't exist a

[8:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=521s) science. So, like these didn't exist a few years ago. So, in order to properly

[8:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=524s) few years ago. So, in order to properly

[8:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=524s) few years ago. So, in order to properly solve this, we're going to require some

[8:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=525s) solve this, we're going to require some

[8:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=525s) solve this, we're going to require some human attention. The humans are going to

[8:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=526s) human attention. The humans are going to

[8:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=526s) human attention. The humans are going to have to think about how to index

[8:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=528s) have to think about how to index

[8:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=528s) have to think about how to index ontologies. This is an escalation

[8:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=532s) ontologies. This is an escalation

[8:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=532s) ontologies. This is an escalation path forward. How are we going to handle

[8:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=533s) path forward. How are we going to handle

[8:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=533s) path forward. How are we going to handle these three buckets? We need to make

[8:55](https://www.youtube.com/watch?v=wWisOHEcMPE&t=535s) these three buckets? We need to make

[8:55](https://www.youtube.com/watch?v=wWisOHEcMPE&t=535s) these three buckets? We need to make sure that errors both get routed into a

[8:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=538s) sure that errors both get routed into a

[8:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=538s) sure that errors both get routed into a place where we can grind out high

[8:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=539s) place where we can grind out high

[8:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=539s) place where we can grind out high performance and then we need to grind

[9:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=541s) performance and then we need to grind

[9:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=541s) performance and then we need to grind out the performance for exceptions and

[9:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=543s) out the performance for exceptions and

[9:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=543s) out the performance for exceptions and for undecided cases. We want to route

[9:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=544s) for undecided cases. We want to route

[9:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=544s) for undecided cases. We want to route those into places to keep them out of

[9:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=546s) those into places to keep them out of

[9:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=546s) those into places to keep them out of the normal system because they're going

[9:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=548s) the normal system because they're going

[9:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=548s) the normal system because they're going to require different types of handling.

[9:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=551s) to require different types of handling.

[9:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=551s) to require different types of handling. And I think probably in retrospect we

[9:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=553s) And I think probably in retrospect we

[9:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=553s) And I think probably in retrospect we spent at least as much time building

[9:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=555s) spent at least as much time building

[9:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=555s) spent at least as much time building case routing as we actually did on the

[9:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=557s) case routing as we actually did on the

[9:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=557s) case routing as we actually did on the final error climbing for the sort of

[9:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=559s) final error climbing for the sort of

[9:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=559s) final error climbing for the sort of critical set. Is the AI ever confused

[9:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=561s) critical set. Is the AI ever confused

[9:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=561s) critical set. Is the AI ever confused about what's going on? Is it unusual a

[9:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=563s) about what's going on? Is it unusual a

[9:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=563s) about what's going on? Is it unusual a case? Abort. Use the AI to describe

[9:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=565s) case? Abort. Use the AI to describe

[9:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=565s) case? Abort. Use the AI to describe what's going on. Categorize it away.

[9:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=567s) what's going on. Categorize it away.

[9:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=567s) what's going on. Categorize it away. Skip it from the pipeline. Is it

[9:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=569s) Skip it from the pipeline. Is it

[9:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=569s) Skip it from the pipeline. Is it plausibly business critical? Okay, then

[9:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=571s) plausibly business critical? Okay, then

[9:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=571s) plausibly business critical? Okay, then send that to the extremely robustly

[9:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=573s) send that to the extremely robustly

[9:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=573s) send that to the extremely robustly tested, you know, thousands of evals

[9:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=575s) tested, you know, thousands of evals

[9:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=575s) tested, you know, thousands of evals pipeline. Hopefully you can ship an

[9:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=577s) pipeline. Hopefully you can ship an

[9:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=577s) pipeline. Hopefully you can ship an answer. If through that pipeline it's

[9:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=579s) answer. If through that pipeline it's

[9:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=579s) answer. If through that pipeline it's still not sure, fine. and ship it back

[9:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=580s) still not sure, fine. and ship it back

[9:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=580s) still not sure, fine. and ship it back to a human as an exception. But vast

[9:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=582s) to a human as an exception. But vast

[9:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=582s) to a human as an exception. But vast majority of the time you can ship an

[9:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=583s) majority of the time you can ship an

[9:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=583s) majority of the time you can ship an answer

[9:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=586s) answer

[9:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=586s) answer is it undecided pipeline that out as

[9:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=587s) is it undecided pipeline that out as

[9:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=587s) is it undecided pipeline that out as well. Um for the majority of cases

[9:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=589s) well. Um for the majority of cases

[9:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=589s) well. Um for the majority of cases though they didn't fall into any of

[9:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=591s) though they didn't fall into any of

[9:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=591s) though they didn't fall into any of these cases and we could just ship as

[9:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=592s) these cases and we could just ship as

[9:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=592s) these cases and we could just ship as usual.

[9:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=598s) um in terms of the critical path. So you

[10:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=600s) um in terms of the critical path. So you

[10:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=600s) um in terms of the critical path. So you know we had this this small set of tasks

[10:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=602s) know we had this this small set of tasks

[10:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=602s) know we had this this small set of tasks that were absolutely business critical

[10:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=604s) that were absolutely business critical

[10:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=604s) that were absolutely business critical once we took out the ambiguous cases and

[10:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=606s) once we took out the ambiguous cases and

[10:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=606s) once we took out the ambiguous cases and once we took out the sort of um unusual

[10:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=608s) once we took out the sort of um unusual

[10:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=608s) once we took out the sort of um unusual escalation cases we're able to get to

[10:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=610s) escalation cases we're able to get to

[10:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=610s) escalation cases we're able to get to about 95% accuracy with just sort of

[10:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=612s) about 95% accuracy with just sort of

[10:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=612s) about 95% accuracy with just sort of standard approaches going from 95 to 3

[10:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=615s) standard approaches going from 95 to 3

[10:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=615s) standard approaches going from 95 to 3 and a half 9s. This is a case of I've

[10:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=618s) and a half 9s. This is a case of I've

[10:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=618s) and a half 9s. This is a case of I've got a whole bag of tricks on it. It's a

[10:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=619s) got a whole bag of tricks on it. It's a

[10:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=619s) got a whole bag of tricks on it. It's a case of hill climbing. It's a case of

[10:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=620s) case of hill climbing. It's a case of

[10:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=620s) case of hill climbing. It's a case of looking at a lot of evals using all of

[10:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=622s) looking at a lot of evals using all of

[10:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=622s) looking at a lot of evals using all of the normal tricks to juice performance

[10:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=624s) the normal tricks to juice performance

[10:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=624s) the normal tricks to juice performance out of LLMs. I basically do believe

[10:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=626s) out of LLMs. I basically do believe

[10:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=626s) out of LLMs. I basically do believe though that if your inputs are well

[10:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=627s) though that if your inputs are well

[10:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=628s) though that if your inputs are well structured and you're looking at a

[10:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=629s) structured and you're looking at a

[10:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=629s) structured and you're looking at a specific set of cases that you've

[10:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=630s) specific set of cases that you've

[10:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=630s) specific set of cases that you've articulated from your business, getting

[10:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=632s) articulated from your business, getting

[10:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=632s) articulated from your business, getting to three and a half nines is possible

[10:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=634s) to three and a half nines is possible

[10:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=634s) to three and a half nines is possible for most automations.

[10:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=636s) for most automations.

[10:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=636s) for most automations. Um, [snorts] we deployed this only once

[10:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=638s) Um, [snorts] we deployed this only once

[10:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=638s) Um, [snorts] we deployed this only once a random sample showed that this

[10:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=640s) a random sample showed that this

[10:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=640s) a random sample showed that this performance was better than the historic

[10:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=641s) performance was better than the historic

[10:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=641s) performance was better than the historic work. This is cool. So, um, I do think

[10:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=645s) work. This is cool. So, um, I do think

[10:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=645s) work. This is cool. So, um, I do think specifically um, I'm going to claim that

[10:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=647s) specifically um, I'm going to claim that

[10:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=647s) specifically um, I'm going to claim that this system performed better than PhD

[10:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=649s) this system performed better than PhD

[10:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=649s) this system performed better than PhD humans at scale. And I'll make that

[10:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=651s) humans at scale. And I'll make that

[10:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=651s) humans at scale. And I'll make that qualifier because I know that a PhD

[10:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=653s) qualifier because I know that a PhD

[10:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=653s) qualifier because I know that a PhD human if they're sitting down on a

[10:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=654s) human if they're sitting down on a

[10:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=654s) human if they're sitting down on a singular case and they're looking at all

[10:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=656s) singular case and they're looking at all

[10:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=656s) singular case and they're looking at all the details, they will outperform this

[10:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=658s) the details, they will outperform this

[10:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=658s) the details, they will outperform this system, but not at scale. Not when they

[11:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=660s) system, but not at scale. Not when they

[11:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=660s) system, but not at scale. Not when they have to look at hundreds of cases a

[11:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=661s) have to look at hundreds of cases a

[11:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=661s) have to look at hundreds of cases a week, not when they have fatigue, not

[11:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=663s) week, not when they have fatigue, not

[11:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=663s) week, not when they have fatigue, not when if you know one in a thousand times

[11:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=664s) when if you know one in a thousand times

[11:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=664s) when if you know one in a thousand times they misclick and not uh you know given

[11:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=667s) they misclick and not uh you know given

[11:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=667s) they misclick and not uh you know given time pressure of their other work.

[11:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=671s) Um so full automation is the happy path.

[11:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=673s) Um so full automation is the happy path.

[11:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=673s) Um so full automation is the happy path. Saves you time. That's great. You get

[11:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=675s) Saves you time. That's great. You get

[11:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=675s) Saves you time. That's great. You get better critical business critical task

[11:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=677s) better critical business critical task

[11:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=677s) better critical business critical task outcomes. That's also great. You know,

[11:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=678s) outcomes. That's also great. You know,

[11:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=678s) outcomes. That's also great. You know, this is a if you can take the places

[11:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=680s) this is a if you can take the places

[11:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=680s) this is a if you can take the places that are most valuable to the business

[11:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=681s) that are most valuable to the business

[11:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=682s) that are most valuable to the business and grind them out until you're

[11:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=683s) and grind them out until you're

[11:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=683s) and grind them out until you're performing better than humans at scale,

[11:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=684s) performing better than humans at scale,

[11:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=684s) performing better than humans at scale, that's great. There's a third benefit

[11:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=686s) that's great. There's a third benefit

[11:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=686s) that's great. There's a third benefit here, which is that it organizes the

[11:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=687s) here, which is that it organizes the

[11:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=687s) here, which is that it organizes the exhaust from the automation in a way

[11:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=689s) exhaust from the automation in a way

[11:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=689s) exhaust from the automation in a way that your experts can pay attention to.

[11:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=690s) that your experts can pay attention to.

[11:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=690s) that your experts can pay attention to. And I think this is pretty neat. So,

[11:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=693s) And I think this is pretty neat. So,

[11:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=693s) And I think this is pretty neat. So, back to that expert from the beginning.

[11:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=695s) back to that expert from the beginning.

[11:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=695s) back to that expert from the beginning. Hopefully, we've cleared their plate

[11:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=696s) Hopefully, we've cleared their plate

[11:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=696s) Hopefully, we've cleared their plate somewhat. We've taken some of the

[11:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=697s) somewhat. We've taken some of the

[11:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=697s) somewhat. We've taken some of the routine work out of their job. In

[11:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=699s) routine work out of their job. In

[11:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=700s) routine work out of their job. In addition, because we've invested so

[11:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=701s) addition, because we've invested so

[11:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=701s) addition, because we've invested so heavily in routing, we've now organized

[11:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=703s) heavily in routing, we've now organized

[11:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=703s) heavily in routing, we've now organized that we can organize their attention in

[11:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=704s) that we can organize their attention in

[11:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=704s) that we can organize their attention in a structured way among the around the

[11:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=707s) a structured way among the around the

[11:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=707s) a structured way among the around the remaining cases. So they're going to get

[11:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=709s) remaining cases. So they're going to get

[11:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=709s) remaining cases. So they're going to get a pile organized undecided and ambiguous

[11:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=712s) a pile organized undecided and ambiguous

[11:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=712s) a pile organized undecided and ambiguous cases, exceptions and unusual cases, and

[11:55](https://www.youtube.com/watch?v=wWisOHEcMPE&t=715s) cases, exceptions and unusual cases, and

[11:55](https://www.youtube.com/watch?v=wWisOHEcMPE&t=715s) cases, exceptions and unusual cases, and then a small number of critical cases

[11:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=716s) then a small number of critical cases

[11:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=716s) then a small number of critical cases the automation wasn't sure about. For

[11:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=719s) the automation wasn't sure about. For

[11:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=719s) the automation wasn't sure about. For those last ones, the sort of ones that

[12:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=720s) those last ones, the sort of ones that

[12:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=720s) those last ones, the sort of ones that are critical, they can quickly review

[12:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=722s) are critical, they can quickly review

[12:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=722s) are critical, they can quickly review them, very very few in number, and ship

[12:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=724s) them, very very few in number, and ship

[12:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=724s) them, very very few in number, and ship them. [snorts] For the ones that are

[12:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=726s) them. [snorts] For the ones that are

[12:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=726s) them. [snorts] For the ones that are organizationally ambiguous, this is

[12:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=728s) organizationally ambiguous, this is

[12:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=728s) organizationally ambiguous, this is actually kind of interesting. You know,

[12:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=729s) actually kind of interesting. You know,

[12:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=730s) actually kind of interesting. You know, the first thought is maybe this is a way

[12:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=731s) the first thought is maybe this is a way

[12:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=732s) the first thought is maybe this is a way to incentivize the organization to

[12:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=733s) to incentivize the organization to

[12:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=733s) to incentivize the organization to specify. You know, you've got 20% maybe

[12:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=736s) specify. You know, you've got 20% maybe

[12:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=736s) specify. You know, you've got 20% maybe of the total tasks that the organization

[12:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=738s) of the total tasks that the organization

[12:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=738s) of the total tasks that the organization wasn't sure with

[12:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=742s) these are often one of the reasons that

[12:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=744s) these are often one of the reasons that

[12:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=744s) these are often one of the reasons that they're not yet fully specified is that

[12:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=746s) they're not yet fully specified is that

[12:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=746s) they're not yet fully specified is that they're only marginally valuable. So the

[12:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=749s) they're only marginally valuable. So the

[12:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=749s) they're only marginally valuable. So the alternative here is that you can

[12:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=750s) alternative here is that you can

[12:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=750s) alternative here is that you can actually if the expert has a big pile of

[12:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=753s) actually if the expert has a big pile of

[12:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=753s) actually if the expert has a big pile of them in front of them they can actually

[12:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=754s) them in front of them they can actually

[12:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=754s) them in front of them they can actually advocate that hey let's permanently

[12:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=756s) advocate that hey let's permanently

[12:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=756s) advocate that hey let's permanently freeze this work let's prioritize it

[12:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=758s) freeze this work let's prioritize it

[12:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=758s) freeze this work let's prioritize it let's not think about it again until

[12:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=760s) let's not think about it again until

[12:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=760s) let's not think about it again until there's a business case too and let's

[12:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=761s) there's a business case too and let's

[12:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=761s) there's a business case too and let's get it off our plate that is a form of

[12:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=764s) get it off our plate that is a form of

[12:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=764s) get it off our plate that is a form of again reducing complexity in an

[12:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=765s) again reducing complexity in an

[12:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=765s) again reducing complexity in an organization having fewer things to

[12:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=767s) organization having fewer things to

[12:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=767s) organization having fewer things to think about so the focus and attention

[12:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=769s) think about so the focus and attention

[12:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=769s) think about so the focus and attention can go in the right place

[12:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=771s) can go in the right place

[12:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=771s) can go in the right place [snorts] I think this is even more

[12:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=772s) [snorts] I think this is even more

[12:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=772s) [snorts] I think this is even more interesting so groundbreaking work like

[12:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=774s) interesting so groundbreaking work like

[12:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=774s) interesting so groundbreaking work like that verve 102 drug for for heart

[12:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=776s) that verve 102 drug for for heart

[12:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=776s) that verve 102 drug for for heart disease it's only like 3% % of new drug

[12:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=779s) disease it's only like 3% % of new drug

[12:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=779s) disease it's only like 3% % of new drug development work. The vast majority of

[13:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=780s) development work. The vast majority of

[13:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=780s) development work. The vast majority of drug development work sort of pattern

[13:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=781s) drug development work sort of pattern

[13:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=781s) drug development work sort of pattern matches. Everybody's familiar with the

[13:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=783s) matches. Everybody's familiar with the

[13:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=783s) matches. Everybody's familiar with the rough shape of what you're looking at.

[13:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=784s) rough shape of what you're looking at.

[13:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=784s) rough shape of what you're looking at. So an expert might see this once a week,

[13:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=786s) So an expert might see this once a week,

[13:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=786s) So an expert might see this once a week, once every other week, one of these

[13:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=788s) once every other week, one of these

[13:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=788s) once every other week, one of these cases, they're scattered throughout. Uh

[13:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=790s) cases, they're scattered throughout. Uh

[13:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=790s) cases, they're scattered throughout. Uh the the expert might think, hey, I could

[13:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=792s) the the expert might think, hey, I could

[13:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=792s) the the expert might think, hey, I could build something to handle these cases.

[13:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=794s) build something to handle these cases.

[13:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=794s) build something to handle these cases. But if you're only seeing it

[13:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=795s) But if you're only seeing it

[13:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=795s) But if you're only seeing it sporadically, it's hard to build an

[13:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=797s) sporadically, it's hard to build an

[13:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=797s) sporadically, it's hard to build an intuition for how important it is, how

[13:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=799s) intuition for how important it is, how

[13:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=799s) intuition for how important it is, how expensive it would be, who would pay for

[13:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=800s) expensive it would be, who would pay for

[13:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=800s) expensive it would be, who would pay for it. You start to bucket those. And now

[13:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=803s) it. You start to bucket those. And now

[13:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=803s) it. You start to bucket those. And now the expert can think maybe when they sit

[13:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=804s) the expert can think maybe when they sit

[13:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=804s) the expert can think maybe when they sit down with two or three hours and look at

[13:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=806s) down with two or three hours and look at

[13:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=806s) down with two or three hours and look at them in aggregate, hey there's patterns

[13:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=808s) them in aggregate, hey there's patterns

[13:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=808s) them in aggregate, hey there's patterns here or hey maybe this is the specific

[13:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=810s) here or hey maybe this is the specific

[13:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=810s) here or hey maybe this is the specific investment I propose to start solving

[13:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=812s) investment I propose to start solving

[13:32](https://www.youtube.com/watch?v=wWisOHEcMPE&t=812s) investment I propose to start solving these for the organization and then if

[13:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=815s) these for the organization and then if

[13:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=815s) these for the organization and then if an an account you know a real customer

[13:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=817s) an an account you know a real customer

[13:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=817s) an an account you know a real customer comes and says hey I have a problem this

[13:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=819s) comes and says hey I have a problem this

[13:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=819s) comes and says hey I have a problem this expert now is ready to figure out how to

[13:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=821s) expert now is ready to figure out how to

[13:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=821s) expert now is ready to figure out how to how to deal with that how to move the

[13:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=822s) how to deal with that how to move the

[13:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=822s) how to deal with that how to move the business forward on it. So we saw this

[13:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=824s) business forward on it. So we saw this

[13:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=824s) business forward on it. So we saw this happen organically on a 8 figure account

[13:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=827s) happen organically on a 8 figure account

[13:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=827s) happen organically on a 8 figure account um and the expert had basically by

[13:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=830s) um and the expert had basically by

[13:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=830s) um and the expert had basically by having the data structured had already

[13:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=832s) having the data structured had already

[13:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=832s) having the data structured had already pre-formed an opinion about where to

[13:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=833s) pre-formed an opinion about where to

[13:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=833s) pre-formed an opinion about where to invest and where to build forward. I

[13:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=836s) invest and where to build forward. I

[13:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=836s) invest and where to build forward. I think this is actually the case for most

[13:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=838s) think this is actually the case for most

[13:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=838s) think this is actually the case for most scaled business processes. If you're

[14:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=841s) scaled business processes. If you're

[14:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=841s) scaled business processes. If you're working with the with with the market

[14:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=843s) working with the with with the market

[14:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=843s) working with the with with the market with data in aggregate you're collecting

[14:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=845s) with data in aggregate you're collecting

[14:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=845s) with data in aggregate you're collecting leading indicators of market demand,

[14:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=847s) leading indicators of market demand,

[14:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=847s) leading indicators of market demand, right? you know, the places that are

[14:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=848s) right? you know, the places that are

[14:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=848s) right? you know, the places that are slightly unusual or the places that um

[14:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=851s) slightly unusual or the places that um

[14:11](https://www.youtube.com/watch?v=wWisOHEcMPE&t=851s) slightly unusual or the places that um that the market's moving now that they

[14:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=852s) that the market's moving now that they

[14:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=852s) that the market's moving now that they weren't a few years ago, but they look

[14:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=854s) weren't a few years ago, but they look

[14:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=854s) weren't a few years ago, but they look like exceptions and if they're

[14:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=855s) like exceptions and if they're

[14:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=855s) like exceptions and if they're encountered infrequently, it's not

[14:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=856s) encountered infrequently, it's not

[14:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=856s) encountered infrequently, it's not enough signal to have somebody think

[14:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=858s) enough signal to have somebody think

[14:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=858s) enough signal to have somebody think about what to do with them. They're

[14:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=859s) about what to do with them. They're

[14:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=859s) about what to do with them. They're difficult to prioritize if they occur

[14:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=861s) difficult to prioritize if they occur

[14:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=861s) difficult to prioritize if they occur randomly. If you can organize this

[14:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=863s) randomly. If you can organize this

[14:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=863s) randomly. If you can organize this signal, this exhaust these exceptional

[14:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=865s) signal, this exhaust these exceptional

[14:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=865s) signal, this exhaust these exceptional cases that don't land in the happy path,

[14:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=867s) cases that don't land in the happy path,

[14:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=867s) cases that don't land in the happy path, your best talent can actually start to

[14:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=868s) your best talent can actually start to

[14:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=868s) your best talent can actually start to think about what to do with it. So,

[14:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=871s) think about what to do with it. So,

[14:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=871s) think about what to do with it. So, quick, we built a full automation. We

[14:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=874s) quick, we built a full automation. We

[14:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=874s) quick, we built a full automation. We can save hours. We can improve task

[14:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=876s) can save hours. We can improve task

[14:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=876s) can save hours. We can improve task level outcomes. We can reduce

[14:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=878s) level outcomes. We can reduce

[14:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=878s) level outcomes. We can reduce organizational complexity by taking a

[14:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=880s) organizational complexity by taking a

[14:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=880s) organizational complexity by taking a big bucket of tasks, encapsulating them,

[14:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=882s) big bucket of tasks, encapsulating them,

[14:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=882s) big bucket of tasks, encapsulating them, and having nobody have to think about

[14:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=883s) and having nobody have to think about

[14:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=883s) and having nobody have to think about them anymore. And then we can route

[14:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=885s) them anymore. And then we can route

[14:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=885s) them anymore. And then we can route talent towards higher value problems.

[14:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=887s) talent towards higher value problems.

[14:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=887s) talent towards higher value problems. This is a form of organizational

[14:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=889s) This is a form of organizational

[14:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=889s) This is a form of organizational refactoring.

[14:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=891s) refactoring.

[14:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=891s) refactoring. I will say that full automations,

[14:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=894s) I will say that full automations,

[14:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=894s) I will say that full automations, especially if you're trying to very high

[14:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=896s) especially if you're trying to very high

[14:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=896s) especially if you're trying to very high quality output, something like doing a

[14:57](https://www.youtube.com/watch?v=wWisOHEcMPE&t=897s) quality output, something like doing a

[14:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=898s) quality output, something like doing a PhD level expert's job, are expensive.

[15:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=901s) PhD level expert's job, are expensive.

[15:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=901s) PhD level expert's job, are expensive. AI engineers are expensive. Having an AI

[15:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=903s) AI engineers are expensive. Having an AI

[15:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=903s) AI engineers are expensive. Having an AI engineer spend hours reading a

[15:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=904s) engineer spend hours reading a

[15:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=904s) engineer spend hours reading a spreadsheet of eval is expensive. Once

[15:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=907s) spreadsheet of eval is expensive. Once

[15:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=907s) spreadsheet of eval is expensive. Once you get to that 95% going from 95 to

[15:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=910s) you get to that 95% going from 95 to

[15:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=910s) you get to that 95% going from 95 to 99.95, that's an expensive process,

[15:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=912s) 99.95, that's an expensive process,

[15:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=912s) 99.95, that's an expensive process, etc., etc., etc. Beyond that, you

[15:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=914s) etc., etc., etc. Beyond that, you

[15:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=914s) etc., etc., etc. Beyond that, you actually have to think if you're going

[15:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=915s) actually have to think if you're going

[15:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=915s) actually have to think if you're going to be refactoring a um a non-engineering

[15:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=919s) to be refactoring a um a non-engineering

[15:19](https://www.youtube.com/watch?v=wWisOHEcMPE&t=919s) to be refactoring a um a non-engineering process, you're going to have

[15:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=920s) process, you're going to have

[15:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=920s) process, you're going to have organizational change, trust building,

[15:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=922s) organizational change, trust building,

[15:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=922s) organizational change, trust building, ownership change. If you're going to

[15:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=924s) ownership change. If you're going to

[15:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=924s) ownership change. If you're going to make an investment of this size,

[15:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=925s) make an investment of this size,

[15:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=926s) make an investment of this size, hopefully you're also working with your

[15:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=927s) hopefully you're also working with your

[15:27](https://www.youtube.com/watch?v=wWisOHEcMPE&t=927s) hopefully you're also working with your sales and marketing team to figure out

[15:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=928s) sales and marketing team to figure out

[15:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=928s) sales and marketing team to figure out how to juice that for revenue, how to

[15:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=930s) how to juice that for revenue, how to

[15:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=930s) how to juice that for revenue, how to increase customer satisfaction, how to

[15:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=931s) increase customer satisfaction, how to

[15:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=931s) increase customer satisfaction, how to expand into a neighboring market. So,

[15:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=933s) expand into a neighboring market. So,

[15:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=933s) expand into a neighboring market. So, this is a major project. Um, thinking

[15:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=936s) this is a major project. Um, thinking

[15:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=936s) this is a major project. Um, thinking about where to do them takes a little

[15:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=938s) about where to do them takes a little

[15:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=938s) about where to do them takes a little bit of of um of judgment. You want to

[15:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=940s) bit of of um of judgment. You want to

[15:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=940s) bit of of um of judgment. You want to pick a domain that is scaled and that

[15:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=942s) pick a domain that is scaled and that

[15:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=942s) pick a domain that is scaled and that already has material ROI. I would

[15:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=944s) already has material ROI. I would

[15:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=944s) already has material ROI. I would actually not propose building full

[15:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=946s) actually not propose building full

[15:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=946s) actually not propose building full automations if you're a startup, if

[15:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=948s) automations if you're a startup, if

[15:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=948s) automations if you're a startup, if you're in discovery, if you're looking

[15:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=949s) you're in discovery, if you're looking

[15:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=949s) you're in discovery, if you're looking at new and new businesses. Keep the

[15:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=950s) at new and new businesses. Keep the

[15:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=950s) at new and new businesses. Keep the flexibility of humans. But if you have a

[15:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=953s) flexibility of humans. But if you have a

[15:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=953s) flexibility of humans. But if you have a bucket of tasks that's associated with

[15:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=954s) bucket of tasks that's associated with

[15:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=954s) bucket of tasks that's associated with seven figures of revenue and that maybe

[15:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=956s) seven figures of revenue and that maybe

[15:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=956s) seven figures of revenue and that maybe there's a customer ticket, hopefully

[15:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=958s) there's a customer ticket, hopefully

[15:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=958s) there's a customer ticket, hopefully there's 10 customer tickets, the

[15:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=959s) there's 10 customer tickets, the

[15:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=959s) there's 10 customer tickets, the organization knows that there's pain

[16:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=961s) organization knows that there's pain

[16:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=961s) organization knows that there's pain here, that can justify an investment and

[16:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=963s) here, that can justify an investment and

[16:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=963s) here, that can justify an investment and that is usually something that a CFO can

[16:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=965s) that is usually something that a CFO can

[16:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=965s) that is usually something that a CFO can understand. You know, if we say, hey,

[16:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=967s) understand. You know, if we say, hey,

[16:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=967s) understand. You know, if we say, hey, this is associated with seven figures of

[16:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=969s) this is associated with seven figures of

[16:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=969s) this is associated with seven figures of revenue and there's five tickets.

[16:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=970s) revenue and there's five tickets.

[16:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=970s) revenue and there's five tickets. Customers don't like it or they want

[16:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=972s) Customers don't like it or they want

[16:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=972s) Customers don't like it or they want something else here. That's a pretty

[16:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=973s) something else here. That's a pretty

[16:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=973s) something else here. That's a pretty good leading signal.

[16:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=976s) good leading signal.

[16:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=976s) good leading signal. Sort of like in terms of the tasks

[16:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=978s) Sort of like in terms of the tasks

[16:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=978s) Sort of like in terms of the tasks themselves, look for business processes

[16:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=980s) themselves, look for business processes

[16:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=980s) themselves, look for business processes that are running. ideal if it has

[16:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=982s) that are running. ideal if it has

[16:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=982s) that are running. ideal if it has written specifications. I wouldn't worry

[16:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=983s) written specifications. I wouldn't worry

[16:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=983s) written specifications. I wouldn't worry too much about if the written

[16:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=984s) too much about if the written

[16:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=984s) too much about if the written specifications are any good.

[16:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=986s) specifications are any good.

[16:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=986s) specifications are any good. Specifications are really hard and I

[16:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=988s) Specifications are really hard and I

[16:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=988s) Specifications are really hard and I think that even if there has been

[16:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=990s) think that even if there has been

[16:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=990s) think that even if there has been momentum to build specifications that

[16:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=993s) momentum to build specifications that

[16:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=993s) momentum to build specifications that reflects that on the inside we the team

[16:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=996s) reflects that on the inside we the team

[16:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=996s) reflects that on the inside we the team believes that this is specifiable the

[16:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=998s) believes that this is specifiable the

[16:38](https://www.youtube.com/watch?v=wWisOHEcMPE&t=998s) believes that this is specifiable the team believes that this is not just raw

[16:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=999s) team believes that this is not just raw

[16:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1000s) team believes that this is not just raw judgment complicated detail orient

[16:45](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1005s) work now which is actually

[16:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1008s) work now which is actually

[16:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1008s) work now which is actually shockingly shockingly new and very very

[16:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1010s) shockingly shockingly new and very very

[16:50](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1010s) shockingly shockingly new and very very interesting. It's ideal if this is

[16:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1012s) interesting. It's ideal if this is

[16:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1012s) interesting. It's ideal if this is detail- oriented work because you can

[16:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1014s) detail- oriented work because you can

[16:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1014s) detail- oriented work because you can take this sort of expensive thought,

[16:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1016s) take this sort of expensive thought,

[16:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1016s) take this sort of expensive thought, memory, um, wrote work off of of humans

[17:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1020s) memory, um, wrote work off of of humans

[17:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1020s) memory, um, wrote work off of of humans who tend to fail at it at scale.

[17:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1022s) who tend to fail at it at scale.

[17:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1022s) who tend to fail at it at scale. Subjective judgment work is not. So

[17:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1024s) Subjective judgment work is not. So

[17:04](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1024s) Subjective judgment work is not. So that's where you have to draw the line.

[17:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1026s) that's where you have to draw the line.

[17:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1026s) that's where you have to draw the line. Uh, AI for like if your business is

[17:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1028s) Uh, AI for like if your business is

[17:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1028s) Uh, AI for like if your business is built around a process, AI to automate

[17:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1030s) built around a process, AI to automate

[17:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1030s) built around a process, AI to automate that specific differentiated process is

[17:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1032s) that specific differentiated process is

[17:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1032s) that specific differentiated process is still in my opinion um, it's not where

[17:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1035s) still in my opinion um, it's not where

[17:15](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1035s) still in my opinion um, it's not where the action is.

[17:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1037s) the action is.

[17:17](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1037s) the action is. So a bag of tricks here. I'm not going

[17:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1038s) So a bag of tricks here. I'm not going

[17:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1038s) So a bag of tricks here. I'm not going to get into this, but if you are working

[17:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1040s) to get into this, but if you are working

[17:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1040s) to get into this, but if you are working on something where you're going from 95

[17:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1041s) on something where you're going from 95

[17:21](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1041s) on something where you're going from 95 to 99.95%,

[17:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1043s) to 99.95%,

[17:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1043s) to 99.95%, here's a bag of tricks. This talk will

[17:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1044s) here's a bag of tricks. This talk will

[17:24](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1044s) here's a bag of tricks. This talk will be online. You can take a photo of this,

[17:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1046s) be online. You can take a photo of this,

[17:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1046s) be online. You can take a photo of this, contact me. I'd love to chat. I'm very

[17:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1048s) contact me. I'd love to chat. I'm very

[17:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1048s) contact me. I'd love to chat. I'm very interested in the ways that models fail.

[17:29](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1049s) interested in the ways that models fail.

[17:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1050s) interested in the ways that models fail. I feel like knowing where models fail is

[17:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1051s) I feel like knowing where models fail is

[17:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1051s) I feel like knowing where models fail is really interesting for what to build

[17:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1053s) really interesting for what to build

[17:33](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1053s) really interesting for what to build next. Um, but I'd be I'd love to chat

[17:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1055s) next. Um, but I'd be I'd love to chat

[17:35](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1055s) next. Um, but I'd be I'd love to chat about any places where you have clean

[17:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1057s) about any places where you have clean

[17:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1057s) about any places where you have clean input, uh, a sort of specified process,

[17:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1059s) input, uh, a sort of specified process,

[17:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1059s) input, uh, a sort of specified process, and you can't get to three and a half

[17:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1061s) and you can't get to three and a half

[17:41](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1061s) and you can't get to three and a half nights of accuracy. I think the two that

[17:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1063s) nights of accuracy. I think the two that

[17:43](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1063s) nights of accuracy. I think the two that I'll flag out of this is the top one

[17:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1064s) I'll flag out of this is the top one

[17:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1064s) I'll flag out of this is the top one which is read and write evals and the

[17:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1066s) which is read and write evals and the

[17:46](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1066s) which is read and write evals and the bottom one which is build a thousand

[17:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1067s) bottom one which is build a thousand

[17:47](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1067s) bottom one which is build a thousand evals and expect to spend time reading

[17:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1069s) evals and expect to spend time reading

[17:49](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1069s) evals and expect to spend time reading them. Um that was really one of the big

[17:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1072s) them. Um that was really one of the big

[17:52](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1072s) them. Um that was really one of the big learnings from my team is is everybody

[17:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1074s) learnings from my team is is everybody

[17:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1074s) learnings from my team is is everybody had to invest in reading exactly why

[17:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1076s) had to invest in reading exactly why

[17:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1076s) had to invest in reading exactly why this time the AI did not do the thing we

[17:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1079s) this time the AI did not do the thing we

[17:59](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1079s) this time the AI did not do the thing we expected on the emails and it's

[18:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1080s) expected on the emails and it's

[18:00](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1080s) expected on the emails and it's exhausting work because you're reading

[18:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1081s) exhausting work because you're reading

[18:02](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1082s) exhausting work because you're reading psychopantic text and you're trying to

[18:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1083s) psychopantic text and you're trying to

[18:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1083s) psychopantic text and you're trying to figure out what specifically went wrong

[18:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1086s) figure out what specifically went wrong

[18:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1086s) figure out what specifically went wrong or against my expectations.

[18:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1088s) or against my expectations.

[18:08](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1088s) or against my expectations. I'm close with this quote. This is one

[18:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1090s) I'm close with this quote. This is one

[18:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1090s) I'm close with this quote. This is one of my favorite quotes is Rockefeller uh

[18:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1092s) of my favorite quotes is Rockefeller uh

[18:12](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1092s) of my favorite quotes is Rockefeller uh paraphrase as soon as you can hire

[18:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1094s) paraphrase as soon as you can hire

[18:14](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1094s) paraphrase as soon as you can hire someone to do your job. Sit down, kick

[18:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1096s) someone to do your job. Sit down, kick

[18:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1096s) someone to do your job. Sit down, kick up your feet, think about how to make

[18:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1098s) up your feet, think about how to make

[18:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1098s) up your feet, think about how to make the company more money. What is he

[18:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1100s) the company more money. What is he

[18:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1100s) the company more money. What is he saying here? He's saying if your job is

[18:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1102s) saying here? He's saying if your job is

[18:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1102s) saying here? He's saying if your job is specifiable, get someone else to do it.

[18:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1105s) specifiable, get someone else to do it.

[18:25](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1105s) specifiable, get someone else to do it. What we want you to be doing is

[18:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1106s) What we want you to be doing is

[18:26](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1106s) What we want you to be doing is thinking, applying your judgment,

[18:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1108s) thinking, applying your judgment,

[18:28](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1108s) thinking, applying your judgment, looking for new opportunity. I think

[18:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1110s) looking for new opportunity. I think

[18:30](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1110s) looking for new opportunity. I think that's perennial advice. It's great

[18:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1111s) that's perennial advice. It's great

[18:31](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1111s) that's perennial advice. It's great advice. My hope is that AI might be able

[18:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1114s) advice. My hope is that AI might be able

[18:34](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1114s) advice. My hope is that AI might be able to help us do this even better. If AI

[18:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1116s) to help us do this even better. If AI

[18:36](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1116s) to help us do this even better. If AI can help route our attention towards the

[18:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1119s) can help route our attention towards the

[18:39](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1119s) can help route our attention towards the most valuable places, the most valuable

[18:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1120s) most valuable places, the most valuable

[18:40](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1120s) most valuable places, the most valuable opportunities, then maybe we can sit

[18:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1122s) opportunities, then maybe we can sit

[18:42](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1122s) opportunities, then maybe we can sit down and actually think about the most

[18:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1124s) down and actually think about the most

[18:44](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1124s) down and actually think about the most valuable things first. Um, my bet is

[18:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1128s) valuable things first. Um, my bet is

[18:48](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1128s) valuable things first. Um, my bet is that orchestration layer and and um the

[18:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1131s) that orchestration layer and and um the

[18:51](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1131s) that orchestration layer and and um the right harnesses, the market's going to

[18:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1133s) right harnesses, the market's going to

[18:53](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1133s) right harnesses, the market's going to sort that out and there's not going to

[18:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1134s) sort that out and there's not going to

[18:54](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1134s) sort that out and there's not going to be a huge amount of differentiated

[18:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1136s) be a huge amount of differentiated

[18:56](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1136s) be a huge amount of differentiated advantage in terms of picking the right

[18:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1138s) advantage in terms of picking the right

[18:58](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1138s) advantage in terms of picking the right AI orchestration software in 2026.

[19:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1141s) AI orchestration software in 2026.

[19:01](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1141s) AI orchestration software in 2026. My bet is that the companies that

[19:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1143s) My bet is that the companies that

[19:03](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1143s) My bet is that the companies that succeed the most with AI are going to be

[19:05](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1145s) succeed the most with AI are going to be

[19:06](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1146s) succeed the most with AI are going to be the ones who figure out how to best

[19:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1147s) the ones who figure out how to best

[19:07](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1147s) the ones who figure out how to best manage their collective attention and

[19:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1149s) manage their collective attention and

[19:09](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1149s) manage their collective attention and judgment because the premium on that is

[19:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1150s) judgment because the premium on that is

[19:10](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1150s) judgment because the premium on that is going to only go up at least for so long

[19:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1153s) going to only go up at least for so long

[19:13](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1153s) going to only go up at least for so long as human judgment holds a premium. Um

[19:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1156s) as human judgment holds a premium. Um

[19:16](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1156s) as human judgment holds a premium. Um that's it for my talk. It's exciting

[19:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1158s) that's it for my talk. It's exciting

[19:18](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1158s) that's it for my talk. It's exciting times everyone. Uh I hope everybody

[19:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1160s) times everyone. Uh I hope everybody

[19:20](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1160s) times everyone. Uh I hope everybody enjoys the conference and uh really

[19:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1162s) enjoys the conference and uh really

[19:22](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1162s) enjoys the conference and uh really excited about all the stuff that we're

[19:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1163s) excited about all the stuff that we're

[19:23](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1163s) excited about all the stuff that we're going to be building.

[19:37](https://www.youtube.com/watch?v=wWisOHEcMPE&t=1177s) [music]