Engineering Agency out of the Happy Path — Matthew Jewkes, Standard Cybernetics

When sales needs a feature to close an account, or an urgent customer ticket comes in, the same domain expert often gets called. The expensive part starts before the decision. Someone who already owns a team or a critical process has to reconstruct a complicated situation, only to discover that the answer looks routine in retrospect. As execution gets cheaper, a growth-stage company can do more things. Its busiest people still have to decide which ones are worth doing. That is the constraint that interests Matthew Jewkes.

In Jewkes’s definition of full automation, a routine input goes all the way to a finished output without human approval. People remain available for exceptions. An agent that sends a notification to an expert’s phone may have done considerable work, yet the expert still has to interrupt theirs to decide what happens next. Changing agent skills and harnesses also give engineers more to maintain, and unusual failures can be hard for the rest of the business to handle. A familiar automation, like one in marketing or DevOps, lets other teams rely on its output without following every internal decision.

Consider a vendor who signs up and starts receiving a whole sequence of marketing emails. Every subsequent step can work as designed once the wrong person enters the sequence. AI can provide an escape hatch at intake by checking whether the signup belongs in the customer workflow. The same approach lets a system reject an adversarial input or escalate something unusually consequential. Giving the model discretion over admission reduces the variety of cases the routine workflow has to handle.

Clarivate maintained an annotated history of drug development spanning tens of millions of documents, with hundreds of thousands of edits a year. Its ontology gave indexers tens of thousands of ways to classify material. Inserting or editing a document could trigger work on both its index entries and its relationships to other documents. Those changes generated tens of thousands of new tasks a month. At that volume, the economics turned on what happened after the initial engineering investment. The finished automation was expensive to build, but Jewkes found it stable and cheap to run.

The first implementation resembled a research and assignment agent. After a few sprints it reached roughly 80–83% on the indexing work. For a team accustomed to spending months fine-tuning a BERT model for a single component of a language-processing pipeline, that was an exciting start. To understand the remaining failures, the engineers assembled 1,000 evaluation examples and sat down with domain experts to examine actual assignments.

On earlier projects, asking a busy expert to fill a blank spreadsheet with 100 evals had produced little momentum. Here an indexer could look at a decision the system had already made, recognize what it could do, and spend the meeting on a case that looked wrong. The engineers could then ask a question the indexer was particularly well placed to answer. What should have happened to this document?

The review turned up a large group of contentious cases, a small group of errors everyone agreed could threaten important accounts, and perhaps 3–5% of cases that needed escalation. A single performance score had put three different problems in the same pile.

• A known answer that the model missed. In Jewkes’s experience, even strong models could fail around 2–3% of the time on multistep workflows. Rerunning the same case could make the failure disappear. When experts agreed on the required output and a mistake put an account at risk, the engineers had a clear target for reliability work.

• A relationship the organization had yet to define. In a simplified gout example, a drug binds to tubulin and interrupts a chain of events, reducing a downstream inflammatory molecule. Should the index list that downstream molecule as a drug target? A direct-binding definition yields one answer. A definition that includes indirect effects allows another. Models went back and forth, and two experts could confidently disagree. A reliable label required a decision about what the index meant by target.

• A decidable case that required new scientific judgment. What made the heart-disease drug candidate Verve 102 an escalation? The desired output was clear. The team wanted to index six mechanisms. Jewkes singled out four of the six as new to science. An expert had to work out how the ontology should represent those mechanisms before the system could treat them as familiar assignments.

Which tasks warranted the most heavily tested pipeline, and where could that pipeline send a case it still couldn’t settle? By Jewkes’s estimate, the team spent at least as much time building case routing as it spent improving accuracy on the final critical set. Routing kept unsettled definitions and unfamiliar inputs out of routine processing. It also concentrated testing on decisions where a wrong answer would be costly.

Most tasks could ship as usual. If a case looked unusual or the model was confused, the system could stop, have the AI describe the difficulty, and file the case under a category. Undecided cases also left the normal path. A plausibly business-critical task went through a pipeline tested against thousands of evals. That pipeline could usually ship an answer itself. An expert received the exception if uncertainty persisted. This second chance to escalate let the team handle high-stakes work without asking a person to approve every answer.

The initial agent’s 80–83% described the broader indexing task. Once the team removed ambiguous cases and unusual escalations, standard approaches reached about 95% accuracy on the small set of business-critical tasks. The engineers then examined failures across many evals and refined the workflow for those structured inputs. On that critical set, they reached 99.95%. Going from about 5% errors to 0.05% meant reducing the remaining error rate by roughly a factor of 100.

The team waited to deploy until a random sample showed better performance than the historical indexing work. Consider the conditions under which experts had produced that work. A PhD-trained indexer might process hundreds of cases a week, with fatigue, pressure from other work, and the occasional misclick. Jewkes would still expect an expert given one case and time to examine every detail to outperform the system. The historical comparison answered the production question because that was the workload the automation would inherit.

Routing changed the expert’s workday twice. Routine decisions disappeared from it, and the remaining work arrived grouped by the kind of judgment it required. Experts could quickly review and ship the few unresolved critical cases. A pile of disputed assignments raised a different question. Was settling the definition worth anyone’s time?

A large backlog might motivate the company to write a specification. It might also show why nobody had written one. Some disputed assignments were only marginally valuable. Seeing them together gave the expert grounds to propose freezing that whole class of work until someone could make a business case for it. The company could make that decision once and spare its indexers the cost of reopening the question document by document.

The novel cases had a different value in aggregate. An indexer might encounter something like Verve 102 once a week or every other week, scattered among drugs with familiar mechanisms. One such document tells the expert little about how often the problem occurs, what it would cost to support, or who might pay. With two or three hours to consider similar exceptions together, the expert could look for a recurring need and propose a specific investment.

That preparation paid off on an eight-figure account. By the time the customer raised a problem, an expert who had studied the grouped cases already had an opinion about where to invest and what to build. A change in market demand can show up first as a series of awkward indexing jobs, before it arrives as a feature request.

Reading a spreadsheet of evals for hours is expensive when the person reading it is an AI engineer. Moving a non-engineering process into software also changes who owns it and whose work the business has to trust. For a project of that size, Jewkes would involve sales and marketing early enough to work out whether the investment could improve customer satisfaction, earn revenue, or open a neighboring market.

A promising candidate is already running at scale and earning a return. Jewkes’s example for a CFO is a category of tasks associated with seven figures of revenue and five customer tickets asking for a change. The revenue establishes what depends on the process. The tickets identify a problem customers actually feel. For a startup still discovering what customers want, he would preserve the flexibility of having people do the work.

Even an imperfect written specification is encouraging. Somebody close to the process thought its decisions could be written down and invested effort in trying. Detail-heavy work is particularly attractive when the burden comes from remembering and applying many rules reliably at volume. Work that depends on subjective judgment calls for more discretion. If that judgment is what makes the business distinctive, Jewkes would look elsewhere for a full automation.

What did Jewkes put at the top and bottom of his list of techniques for going from 95% to 99.95%? At the top, read and write evals. At the bottom, build 1,000 of them and expect to spend time reading them. He sees clean inputs and a specified process as promising conditions for that level of accuracy. The cases that still defeat a model under those conditions interest him because they can suggest what to build next.

Everyone on the team had to spend time with individual failures. Reading a model’s agreeable prose and finding the point where it departed from the expected decision was exhausting. Yet that was how the team could tell whether an assignment needed an engineering fix, a definition from the business, or a scientist’s attention. A score alone couldn’t tell them whom to ask.

Jewkes closes with a paraphrase of advice he attributes to Rockefeller. Once you can specify your job well enough to hire someone to do it, sit down and think about the next opportunity. At Clarivate, the grouped exceptions gave the expert both that time and something concrete to think about. Jewkes bets the market will sort out competing orchestration tools and harnesses. He expects the greater advantage to come from how companies direct their collective attention, for as long as human judgment commands a premium.