Just-in-Time Ontologies: What Cheap Intelligence Means for Data — Hrishi Olickel, Southbridge AI
Read the talk
Just-in-Time Ontologies: What Cheap Intelligence Means for Data
Hrishi Olickel explains how decision-specific definitions, checked pipeline segments and bounded experiments let models reduce transformation costs—and why several models can still agree on the same wrong answer.
From a talk by Hrishi Olickel
At a glance
Ideas worth remembering
Build transformations around the decision: source observations, storage limits and intended use determine which distinctions matter.
Separate pipeline designers, repeatable workers and optimizers so each role can use models suited to its cost and reasoning requirements.
Definitions can change results without changing models: specifying sufficient household evidence corrected confident negative answers in the Ohio development work.
Cross-model disagreement can focus review, but shared assumptions can produce unanimous errors. Preserve domain-specific checks for categories, dates and sign conventions.
Optimize bounded, checked work segments against known intermediates, and preserve the context and evidence needed to reuse them.
A $10,000 projection becomes a job under $800
An Ohio campaign-finance pipeline had a concrete job: determine whether records belonged to the same person, company or committee. Its projected cost was about $10,000. When a new decision model became available, Southbridge’s systems tested more than a dozen approaches and integrated it. Ten days later, Hrishi Olickel reports, the team had completed the donor cases for under $800.
That result motivates a larger architectural question. Data infrastructure assumes transformation is expensive, especially when interpreting records requires judgment. Olickel’s premise is that increasingly cheap, accessible intelligence changes that constraint, just as cheaper compute and storage changed earlier architectures. He calls the search among cost, speed and accuracy “Pareto surfing”: finding a better combination for a particular workload rather than choosing one model permanently.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The temperature is only useful once you know the question
Olickel once wanted a universal enterprise schema himself. Ontologies—his “enterprise version of mind maps”—promise to give everything a stable place. Failed adoption and unfinished migrations explain some disappointments, but even an instantaneous migration would leave a deeper problem: the structure needed to interpret information changes with the decision.
Consider a request to write down the temperature. Three different constraints already shape what that record means:
- Source. A phone supplies regional temperature; a desk sensor supplies a more local, potentially less accurate reading; a watch supplies another observation. The person’s willingness to keep measuring also affects what data exists.
- Storage. A written record preserves only part of the observation. Device firmware may be omitted, and recording a timestamp to the second is impractical if writing the record itself takes longer than a second.
- Decision. The intended use determines which omissions matter. Measuring comfort requires observations throughout the day and confidence about who measured them. Mapping weather makes location, time, device and device changes important. Choosing where to install sensitive equipment may require removing the person’s influence from the measurement.
The same conflict appears inside the apparently simple word customer. Billing needs whoever pays. Support needs whoever calls. Compliance needs whoever the organization is responsible for. A refund question and a credit-extension question make different relationships relevant. A single customer box cannot settle those distinctions before the question is known.
A just-in-time ontology performs that reconciliation when the decision is known. Transformation merges the context in which information was recorded with the context in which it will be used. It still requires discernment; postponing the merge makes the requirements clearer. A fourth influence—the model’s own assumptions—will complicate that merge later.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three workloads, three reasons to preserve checks
Southbridge organizes data work by how often the implementation changes and how often it runs. The distinction matters because a one-off answer, a migration and a maintained pipeline have different opportunities for reuse:
- Write once, run once. Build a single-use transformation to join a dataset, mine information or answer one question, then tear it down. Olickel likens these jobs to enormous immediately invoked functions.
- Write many, run once. Build many workloads toward one large move, such as migrating years of records from poorly remembered systems. Each workload needs verification against its source and checks against the others because the move happens only once.
- Write many, run many. Maintain repeatable pipelines as new information arrives. Southbridge uses these for work ranging from benchmark design and coding to movie subtitles.
Across all three, the obligation is the same: people must be able to rely on the result. Southbridge built Hankweave, an open-source harness for data work, to support models working together at scale. The self-optimizing system Olickel describes needs more than a problem statement: it also needs a place to work and a way to fail safely before it can select models, run pilots and scale a promising design.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate the designer, the worker and the optimizer
At a previous job, Olickel sometimes had to prevent a number from being produced at all. A request to make a decision “data-led” could be sensible with thousands of vessels on a platform, yet misleading when an experiment had only fifty users. His concern was that an A/B test would still return a result, and the existence of that number would give it persuasive force. Interactive agents amplify this problem by producing confident answers quickly, allowing a casual transformation mistake to persist in production.
One remedy is to separate three roles that coding agents often combine in one context window:
- Designer. Break down the problem, write specifications and authorize spending. This role benefits from slower, more deliberative models that can read documentation, consider alternatives and anticipate pipeline behavior.
- Worker. Read PDFs, write database records and check individual results. This role benefits from cheap, fast, specialized models that follow instructions repeatedly and have knowable defects. Decision models fit here because they make brief, repeated contact with information across large workloads.
- Optimizer. Start from a working run, try alternatives, debug, take notes and vary budgets to limit exposure. These “hill climbers” need reusable knowledge and a stable execution environment to improve the design.
This division explains why a new worker model could enter the Ohio pipeline without taking over its design. The system could test where that model helped while leaving specification and optimization to models suited to those jobs. Choosing the best worker therefore became an experiment inside the pipeline rather than a wholesale commitment to a new agent.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The household-link failure was a definition problem
The Ohio development work began with three architectures: a cheap reasoning model answering alone; a decision model escalating uncertain cases to a reasoning model; and a reasoning model proposing answers for the decision model to check. These designs place reasoning and checking differently. Escalation depends on the first model recognizing uncertainty; proposal-and-check gives the second model an answer to inspect.
On about 200 held-out families, the cheap reasoning model alone matched the reported comparison model at about 2% of its cost. The decision-first approach got the core answers right but missed nine household links because it confidently answered no. What changed those answers? The team rewrote the household question to specify what counted as sufficient evidence, leaving models and thresholds unchanged. All thirteen development questions then changed from no to yes. Those thirteen development questions and the nine missed links are distinct reported counts; the result demonstrates sensitivity to the definition, rather than establishing general accuracy.
Scaling statewide required more than a dozen rounds of this qualification. Each round clarified a domain term or relationship: what counts as a committee, what a filer means, or what sharing an address establishes about two people. The useful ontology emerged through those decisions. All 203,317 cases received a label describing what the answer rested on and how far it had been checked.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Checking cost four times as much as answering
Where did the savings—and the remaining expense—come from? The statewide job cost $795 against an $800 budget set in advance. First answers cost $128; second opinions, disagreement resolution and auditing cost $545, roughly four times as much. These figures make verification a major workload in its own right. Olickel expected the next state’s audit budget to approach zero because the pipeline was now built and verified; that was a forecast, not a reported second deployment.
The optimization effort was substantial even though the dollar bill was small. Over ten days, the agent tried more than a dozen architectures. Its hanks ran more than 4,000 times alongside 90,000 decision-model calls, and models inside those runs processed roughly six billion tokens. Most approaches emerged incrementally through building, running, debugging and trying alternatives.
Every iteration was a version-controlled hank, making changes easier to revert, understand, debug and critique. Lessons returned to a library of agentic patterns. That record is part of what makes the next job cheaper: the system retains tested methods and inspectable changes instead of repeatedly rediscovering them inside a chat.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The fourth schema can silently rewrite the data
The fourth schema is the model’s prior: its beliefs about what the world and the data should look like. More capable models do not automatically remove this problem. Olickel observes that they can defend an assumption more forcefully. A transformation can therefore be internally consistent while changing the meaning of its input.
Which conventions were lost in the Ohio work? Three failures show why ordinary reconciliation checks were insufficient:
- Category spelling. A pipeline searched for an in-kind category with an underscore, while the file used a capital I and a hyphen. It classified 15,658 in-kind contributions as ordinary money. Loan repayments also disappeared through the same kind of mismatch.
- Date formats. A parser tried four formats while the file used a fifth. As a result, 99.4% of transaction dates became empty. Row counts and dollar totals still matched, so the agent continued.
- Signed amounts. Several runs treated negative loan amounts as redundant notation and stored absolute values. A net $784,000 became $5.3 million of debt, with multiple models validating the wrong result.
A human familiar with campaign finance caught the missing in-kind contributions by knowing to look for them. Olickel describes this as intersecting human and model priors: a different set of expectations exposes an omission the model accepts. Human review also costs time, interrupts work and varies between reviewers, so the team tested whether models with different backgrounds could provide a useful first filter.
On 200 held-out families, two models agreed on 194, and none of those shared answers was wrong; every observed error fell among the six disagreements. That concentrates review on a much smaller set. Different model families can bring different assumptions, although distillation can make apparently separate models less independent. Agreement remains a fallible signal: in another benchmark, four model families produced the same total down to the cent because they shared a sign convention that differed from the answer key. Shared mistakes need explicit checks for the known failure, even after disagreement-based review has done its work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give optimization a stable runtime and a defined way to fail
Self-optimization needs four supporting capabilities. A model’s ability to propose another approach supplies only part of the system:
- Stable execution. Hankweave wraps changing agent runtimes such as Claude Code and Pi with a consistent interface, checkpointing, rollback and budget enforcement. The optimizer needs dependable machinery beneath its experiments.
- Bounded experiments. Cost, time and access must be controllable resources. Otherwise, trying alternatives has no reliable limit.
- Reusable methodology. Agentic patterns, debugging techniques and prompting knowledge give models concrete methods to apply. A useful abstraction makes that accumulated experience compact enough to carry into the next problem.
- A failure surface. The system needs a defined way to recognize an unsuccessful attempt. This gives revisions something to be tested against.
The failure surface is the hardest requirement to express in tooling and instructions. A stable runtime makes experiments repeatable, and resource limits make them affordable, but neither tells the optimizer whether its output is useful. That criterion has to come from the task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Optimize checked pieces, then preserve their context
The practical process starts by specifying and splitting the job. In Ohio, roughly 800,000 rows became about 200,000 specialized cases, with each case broken into typed questions about donor identity and related distinctions. Before testing architectures, the team established a baseline using capable models on a small data slice, with some manual checking. Other tasks can use schema checks, automated failure checks or measurable behavior. The requirement is to define failure before starting the search.
Minimal, bounded pilots then test candidate architectures. Once something works, the checks move into rigs: code that runs between bounded work segments called codons. Codons can be reused, looped independently, validated and placed in sequence. This decomposition lets a particular segment be revised and checked without treating the entire workload as one opaque attempt.
Pareto surfing happens around a working baseline. Swap a model or another component for a faster or cheaper alternative, compare the result against known intermediate outputs, and tighten the requirements for success. Each piece can improve toward its own useful tradeoff. Working pipelines then enter production and continue to mature; reusable pieces can also run in parallel across datasets.
This completes the meaning of a just-in-time ontology. Build the structure when the job makes its requirements clear, then keep it with the context and evidence that explain its answers. The next job can reuse that intermediate when its needs match. Olickel’s closing claim is conditional: many schemas can be valid when their context is labeled and preserved. Cheaper intelligence makes creating and adapting them more practical; the evidence attached to them makes their reuse intelligible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- Beating RL With Reflection: GEPA and Optimize Anything
Explains a complementary optimization mechanism: execution traces guide edits to prompts and programs, while a Pareto pool preserves useful alternative candidates.
Read the complete timestamped transcript
- 0:04
So I really love this article from Hannes at DuckDB, where he asks if we lost a decade chasing distributed architectures that presumed compute and storage were slow and expensive. Capability in every era, though, defines architecture and tooling, and the tooling lags behind consistently. We first made compute cheaper than storage, and then we gave it to everybody, and then we made it faster. And every single time architectures undergo this massive change. All of our data infrastructure today is
- 0:34
built on the idea that transformation is expensive. What if that wasn't true anymore? I believe we've solved the final bottleneck in transformation, general intelligence or judgment, and we've made it accessible on tap all along the Pareto frontier, which puts us right on the verge of a paradigm shift. So in this talk, we're gonna cover why single ontologies always fail, the three roles models play in pipelines, intersecting model priors for better
- 1:04
results, and Pareto surfing, how today's models can tune a workload for cost, speed, and accuracy all at the same time. Here's a quick example. When Jeff launched, we already had an entity resolution pipeline running, working out Ohio's campaign finance records and whether they belong to the same person, company, or committee. It was projected to cost about ten thousand dollars. Our systems took that new model, worked out where it fit, tested over a dozen approaches, and
- 1:34
integrated it. Ten days after that first call, we had final ground truth for every donor case in Ohio under eight hundred dollars. That is the power of Pareto surfing and just-in-time transformations. So let's get into it.
- 1:52
Ontologies are the enterprise version of mind maps. It's this tar pit that sucks us all in, including me. I talked about a single universal schema when I founded Southbridge two years ago. The popular conception, and it's not wrong, it's just a small part of the picture, is that migrations and ontologies fail because of a lack of buy-in, that they're abandoned halfway through, that organizations and people never switch over. All of that is true, a hundred percent. But
- 2:21
buy-in isn't the primary reason. Say, if you could integrate and migrate all of your information into a single super schema in the blink of an eye, you still have to contend with the truth that there is no super schema. There actually are four schemas. The first one belongs to the source, wherever the information comes from. Let's take temperature as an example. Say I ask you to write down the temperature. The source schema is whatever you have access to. Your phone, which gives you your region's temperature,
- 2:52
sensor on your desk, which is less accurate, but it's more local, or your watch. But even your patience in following through on my request is part of that source schema. The second one is the storage schema. There's only so much you can write down. You can't write down the firmware of the device you're using. You can't even write the time down to the second because you couldn't finish writing all of that inside of a second. This is the schema imposed on you by storage. The third one belongs to the decision.
- 3:22
Why do I need you to write the temperature down? Maybe I'm figuring out what makes you comfortable, in which case I need temperatures from you spread throughout the day, and I need to make sure that it's you that's doing the measuring. Maybe I'm building weather migration maps, in which case I don't care who does the measuring, but I care a lot about your exact location, the time, the device you used, and whether that's changed. Or maybe I'm deciding whether to install sensitive equipment and where, and you yourself become this variable that I need to cancel out.
- 3:53
There's a fourth one, though, which we'll get to when we talk about priors. It's this recent addition. The true problem is that you will never know all four schemas at the same time, except at the moment that the decision is being made. In almost every enterprise and every home, most of the data being recorded has little idea what it's gonna be used for. Take a customer, the first box in every ontology. To billing, a customer is whoever pays. To support, it's whoever calls. To compliance, it's
- 4:23
whoever we're on the hook for. And the only time you know is when the decision is known. Should we refund this order? Can we extend this account more credit? Suddenly, what you need to know, how and to what degree, becomes extremely clear. This is why I was wrong. There is no universal schema. There is no concept of a customer that's divorced from the question that you wanna answer. And pulling these schemas together is the process of transformation, of making information actually
- 4:52
useful, taking the context that it was recorded in and merging it with the context of the decision, and that takes discernment. Doing that merge when the decision space is known instead of ahead of time is what I mean by a just-in-time ontology. So we'll come back to discernment with a story from my last job, where, like, most of my job was to make sure that certain numbers just did not exist ever.
- 5:19
Before that, a little bit about me and us. We are best described as obsessives. Obsessives who embed intelligence into pipelines. Pipelines that can scale, that can run over and over again to take work truly off our hands, and pipelines that can mature in cost, performance, and speed all on their own. And most of the work we do in data breaks into three regimes. The first is write once, run once. If you do any kind of development, think of these as
- 5:48
monster scale immediately invoked functions. They're built for something single use, like joining a dataset, mining information, or answering one specific question, and then they get torn down. The second is write many, run once, where lots of smaller and larger workloads are built for one large common goal. The closest thing you might know is like large migrations. Years of records moved out of systems nobody fully remembers. Lots of workloads built for one move and verified
- 6:18
individually at source, checked against the others because all of it only happens once. The third, though, and this is bread and butter, is the most common, is write many, run many. These are repeatable pipelines that are built and maintained over time as we add more information in. And at Southbridge, we've moved so much work off the main thread and into these repeatable systems from benchmark design, tracking, coding, to making subtitles for movies that we want to watch. Almost everything we do in the modern world
- 6:48
is a transformation of data, or it's bottlenecked by how hard it is to transform data. And in all of these cases, our job is simple: make sure people can rely on the results. In just two years, we've gone from figuring out how to make early three point five turbo level models reliable on things like healthcare, aging, mining research, to building Hankweave, our open source harness for data work, and then using that to get models to work together at terabyte scale. Today, we have systems that can self-optimize. You give
- 7:18
them the problem, a way to fail safely, a place to work, and they find the shape of the solution. They pick the right models, they run the pilots, they scale up, and they Pareto surf their way to the best option they can find.
- 7:34
So storied. At my last job, the question we heard most often was, "Can we make this decision data-led?" And sometimes that was the right question when we had thousands of vessels on the platform. The problem is that we also asked it when we only had fifty users. And if you know statistics, you can explain really well why A/B testing fifty users is a bad idea. Not because you're not gonna get a result, but because you will. So the only real option was to not
- 8:04
produce a number at all because the moment a number exists, it argues for a result. Today, though, it is worse. Interactive agents will always give you an answer. They'll give it to you quickly. You ask for a transformation, whatever, and you get a confident, "Yes, sir." What this means is mistakes add up quickly. Moment in chat can be a prod nightmare forever, just waiting to be discovered. The primary reason that this happens is that intelligence, as it relates to data work today,
- 8:34
occupies three separate roles that coding agents kind of mash into one context window. Let's pull those apart because it'll help a lot when we get to decision models because they fit into exactly one of these three. The first one is building and designing the pipeline, which is breaking down the problem, writing the early specs, authorizing spend. The second one, and this is usually confused with the first and given to the same models, is working inside the pipeline, actually doing the work, reading the PDF, writing to databases,
- 9:04
checking results. Before we get to the third one, I want to make it clear that these two roles pull in very different directions. The first one wants slower models with heavier, thicker tokens, models that can think more, walk around a problem and see the angles, digest documentation, understand how these pipelines get built, and predict how something will do before it even starts, like Opus or up. The models you want inside pipelines, though, are the opposite. You want them cheaper, faster,
- 9:34
specialized, good at following instructions, repeatable workers with knowable defects. You see the same thing in nature. A leafcutter ant colony, for example, runs on four different sizes of worker, small, medium, high, X high. As you watch any system that you design, review, or hear about, listen for what slots and where in this layer cake. It makes a big difference. Because we had this distinction, bringing in Jev in this newly exploding class of decision models was very easy because they're workers. Buil--
- 10:04
they're built for quick contact with information, and they're deployed at the bottom end of large workloads. So third category, though. This is something we call the hill climbers or surfers, models that can take a first run and self-optimize. Given the right knowledge, instructions, and an incredibly stable surface to stand on, they can try new things, vary budgets to limit cost exposure, debug quickly, take notes, surf their way not just to the right solution, but the optimal one. We
- 10:34
call that Pareto surfing, and we'll cover it at the end. But first, let's talk about our experience with Jev. Jev launched on a Tuesday when we were deep into an entity resolution pipeline. We were working on donors in Ohio's campaign finance. And at this point, a lot of people have seen this chart. We talked about the results before and adapted some of the ongoing work to the dataset, so you can download it for yourself, prompts and everything. What we didn't talk about is everything behind that one arrow
- 11:04
and what happened when we ran the same idea on the whole state. Here are all of the architectures we tried. We started with three designs on a small development set, a cheap reasoning model on its own, Jev with a reasoning model for anything it wasn't sure about, and a reasoning model proposing answers with Jev checking them. On about two hundred held-out families, the cheap reasoning model on its own did as well as Fable for about two percent of the cost. Jev first got the core answers right, but it missed
- 11:34
nine household links because it had confidently said no to every one of them. So we rewrote the questions without touching the models or the thresholds. The household question now said what counts as enough evidence, and Jev went from no on all thirteen development questions to yes. And as we scaled it statewide, we ran over a doz-dozen rounds of such qualification. Each round fixed a definition, what a committee is, what a filer means. When two people live at one address, what does that mean? Every one of Ohio's
- 12:04
two hundred and three thousand three hundred and seventeen cases now has a label that says what the answer rests on and how far it was checked. This is the hierarchy we discovered for this specific problem. And here's everything we learned about putting decision models into pipelines. I'll leave this up so you can screenshot it. But here's where the money went. The whole statewide job came to seven ninety-five on an eight hundred dollar budget that was set ahead of time. Checking cost about four times as much as doing, five
- 12:34
forty-five on second opinions, settling disagreements, and the audit against a hundred and twenty-eight for first answers. And before Jev, we expected this work to cost about ten thousand dollars. And now that the pipeline is built and verified, we expect the audit budget for the next state to be close to zero. The most amazing thing is that our agent, four A, running on Astra and then later on Opus, found most of these approaches incrementally on its own by building the system, running it,
- 13:03
debugging it, and trying new things. It ran for ten days, tried over a dozen architectures. Its hanks ran for more than four thousand times alongside ninety thousand calls to Jev, tried practically every model on the Pareto frontier, and the models inside those runs went through roughly six billion tokens. And once all of that was done, we wrote what we learned back into our library of agentic patterns. This work leaned on these existing ones, and we're adding new tested ones like the checking letter.
- 13:34
This is, to me, the value of taking work off the main thread for humans and for agents, because every single one of these iterations was a version-controlled hank. They were easy to revert, understand, debug, and critique. So can models do this natively now? Not quite. They're missing four things which we'll, we'll go over, and there's something that doesn't go away, though, as models get bigger and more capable, the prior. Just like humans, models bring their own
- 14:04
priors, their beliefs about what should be in the world to every task. This is the fourth schema that I mentioned before, the one the model brings with it.
- 14:15
Bigger models, in a strange way, are worse at this sometimes. They hold onto their beliefs and argue for them even more vehemently. Here's a few that we ran into. One pipeline looked for in-kind spelled with an underscore. The file spelled it with a capital I and a hyphen, so the model counted fifteen thousand six fifty-eight in-kind contributions as ordinary money, didn't double-check. The loan payment repayments also disappeared the same way. Another one tried four date formats
- 14:45
when the file used a fifth. Ninety-nine point four percent of the transaction dates then came out empty, and every row count and dollar so matched, so the agent moved on. Several runs noticed that loan amounts were negative, decided that the sign was redundant, and stored the absolute value. A net of seven hundred and eighty-four thousand became five point three million dollars of debt in the end, validated by multiple models. And this is quickly becoming the largest class of error I see
- 15:14
across companies at Datawork as models take over. Sometimes you can solve this just by not vibing, by checking results and applying human intuition. And what you're actually doing there when you do this is interesting. You're intersecting your priors with the models. In that first example, any human would catch missing in-kind contributions if they knew campaign finance or think to check. It's how we found it. But humans, speaking as a human on this particular task, are expensive. They are slow, they pause
- 15:44
timelines, they're even more non-deterministic than models, and they hate interruptions. Uh, but it turns out you can also get really far with cross-model prior intersection. On our two hundred held-out families, Jev and Luna agreed on a hundred and ninety-four, and none of those shared answers was wrong. Every error sat in the six families where they disagree. Models from different families, different pedigrees tend to disagree more and have different priors than models in the same group, for example. The exception,
- 16:14
I'm not gonna say who, are models that are distilled outside dif-- the same company. So y- with that kind of work, you can shrink the surface quite a bit, and the errors along with it down to a much smaller but persistent class of problems that apply to LLMs as a species. So in one benchmark, four different model families produced the same total down to the cent because they all shared a sign convention that the answer key didn't. And once you get that far, the only way forward is to list them, look out for them, and
- 16:44
apply specific solutions. So going back, what are the four things we're missing to be able to self-optimize? Number one is a stable deployment surface. We use Hankweave for this, a harness that takes the fast-moving surface of agent runtimes like Claude Code and Pi, and that exposes a consistent, well-tested login API interface with checkpointing, rollback, budget enforcement, everything built in, something you can build on. But other things fit this description.
- 17:14
To be fair, for headless data work, Hankweave, I would say, is still the oldest, most stable, and the most open. I'll leave the skill up if you'd like to try it. Number two is some way to bound your experiments. You can't test unless you can control cost, time, and access as resources. The third one is know-how and methodology. We found that this new class of models can take everything we've learned from two years of deployments, agentic patterns, debugging techniques, prompting, and apply it to complex problems.
- 17:45
With the right abstraction layer, you can express it very token efficiently, and the models can implement it. The fourth and the most important one is a failure surface. Archimedes said, "Give me a place to stand and I will move the Earth." And in our case, we haven't met a problem or a constraint that we couldn't iterate ourselves to solving, given a stable stack to build on and a failure surface to iterate against. That last one has been the hardest to put into words and into tooling and instructions that helps models get there.
- 18:15
The current best models are still only barely good at this, sparks of intelligence, so to speak. The whole process though involves going through six stages. First, we spec the job and we split it. For Jeff, that means splitting some eight hundred thousand rows into two hundred thousand smaller specialized cases, and each of those cases into typed questions. What kind of donor, one donor, you know, different, different questions that needed to get answered. Then we decide the failure surface. In this case, we ran good models on a small slice of the
- 18:45
data, with some of it being manually checked, to set a baseline that we can fail against. In other cases, we've set automated failure systems, we've set schema checks, we've also covered behavior as a thing that we can measure against. But it's very important to have to define a failure surface before you ever start. But now comes the fun part. We run minimal bounded pilots to test architecture. And once something starts working, we move those checks into rigs. Code that runs between
- 19:15
bounded work segments that we call codons. These codons are then reused, looped independently, validated, and placed in sequence.
- 19:25
Finally, working pipelines get deployed to production where they can slowly mature, and that middle part is where the surfing happens. We find an architecture that gives a solid result. We use it as a baseline, and we swap models and some other parts for faster, cheaper options, compare against known intermediates, tighten the requirements for success, each time moving closer and closer to the ideal, and you just watch the frontier move. Being able to break a complex problem into pieces means each piece can move its own--
- 19:55
to its own best position, as Adam Smith would say. In a system like this, ontologies exist as intermediates between intelligence and result, which makes them easier to reuse. That's the other half of what I mean by just-in-time ontologies. It's built when the job needs it, kept with its receipts, and reused when the next job needs the same thing. And once you have those, you can make these things cheaper. You can, you can run them in parallel across data sets to scale up. I close with an
- 20:25
apology, right? That the answer is there is no ontology. Not a universal one, at least. There never was one. The good news is that this makes every ontology and schema valid, as long as you can label and preserve the context around it. And if we can harness this cheap intelligence that's starting to percolate through the world around us, we're rapidly approaching a future where any information in any format is made as useful as it can be. And at Southbridge, we're working as hard as we can to make that future a reality.
- 20:55
Thank you.