Just-in-Time Ontologies: What Cheap Intelligence Means for Data — Hrishi Olickel, Southbridge AI

An Ohio campaign-finance pipeline had a concrete job: determine whether records belonged to the same person, company or committee. Its projected cost was about $10,000. When a new decision model became available, Southbridge’s systems tested more than a dozen approaches and integrated it. Ten days later, Hrishi Olickel reports, the team had completed the donor cases for under $800.

That result motivates a larger architectural question. Data infrastructure assumes transformation is expensive, especially when interpreting records requires judgment. Olickel’s premise is that increasingly cheap, accessible intelligence changes that constraint, just as cheaper compute and storage changed earlier architectures. He calls the search among cost, speed and accuracy “Pareto surfing”: finding a better combination for a particular workload rather than choosing one model permanently.

Olickel once wanted a universal enterprise schema himself. Ontologies—his “enterprise version of mind maps”—promise to give everything a stable place. Failed adoption and unfinished migrations explain some disappointments, but even an instantaneous migration would leave a deeper problem: the structure needed to interpret information changes with the decision.

Consider a request to write down the temperature. Three different constraints already shape what that record means:

• Source. A phone supplies regional temperature; a desk sensor supplies a more local, potentially less accurate reading; a watch supplies another observation. The person’s willingness to keep measuring also affects what data exists.
• Storage. A written record preserves only part of the observation. Device firmware may be omitted, and recording a timestamp to the second is impractical if writing the record itself takes longer than a second.
• Decision. The intended use determines which omissions matter. Measuring comfort requires observations throughout the day and confidence about who measured them. Mapping weather makes location, time, device and device changes important. Choosing where to install sensitive equipment may require removing the person’s influence from the measurement.

The same conflict appears inside the apparently simple word customer. Billing needs whoever pays. Support needs whoever calls. Compliance needs whoever the organization is responsible for. A refund question and a credit-extension question make different relationships relevant. A single customer box cannot settle those distinctions before the question is known.

A just-in-time ontology performs that reconciliation when the decision is known. Transformation merges the context in which information was recorded with the context in which it will be used. It still requires discernment; postponing the merge makes the requirements clearer. A fourth influence—the model’s own assumptions—will complicate that merge later.

Southbridge organizes data work by how often the implementation changes and how often it runs. The distinction matters because a one-off answer, a migration and a maintained pipeline have different opportunities for reuse:

• Write once, run once. Build a single-use transformation to join a dataset, mine information or answer one question, then tear it down. Olickel likens these jobs to enormous immediately invoked functions.
• Write many, run once. Build many workloads toward one large move, such as migrating years of records from poorly remembered systems. Each workload needs verification against its source and checks against the others because the move happens only once.
• Write many, run many. Maintain repeatable pipelines as new information arrives. Southbridge uses these for work ranging from benchmark design and coding to movie subtitles.

Across all three, the obligation is the same: people must be able to rely on the result. Southbridge built Hankweave, an open-source harness for data work, to support models working together at scale. The self-optimizing system Olickel describes needs more than a problem statement: it also needs a place to work and a way to fail safely before it can select models, run pilots and scale a promising design.

At a previous job, Olickel sometimes had to prevent a number from being produced at all. A request to make a decision “data-led” could be sensible with thousands of vessels on a platform, yet misleading when an experiment had only fifty users. His concern was that an A/B test would still return a result, and the existence of that number would give it persuasive force. Interactive agents amplify this problem by producing confident answers quickly, allowing a casual transformation mistake to persist in production.

One remedy is to separate three roles that coding agents often combine in one context window:

• Designer. Break down the problem, write specifications and authorize spending. This role benefits from slower, more deliberative models that can read documentation, consider alternatives and anticipate pipeline behavior.
• Worker. Read PDFs, write database records and check individual results. This role benefits from cheap, fast, specialized models that follow instructions repeatedly and have knowable defects. Decision models fit here because they make brief, repeated contact with information across large workloads.
• Optimizer. Start from a working run, try alternatives, debug, take notes and vary budgets to limit exposure. These “hill climbers” need reusable knowledge and a stable execution environment to improve the design.

This division explains why a new worker model could enter the Ohio pipeline without taking over its design. The system could test where that model helped while leaving specification and optimization to models suited to those jobs. Choosing the best worker therefore became an experiment inside the pipeline rather than a wholesale commitment to a new agent.

The Ohio development work began with three architectures: a cheap reasoning model answering alone; a decision model escalating uncertain cases to a reasoning model; and a reasoning model proposing answers for the decision model to check. These designs place reasoning and checking differently. Escalation depends on the first model recognizing uncertainty; proposal-and-check gives the second model an answer to inspect.

On about 200 held-out families, the cheap reasoning model alone matched the reported comparison model at about 2% of its cost. The decision-first approach got the core answers right but missed nine household links because it confidently answered no. What changed those answers? The team rewrote the household question to specify what counted as sufficient evidence, leaving models and thresholds unchanged. All thirteen development questions then changed from no to yes. Those thirteen development questions and the nine missed links are distinct reported counts; the result demonstrates sensitivity to the definition, rather than establishing general accuracy.

Scaling statewide required more than a dozen rounds of this qualification. Each round clarified a domain term or relationship: what counts as a committee, what a filer means, or what sharing an address establishes about two people. The useful ontology emerged through those decisions. All 203,317 cases received a label describing what the answer rested on and how far it had been checked.

Where did the savings—and the remaining expense—come from? The statewide job cost $795 against an $800 budget set in advance. First answers cost $128; second opinions, disagreement resolution and auditing cost $545, roughly four times as much. These figures make verification a major workload in its own right. Olickel expected the next state’s audit budget to approach zero because the pipeline was now built and verified; that was a forecast, not a reported second deployment.

The optimization effort was substantial even though the dollar bill was small. Over ten days, the agent tried more than a dozen architectures. Its hanks ran more than 4,000 times alongside 90,000 decision-model calls, and models inside those runs processed roughly six billion tokens. Most approaches emerged incrementally through building, running, debugging and trying alternatives.

Every iteration was a version-controlled hank, making changes easier to revert, understand, debug and critique. Lessons returned to a library of agentic patterns. That record is part of what makes the next job cheaper: the system retains tested methods and inspectable changes instead of repeatedly rediscovering them inside a chat.

The fourth schema is the model’s prior: its beliefs about what the world and the data should look like. More capable models do not automatically remove this problem. Olickel observes that they can defend an assumption more forcefully. A transformation can therefore be internally consistent while changing the meaning of its input.

Which conventions were lost in the Ohio work? Three failures show why ordinary reconciliation checks were insufficient:

• Category spelling. A pipeline searched for an in-kind category with an underscore, while the file used a capital I and a hyphen. It classified 15,658 in-kind contributions as ordinary money. Loan repayments also disappeared through the same kind of mismatch.
• Date formats. A parser tried four formats while the file used a fifth. As a result, 99.4% of transaction dates became empty. Row counts and dollar totals still matched, so the agent continued.
• Signed amounts. Several runs treated negative loan amounts as redundant notation and stored absolute values. A net $784,000 became $5.3 million of debt, with multiple models validating the wrong result.

A human familiar with campaign finance caught the missing in-kind contributions by knowing to look for them. Olickel describes this as intersecting human and model priors: a different set of expectations exposes an omission the model accepts. Human review also costs time, interrupts work and varies between reviewers, so the team tested whether models with different backgrounds could provide a useful first filter.

On 200 held-out families, two models agreed on 194, and none of those shared answers was wrong; every observed error fell among the six disagreements. That concentrates review on a much smaller set. Different model families can bring different assumptions, although distillation can make apparently separate models less independent. Agreement remains a fallible signal: in another benchmark, four model families produced the same total down to the cent because they shared a sign convention that differed from the answer key. Shared mistakes need explicit checks for the known failure, even after disagreement-based review has done its work.

Self-optimization needs four supporting capabilities. A model’s ability to propose another approach supplies only part of the system:

• Stable execution. Hankweave wraps changing agent runtimes such as Claude Code and Pi with a consistent interface, checkpointing, rollback and budget enforcement. The optimizer needs dependable machinery beneath its experiments.
• Bounded experiments. Cost, time and access must be controllable resources. Otherwise, trying alternatives has no reliable limit.
• Reusable methodology. Agentic patterns, debugging techniques and prompting knowledge give models concrete methods to apply. A useful abstraction makes that accumulated experience compact enough to carry into the next problem.
• A failure surface. The system needs a defined way to recognize an unsuccessful attempt. This gives revisions something to be tested against.

The failure surface is the hardest requirement to express in tooling and instructions. A stable runtime makes experiments repeatable, and resource limits make them affordable, but neither tells the optimizer whether its output is useful. That criterion has to come from the task.

The practical process starts by specifying and splitting the job. In Ohio, roughly 800,000 rows became about 200,000 specialized cases, with each case broken into typed questions about donor identity and related distinctions. Before testing architectures, the team established a baseline using capable models on a small data slice, with some manual checking. Other tasks can use schema checks, automated failure checks or measurable behavior. The requirement is to define failure before starting the search.

Minimal, bounded pilots then test candidate architectures. Once something works, the checks move into rigs: code that runs between bounded work segments called codons. Codons can be reused, looped independently, validated and placed in sequence. This decomposition lets a particular segment be revised and checked without treating the entire workload as one opaque attempt.

Pareto surfing happens around a working baseline. Swap a model or another component for a faster or cheaper alternative, compare the result against known intermediate outputs, and tighten the requirements for success. Each piece can improve toward its own useful tradeoff. Working pipelines then enter production and continue to mature; reusable pieces can also run in parallel across datasets.

This completes the meaning of a just-in-time ontology. Build the structure when the job makes its requirements clear, then keep it with the context and evidence that explain its answers. The next job can reuse that intermediate when its needs match. Olickel’s closing claim is conditional: many schemas can be valid when their context is labeled and preserved. Cheaper intelligence makes creating and adapting them more practical; the evidence attached to them makes their reuse intelligible.