State of Data

Picture an AI data supplier, and the familiar image is a room of annotators labeling pictures. That work remains part of the market: Sean Cai puts spending at $10–15 billion per lab annually, an estimate whose accounting scope he does not establish. But the more consequential scarcity is the data that takes a model from general competence to professional expertise.

If data is an industrial input to automating labor, its market extends across professions rather than stopping at a single software vertical. The supply chain also begins to separate into specialties. Early suppliers such as Mercor, Surge, and Scale AI integrated multiple functions because immature unit economics made that necessary. As the market develops, specialists can compete separately in sourcing people, building environments, designing rewards, and running evaluations.

More volume does not automatically produce proportionately more quality. Cai reports that labs work with 20–30 vendors because they distrust any one supplier’s ability to scale quality with quantity. He sees this fragmentation as a lasting feature of the market, not an intermediate stage before consolidation.

This creates demand for what Cai calls Antikythera mechanisms: bespoke systems that translate messy business context into evaluations. The acquisition playbook associated with GPQA—collecting difficult expert questions—does not straightforwardly extend to work that unfolds over many dependent steps and lacks a simple checkable answer.

Model improvement depends on compute, data, and talent working together. Cai frames this with a rudimentary model that penalizes imbalance among the inputs: abundant compute cannot deliver its full potential when the data needed to teach a capability is missing. This is a conceptual explanation of bottlenecks, not an established quantitative law.

He connects that imbalance to rising capital expenditure and AI revenue that has not kept pace. In this account, data is the underfunded input because it supplies the professional expertise that additional general capability alone does not provide. The opportunity is to relieve that constraint, rather than assume every increment of spending produces the same increment of useful performance.

An ERP row or a saved document records a state: what exists after some work has happened. Process data records how a professional reached that state—the trajectory, reasoning trace, and sequence of decisions between a blank page and a finished output. A collection of finished artifacts can omit precisely the decisions an agent needs to learn.

Cai adds a second distinction, concerning where the examples come from:

Type 1 involves minimal reward shaping by nonexperts. Type 2 can be useful while a model is learning elementary capabilities, but constructing a plausible exercise does not necessarily reproduce professional work. Cai’s illustrative move from 20% to 80% describes the role he assigns to realism, not a measured performance gain.

That distinction becomes more important as datasets depreciate. Their value changes when the model frontier moves: yesterday’s difficult examples may no longer teach much. A live business can keep generating new work and new decisions; an acquired, inactive startup’s codebase is a finite snapshot. Cai also alleges that suppliers routinely sell manufactured Type 2 examples as authentic Type 1 data, making provenance part of the buyer’s evaluation problem.

Which professional tasks become tractable first? Verification offers a useful starting point. Cai invokes Jason Wei’s verifier’s rule, calling it Verifier’s Law: tasks become easier to train when their results are easier to verify. Cai then develops his own three-axis framework.

These are Cai’s operational categories. They separate the ability to check an individual result from the availability of a continuing supply of checked work.

Coding benefits from all three. Unit tests make parts of correctness objectively checkable; developers generally share conventions about working code; and public repositories supply commits with messages that explain changes. GitHub therefore provides both work products and some of the context behind their production.

In biology, security, finance, healthcare, law, and matters of taste, correctness can be harder to establish and workflow evidence less accessible. Much of that evidence remains inside enterprises rather than in public collaboration systems. Cai points to attempts to buy Slack and Jira logs as one expression of demand for those missing traces.

His proposed progression runs from code to search, then finance, healthcare and law, followed by cybersecurity, biology, scientific discovery, and taste. This is a predictive market thesis, not a universal ranking of every task in those fields: the relevant unit of analysis is the profession’s individual tasks and how they score on the verification axes.

Verification becomes less trustworthy when the supplier controls both the target and the material sold to reach it. Cai criticizes a Type 2 production loop:

1. Hire domain experts and use chat to generate plausible tasks.
2. Have the experts solve those tasks.
3. Select cases where the model diverges from the expert answer.
4. Package those cases as a difficult benchmark.
5. Sell data intended to improve performance on that same benchmark.

The resulting difficulty may reflect the supplier’s selection process more than the demands of real work.

Cai describes this as Goodhart’s law with a commercial incentive: once the measure becomes a target, improving the target need not improve the underlying capability. The problem is sharper when the people defining correctness lack the expertise needed to represent the work faithfully. Labs, vendors, and enterprises can all end up optimizing without a clear view of which data actually helps.

There is also a structural limitation independent of vendor intent. An isolated, in-distribution question does not establish that a model can sustain correct reasoning through a long episode in which each decision changes the next one. Success on individual questions is insufficient evidence of competence across dependent work.

Even a well-chosen task does not produce a score independent of its evaluation setup. Cai identifies differences in the harness—the surrounding scaffold through which the model performs the task—and execution infrastructure as important sources of benchmark disagreement. Aggregate results can also hide false positives and false negatives.

His useful framing is statistical: one benchmark score under one scaffold is a sample from a distribution whose spread may never have been measured. Without varying the conditions, it is hard to know whether a difference reflects the model, its operating environment, or their interaction. Cai introduces his private benchmarking work as a way to examine that ambiguity at the level of individual rubrics.

The finance example uses three tasks from three real-world vendors: an annual recurring revenue (ARR) waterfall reconciliation, a leveraged buyout (LBO) valuation memo, and a hedge-fund long-short pair trade. These require extended work whose overall quality is not reducible to one exact answer. Cai evaluates them using deterministic verifiers paired with an LLM judge.

In his private finance evaluation, Cai reports that Opus 4.8 underperformed Opus 4.7 on multiple rubrics. He associates the regression with excessive self-reflection: additional reconsideration did not necessarily improve the work.

Cai reports that GPT 5.5 and Opus 4.8 scored within three points on the same finance task, while failing in opposite directions. GPT handled arithmetic better; Opus handled methodology better but lost ground on arithmetic. The aggregate gap therefore concealed different needs for further training. These are private observations, not a general model ranking; the talk does not provide the rubric scale, model snapshots, repetitions, or judge configuration needed to reproduce them.

Rubric analysis can expose the consequences of different post-training choices and identify which data a model needs next. A single total score cannot do that. Cai compares judging a model from an ill-matched setup to using a Swiss Army knife’s screwdriver to cut cheese and concluding that the knife is broken. His finance analysis feeds an RL environment report for labs, where those specific failure modes are more useful than an undifferentiated leaderboard position.

Data procurement can offer an upstream signal of where application development is heading. Cai reports Anthropic buying cybersecurity data in January and biological data in March–April, with related offerings following roughly two to three months later. He connects those purchases to offerings he calls Mythos, Cyber, and Claude Bio/Life Sciences. The procurement connection is his observation, not established causation, and should not be read as dating the initial life-sciences launch: Anthropic’s healthcare and life-sciences announcement places that earlier.

To assess a domain more directly, first classify its tasks on the three verification axes. Then inspect what a vendor means by long horizon:

• Enforced length: The task requires an extended sequence, rather than merely permitting one.
• Different tools: Tool calls perform distinct jobs and are not interchangeable.
• Consequential state: Earlier transitions constrain what can happen later.
• Failure recovery: The agent must recover when something goes wrong.

Vendor specifications vary, so the label alone does not establish these properties.

Next, inspect the raw evidence. Look for sequential decisions about the same entity, an expert action that can be inferred at each step, and outcomes recorded by independent parties. Economic availability matters too: the work must be obtainable on viable terms, and high-wage work offers a stronger economic reason to automate it.

Robotics shows why volume cannot replace that inspection. The choice among egocentric capture, teleoperation, and Universal Manipulation Interface (UMI) remains bound up with research questions about how demonstrations transfer into robot behavior. Cai recounts offers of 100,000 hours of iPhone footage as an example of sellers presenting quantity without establishing training suitability. That anecdote does not demonstrate the value of the footage.

A vendor choice in such a market can embed a bet on an unresolved modality. Cai sees successful RL environment companies as research accelerators: valuable, but potentially boutique businesses. His proposed route to a larger business is reusable infrastructure that also serves enterprise applications, rather than dependence on today’s procurement market or a single unsettled data modality.

Cai maps the opportunity with a conceptual chart: share of white-collar work on the vertical axis and task horizon on the horizontal axis. Short tasks are already more addressable. Farther right lies work with deep dependencies, where completing one step correctly is only useful if it supports the steps that follow.

He locates substantial economic value—and the need for new data infrastructure—in that longer tail. Each effective pipeline into real work can move the addressable threshold rightward. The chart expresses a direction of progress, not a numerical forecast: better access to authentic dependent work expands the kinds of tasks models can learn to perform.

Who captures that opportunity depends partly on whether applications can become independent of their model providers. Cai asserts that infrastructure pioneers never retain more than 10% of their markets in the long run; he offers a historical analogy rather than evidence for a universal ceiling. Railroads and company towns illustrate his concern about infrastructure owners extracting rents, while AWS and Google illustrate his distinction between controlling infrastructure and owning the application layer. He puts anti-distillation restrictions and enterprise exclusivity in the same discussion of attempts to preserve control.

Cai cites GLM 5.2 surpassing GPT on unspecified real-world rubrics as evidence that applications can decouple from a particular provider. Without named tasks or a GPT version, that comparison cannot establish a general ranking. Nor does substitution make models interchangeable commodities: differences in efficiency and modality remain consequential even when durable provider lock-in becomes harder to maintain.

For data companies, this shifts the customer and the product. If an enterprise can own its adaptation of a model, suppliers can help it build that intelligence rather than only sell training material to frontier labs. Cai says successful data businesses are already moving toward enterprise work and claims that Mercor and Handshake receive substantial enterprise revenue, without providing figures.

Owning enterprise intelligence requires an abstraction layer above individual base models. Cai describes three functions:

• Serve and route models: Select smaller, specialized models according to cost, latency, and performance requirements.
• Preserve the training investment: Manage RL datasets across base-model changes so adopting a new open-source base can trigger post-training again instead of starting from scratch.
• Translate business context into evaluations: Build the Antikythera mechanisms that express what successful work means inside a particular enterprise.

These are proposed infrastructure capabilities, not a deployment demonstrated in the recording. Together, they make the workflow and its learning process more durable than the current choice of base model.

Cai’s endpoint is that data companies become what he calls Neo labs: organizations whose value extends into services and applications that perform actual work. For researchers, that makes independent judgment about realism essential. Buying tasks and evaluations from a supplier should not also mean delegating the definition of real competence to that supplier.

For builders, the defensible asset is the pipeline into real work plus the infrastructure to keep retraining on it. A static dataset depreciates; a continuing relationship with a working business can generate fresh decisions, outcomes, and training opportunities as models improve underneath it.

Cai closes by announcing work on Antikythera mechanisms with real-world companies: monetizing their data assets and helping enterprises implement reinforcement learning as a service. It brings the market thesis back to an operational problem—turning the work a business already does into a continuing source of model improvement.