Stop Making Models Bigger, Make Them Behave — Kobie Crawford, Snorkel

Can a four-billion-parameter model outperform a 235-billion-parameter model at financial-analysis tool use? That is the concrete challenge behind Kobie Crawford’s presentation. Crawford introduces himself as a developer advocate at Snorkel, whose dataset work brings domain experts into the process of assuring data quality. The research, conducted with UC Berkeley’s rLLM and Agentica team, asks where that data can change task performance. The objective is to improve a particular behavior, not to argue that large models are intrinsically undesirable.

When an enterprise application falls short, replacing its model with a larger one is an obvious response. More reasoning capability may help, but it also adds serving load and inference cost. Production imposes other requirements: reliable execution, safety, security, and predictable behavior. A larger model does not automatically satisfy those requirements.

The deployment question becomes sharper after a successful proof of concept. Can the service run on premises? Can the organization host it without exporting sensitive data or depending on an external provider? In finance and healthcare, these constraints can make a smaller model attractive independently of its benchmark score. Crawford’s proposed intervention is reinforcement learning: train the model’s tool-use behavior, rather than treat the problem primarily as missing knowledge inside its weights.

The rLLM team calls the mismatch the “Terence Tao effect.” A financial analyst often needs to retrieve records with SQL and perform straightforward arithmetic; broad mathematical brilliance may be unnecessary for that workflow. Adding capability can resemble using a sledgehammer to crack a walnut if the actual obstacle is knowing how to interact with the database.

The demonstration gives Qwen3-235B-A22B a question about the year-over-year growth of YouTube ads revenue from 2023 to 2024. The model begins by querying a table that does not exist. It has not inspected the environment to discover the available tables. After the first failure, it guesses again, receives no usable result, and produces an answer anyway—an answer Crawford identifies as hallucinated. The failure occurs before the growth calculation: no revenue values have been successfully retrieved.

The model’s reasoning capability does not rescue this sequence. It needs the discipline to discover what it can query and to ground its answer in returned data. Crawford leaves the question open for a later comparison with the trained four-billion-parameter model.

Training begins with the dataset. Snorkel uses an internal platform to work with financial-analysis practitioners, PhD-level specialists, and experienced industry contributors. Their role is to make the tasks relevant to the domain. Verification then checks that each task fits the intended application, can actually be answered by querying the available data, and has a verifiable answer. A plausible financial question is insufficient if the environment cannot supply the evidence needed to solve it.

The training stack combines group relative policy optimization (GRPO), a four-billion-parameter starting model, the rLLM framework, and the team’s FinQA environment. Crawford reports that the training job cost under $500 per run. That is the reported run cost, not an all-in budget for expert data creation and the research project. The practical audience is teams already considering small, self-hosted models: targeted RL may be a tractable way to improve the model they want to deploy.

FinQA packages a specific tool set and the data needed to answer its financial questions. Crawford describes the deployed environment as self-contained, using Harbor and OpenEnv as familiar comparisons. That description applies to the environment package; it does not establish that model serving or reward judging for an entire training workflow runs offline.

Crawford describes availability through Prime Intellect and OpenEnv’s FinQA environment, including the OpenEnv GitHub repository. He also points to PyTorch and Hugging Face’s collaboration around hosting environments in Hugging Face Spaces. These packages make the environment something developers can inspect and adapt, rather than a task definition detached from its tools.

The talk distinguishes two evaluation sets:

These are benchmark counts, not the number of examples used to train the model. The published OpenEnv README explicitly identifies its 290-question dataset as evaluation-only.

Crawford reports that the trained four-billion-parameter model outperforms the 235-billion-parameter comparator on this financial tool-use evaluation, with pass@1 approximately doubling from its own starting level. Pass@1 measures success on a single attempt at a task. The resulting model, rLLM-FinQA-4B, is fine-tuned from Qwen3-4B-Instruct-2507. The relevant question is what changed inside its interaction with the environment.

Return to the YouTube ads revenue question. This time the agent starts by discovering tables with get_table_names, a tool that was also available to the larger model. It then calls get_table_info to inspect the schema before constructing a query. These are the demonstrated rLLM tool names; the current OpenEnv interface names discovery get_descriptions, so the two interfaces should not be substituted silently.

The next query is still imperfect: it asks for a revenue column that is not present. But the agent observes the error and corrects its column choice. It then reaches the answer Crawford identifies as correct. The successful trajectory is therefore:

1. Discover the available tables.
2. Inspect the relevant schema.
3. Issue a query.
4. Use the returned error to repair an invalid column reference.
5. Answer from the retrieved data.

Tool discipline includes recovery, not just avoiding mistakes. The trained model succeeds despite an initial SQL error because it treats the environment’s response as information.

Once the two revenue values have been retrieved, the calculation itself is small. A parameterized SQL expression for the same 2023–2024 question is:

SELECT
    100.0 * (:revenue_2024 - :revenue_2023)
    / NULLIF(:revenue_2023, 0) AS yoy_growth_percent;

The parameters represent the retrieved revenues in consistent units; NULLIF avoids dividing by a zero baseline. This separates the ordinary arithmetic from the behavior that made the demonstration succeed: obtaining the right values through tools.

The ablation study compares three ways to select training tasks:

Crawford reports that single-table-only training produces the greatest uplift among these regimes. Adding harder tasks, either immediately or progressively, does not produce the best result in this experiment.

The benefit also transfers to the harder, multi-table FinQA Reasoning benchmark. The displayed results give the comparison explicitly:

The trained model nearly doubles its baseline success rate on these harder questions, even though the winning training regime uses only single-table tasks.

Crawford interprets the transfer as evidence that the core bottleneck was tool use. Discovering tables, inspecting schemas, and recovering from errors remain useful when a question spans multiple tables. Single-table training does not mean a single tool call—the successful revenue example already requires a sequence of interactions. Fixing a shared failure mode can improve tasks beyond those used to teach it.

Finding the specific failure is the next engineering problem. A final correct-or-incorrect score tells you whether the agent succeeded, but it does not explain where the process broke. Crawford proposes richer evaluation rubrics: decompose a response into individually answerable questions, then inspect which behaviors fail. For the revenue example, useful questions would include whether the agent discovered tables, inspected the schema, responded to a query error, and grounded its answer in retrieved values. These checks distinguish a retrieval failure from an arithmetic failure.

That richer feedback guides dataset selection and generation. If the failure is schema discovery, training data should create opportunities to learn schema discovery; making the financial reasoning harder may miss the problem. The optimization loop still receives a single reward value in Crawford’s GRPO setup. Rubrics diagnose what to train; the scalar reward drives the training cycle. Keeping those roles separate lets evaluation provide more information than the optimizer directly consumes.

The closing direction is to solve the specific problem responsible for poor performance. Crawford points to Snorkel’s companion study for more detail and to the Agentica partner post linked from it. The recording ends without a Q&A because time has run out; he offers to continue the discussion outside.