Zubin Aysola

Zubin Aysola is an AI engineer at Weights & Biases who helps build CoreWeave ARIA, a research agent that connects experiment analysis with the next experiment. His work centers on making that loop measurable: reproducing user interactions, comparing agent variants, and checking whether improvements in offline tests carry into production.

From training data to model behavior

At the University of Michigan, Aysola co-authored RedCaps with Karan Desai, Gaurav Kaul, and Justin Johnson. Released in 2021, the project assembled more than 12 million image–text pairs from Reddit for vision and language research. Its organizing insight was that online communities provide useful structure: selecting particular subreddits helps shape a dataset’s subject matter, while captions written by participants supply language grounded in the images they share. The RedCaps paper explains how community selection can steer dataset composition without annotating every image individually.

Aysola subsequently studied artificial intelligence engineering at Carnegie Mellon, completing his master’s degree in fall 2024. His research included language-model alignment, evaluation, and multi-agent systems, with work alongside Maarten Sap.

Aysola and Joel Mire were equal-contribution authors of a 2025 study of dialect bias in reward models, co-authored with Daniel Chechelnitsky, Nicholas Deas, Chrysoula Zerva, and Sap. Reward models help determine which language-model responses should be preferred during alignment. The team examined whether those judgments treated African American Language differently from White Mainstream English, using paired texts and human preferences. Across the models and experiments studied, they found weaker agreement with human preferences for African American Language, a tendency to favor White Mainstream English responses, and conversational steering toward that dialect even when the input used African American Language.

The study identifies a concrete problem in alignment: a system intended to reward helpful responses can also reward conformity to a dominant language variety. In his agent work, Aysola examines a different question about measurement: whether the tests used to improve a system represent the behavior people experience.

Making agent improvements measurable

At Weights & Biases, Aysola helps build ARIA and develops its offline evaluation framework. ARIA works with experiment histories, metrics, training code, and artifacts. It can form hypotheses, configure experiments, launch runs through W&B Launch with researcher approval, compare results with baselines, and create reports. These capabilities support the repeated work between noticing a problem and deciding what to try next.

Aysola applies that research loop to the agent itself. His approach to agent evaluation addresses a moving target: benchmarks, evaluations, agent code, and configurations can all change together. His team uses a model-agnostic harness and YAML-defined configurations to run variants in parallel, while keeping measurement tied to the deployed system. Three choices define this work:

• Reproducing production behavior: Research and production environments run identical agent code and record behavior in the same format through Weave. Production traces can therefore become reproducible test cases. Failures supply regression tests; successful interactions provide behavior worth preserving. Keeping the implementations aligned reduces the risk of improving a laboratory version that behaves differently from the deployed product.

• Comparing how agents work: Pass/fail scores establish whether an agent accomplished a task. Relative comparisons address choices such as whether asking clarifying questions produces better behavior than proceeding immediately. At the time of his presentation, the team used 886 evaluation tasks, including simulated multi-turn user interactions. These tests cover conversations as well as isolated prompts.

• Automating experiments while retaining judgment: ARIA can inspect traces, create evaluation tasks, propose prompt or skill changes, and benchmark candidates against the production version. Aysola demonstrated this loop with improper weave.log SDK usage in a sandbox: ARIA turned a production trace into a regression task and tested a targeted instruction change. He sees this automation as a way to devote more attention to choosing useful improvements and designing guardrails. People remain responsible for deciding which behaviors matter and whether the evaluations measure them well.