Does GenAI "belong" to data scientists?

Does agent development belong to data scientists and machine learning engineers? The practical question is who can turn an experiment into a working product. Phil Hetzel introduces himself as the leader of Braintrust’s solutions engineering team, with twelve years in consulting and systems implementation behind him. While leading Slalom’s global Databricks business unit, he repeatedly saw customers produce generative AI proofs of concept but struggle to bring them into production. That gap led him to use Braintrust and, eventually, join the company.

The quality problem has two connected parts. In Braintrust’s evaluation framing, evals help a team assess changes during experimentation and build confidence before release. Observability helps maintain that confidence once real users confront the agent with real usage. These are complementary activities, not a rule that evaluation stops at deployment. A convincing prototype still needs evidence that its behavior holds up in the product.

Hetzel describes two organizational patterns he has observed. In a traditional enterprise, an executive decides the company needs agents. The mandate passes down to an existing ML or data science platform team: that team already has tooling, and generative AI has “AI” in its name. The assignment feels natural because it follows the existing organization.

Smaller AI-native companies often start from a different position. Their offering was built around agents, so they may have no inherited AI platform boundary. A small engineering team spans product and AI development, and individual builders remain close to the problem the agent is supposed to solve. These are observed patterns, not universal categories, but they expose an important distinction: owning the relevant technology does not necessarily mean understanding the intended outcome.

Integrating a pretrained LLM changes what the application team must build. Much traditional ML work revolves around a data pipeline: prepare data, train and test a model, control overfitting, and deploy the result for a downstream product team. Natural-language inputs introduce another way to add value, with a different set of contributors able to influence behavior.

For a hosted LLM, providers such as Anthropic, OpenAI, and Mistral have already developed the underlying model and exposed it through an endpoint. Their testing does not establish that a particular application works. The application team still needs product-specific evals after integrating the API. The responsibility moves from reproducing the foundation-model pipeline to assessing what the product does with the model.

The available ways to change behavior also shift:

Prompts and context can change behavior without fine-tuning. Hetzel characterizes fine-tuning as relatively rare in the work he is discussing, rather than unnecessary in every application. This makes proximity to users especially valuable: someone who understands the task can improve the information and instructions supplied to the model.

There is a strong case for giving data scientists responsibility for agents. Agents use models, and many organizations already place model governance with data science. Knowledge of neural networks supports an appreciation of the risks. Established release processes and a rigorous testing mindset can protect the company and its users from poorly understood behavior.

But the application team is not necessarily repeating model training and cross-validation. The harder question becomes whether it knows what to test for. Hetzel has seen teams fixate on precision, recall, and F1 because those metrics served them well in traditional ML. An agent has a broader surface of behavior: evaluation must address its functional performance in the task, not only familiar technical measures. Knowing how to measure a model is different from knowing whether an agent accomplishes its purpose.

The case for involving other disciplines starts with the application boundary. Product engineers routinely send payloads to APIs, retrieve information, and turn responses into useful behavior for users. Consuming an LLM endpoint fits that part of their work.

A complex agent can also be a distributed system. A supervisor may call child agents running on different infrastructure, and those children may call different systems in turn. Making that arrangement work is a systems engineering problem as well as a model problem. Statistical expertise alone does not cover the full execution environment.

Subject matter experts and product managers contribute another essential skill: they understand what success should look like. They can shape the prompts supplied to the agent and review its behavior. In a human annotation workflow, a domain expert examines an agent trace, judges whether it performed well, and explains why. That explanation makes the judgment useful for improving the product, rather than merely recording approval or disapproval.

Broadening the team does not require data scientists to discard their skills or step away from implementation. They can contribute directly to the product while working with other technical and nontechnical specialists. The point is to bring more relevant expertise into development.

Beyond implementation, Hetzel identifies three contributions:

• Technical risk guidance. Statistical knowledge can temper overconfidence in LLM behavior. His reminder that the model predicts tokens is a caution against treating fluent output as evidence of understanding or reliability.
• Validating LLM judges. A judge is itself a prompt-and-model system. It should not be trusted merely because it returns an evaluation. A labeled dataset provides a basis for checking its judgments with measures such as precision, recall, and F1.
• Use-case-specific fine-tuning. When an application needs a fine-tuned open-source model, data scientists and ML engineers can contribute directly to that specialized work.

The judge example gives familiar classification metrics a specific target: assessing the evaluator against labeled examples. That is a narrower, appropriate use of those metrics, distinct from treating them as a complete description of agent quality.

Product, application, and systems engineers turn domain requirements into working behavior. They also make sure the infrastructure on which agents execute supports a good user experience. Together with data scientists, they can implement evaluation and observability pipelines that connect what happens in production with what the team tests during experimentation.

Nontechnical experts need an active role in that process through human annotation, prompt development, and context engineering. Their proximity to the problem helps keep the agent relevant to its intended use. The proposed team therefore preserves data science expertise while giving other contributors meaningful influence over both behavior and assessment.

The first audience question challenges the ownership framing itself. If agents are tools, anyone in the organization might build or own one, especially when that person has the relevant domain expertise. Why begin with which team owns the technology instead of the problem it is meant to solve?

Hetzel agrees: the agent is a product built by a diverse team. The organizational mistake he is criticizing is treating it as another predictive-model project and isolating it within ML engineering or data science. Starting from the problem changes whom the team needs to involve.

The second audience exchange asks how tooling can help domain experts close the improvement loop and update the system. Hetzel initially points to Braintrust’s human labeling tools and agent and prompt playground: interfaces where contributors can provide judgments, experiment with prompts, and send them to the underlying agents.

The question then becomes more precise: how does a team keep both the evaluator and the agent up to date, and determine whether an error comes from the evaluator or the system being evaluated? Improving one does not automatically establish the correctness of the other.

Hetzel proposes continually gathering production examples and adding them to the offline evaluation dataset. Along the way, the team needs grounded judgments against which it can check its evals. The test is whether those evaluations increasingly agree with human judgment—or begin to diverge. This is a feedback process for checking and recalibrating evaluation, not an automatic method for attributing every error. Production data supplies new cases; human agreement remains something to examine, not something the loop guarantees.