Doug Guthrie is a solutions engineer and open-source developer whose work spans financial-data software, analytics infrastructure, and AI application evaluation. He created yahooquery, built developer tools around dbt, and joined Braintrust in 2025 to help teams measure and improve AI products.
Earlier in his career, Guthrie worked in financial services as a data analytics manager. Building an internal application that required market information inspired yahooquery, an open-source Python interface to unofficial Yahoo Finance endpoints. First released in 2019, the library supports multiple ticker symbols, asynchronous requests, and pandas DataFrames; Guthrie described its origins in his article about accessing Yahoo Finance data.
He later worked at dbt Labs as a senior solutions architect. His dbtc project provides a Python interface to dbt Cloud’s REST and GraphQL metadata APIs, while his dbt Semantic Layer voice agent combines conversational queries, semantic search, and visualizations to make governed business metrics accessible through voice or text.
Guthrie had recently joined Braintrust when he appeared at the 2025 AI Engineer World’s Fair; the 2026 event also identified him as a Braintrust solutions engineer. His introduction to practical AI evaluations outlines an engineering approach to unreliable model outputs:
Start with a usable evaluation baseline. Define the task, assemble representative examples, and select scorers; improve the dataset and application together instead of waiting for a perfect test suite.
Use targeted, calibrated scoring. Separate deterministic checks from LLM-as-a-judge scoring, evaluate accuracy and formatting independently, and compare automated judgments with human assessments. Stronger models can assess outputs from cheaper production models.
Inspect individual workflow steps.Span-level evaluation can expose a misinterpreted question, failed tool call, or weak retrieval result before the error contaminates an entire agentic workflow.
Connect production behavior to future tests. Turn low-scoring traces, user feedback, and human-reviewed edge cases into new evaluation examples, while tracking model costs, latency, and regressions through development and deployment.
Guthrie’s projects follow a consistent practical orientation: give developers clearer interfaces to complicated systems and the evidence needed to improve them.
Build evaluations from tasks, datasets, and focused scorers, then use experiments, production traces, and human review to improve the same application.
A changelog generator provides a practical path through prompt comparisons, code-defined evals, production tracing, and human feedback that improves both the application and its judges.
How do you know a change makes an AI application better?
Notion’s evaluation workflow and a Braintrust changelog workshop show how to turn product expectations, production traces, and human feedback into repeatable tests.
Doug Guthrie connects tracing, online scoring and topic discovery to a practical improvement loop: find support-agent failures, turn them into evaluation cases, and give a coding agent enough context to propose reviewable changes.
Traces supply the intermediate evidence; scorers and topic labels make relevant failures easier to find within a large stream.
A coding assistant needs repository access plus selected trace evidence to propose concrete changes. The CLI and improvement skill connect those two kinds of context.
Automation can package code changes, reasons, regression cases and evaluation evidence into a pull request; a reviewer still decides whether the proposal should ship.
