How to build world-class AI products — Sarah Sachs (Notion) and Carlos Esteban (Braintrust)

Your prompt finally works on the train to work, or produces a convincing answer in a demo for a VP. What tells you it will work consistently for the people using the product? Evaluation and observability turn that isolated success into an engineering process. Sarah Sachs estimates that Notion’s team spends 10% of its time prompting and 90% evaluating, improving evaluations, and inspecting usage in Braintrust. The work extends beyond engineers running experiments: technical leads, leadership, data labelers, and specialists all inspect the same system.

Notion’s connected workspace combines documents, project management, and asynchronous work with tools such as Slack, Jira, and Google Drive. Sachs reports more than 100 million Notion users overall—not 100 million active AI users. Trials for almost all its AI products mean a feature’s potential audience extends well beyond paying AI customers. That creates a difficult combination: broad availability, the polish expected of a design-driven product, and rapid adoption of new foundation models alongside Notion’s own fine-tuned models. Sachs says new models usually reach production users in less than a day. Evaluations provide the evidence needed to move at that pace.

The AI product suite shown next makes the scope concrete. Meeting notes combine speech-to-text transcription with generated summaries; task-database integration and workspace-aware action items are described as forthcoming. Enterprise search helps recover context from a long Slack thread without reading every message. Deep research goes further, executing parallel searches and using reasoning to choose tools and spend more time on a task. That is a transition from a fixed sequence of steps toward an agent deciding which steps to take.

Sachs traces Notion AI’s beginnings to before ChatGPT, starting with an inline AI Writer that generated text on a page. Database Autofill expanded the unit of work: AI could act across database properties, run frequently, and translate content into different languages across columns. That produced usage the team had not anticipated and coincided with building a dedicated AI team. Retrieval-augmented Q&A then brought the data platform into the work. Offering it to free users meant maintaining embeddings broadly and handling multilingual workspaces. Cross-app search and attachment search followed, before the newest suite. Each layer used what the models could reliably do at the time.

Because Notion employees use Notion to build Notion, internal usage supplies abundant training and evaluation examples. The bottleneck became understanding them. Human specialists were looking through prompt dumps in Google Sheets, writing formulas to recover what the user had actually asked, and trying to inspect few-shot examples inside that representation. More rows did not solve that problem. The team needed a way to extract high-quality labels and insights, manage large datasets, and retain development feedback such as thumbs-down responses.

Suppose universal search is gaining a Jira connector. The evaluation problem starts with product semantics: can the AI recognize a task, understand a sprint, and construct a useful query against the workspace? Notion’s data specialists build handcrafted examples from logs and prototypes. Their role combines product management, analysis, and annotation; on a smaller team, the people closest to the data may be a PM or an engineer.

The iteration procedure is deliberately small at first:

1. Curate ten examples. Check that their structure contains the information the task and evaluators will need before generating more.
2. Inspect the data before choosing scores. Define product-specific success criteria instead of selecting a generic scorer first.
3. Use deterministic checks where possible. For the targeted Jira cases, inspect whether the tool query actually specifies Jira.
4. Run the evaluations, inspect failures, and revise. The dataset, scorer, and application all remain open to improvement.

This avoids a costly failure mode Sachs has encountered: creating a large amount of dummy data in the wrong format.

Multilingual behavior needs its own targeted examples. A Toyota workspace may contain Japanese and English, and a Japanese request does not by itself settle which language the requested paragraph should use. The product team must decide the intended behavior, encode it in a curated dataset, and check that a change preserves language switching. Notion runs these evaluations before shipping relevant changes as a form of ad hoc CI. At the time of the talk, it does not run them on every CI submission: evaluation duration and the volume of code changes make that too expensive for its existing pipeline.

Inspection also changes the product specification. Internal thumbs-down logs for research mode showed that people often wanted to draft reports, rather than conduct research alone. PMs and designers could see that behavior directly in Braintrust. The same trace that helps an engineer find a failure can serve as user research into what someone expected the feature to accomplish.

An LLM judge can apply one criterion—such as concision or faithfulness—to every row in a dataset. Notion also uses a more laborious approach: write a particular evaluation prompt for an individual example. A rubric might require Japanese, a specific bullet format, coverage of specified topics, and a reference to page A. Those requirements express acceptable behavior more directly than character-level similarity to a golden answer. Levenshtein distance cannot tell whether a differently worded response satisfies the user’s needs, while one exact reference answer can reject useful variation.

Rules also help when the underlying content changes. For an offsite search, the expectation can be that the first result is the most recent document about the Q1 offsite. A new document may appear after the example was created. A fixed document ID would then encode stale expectations, whereas the recency rule can remain useful. The evaluator still needs enough context to assess the rule, but the dataset no longer depends on one historical answer remaining correct forever.

This evaluation structure supports selective model replacement. Notion has different prompts for different tasks and different models behind those prompts. When Nano becomes available, Sachs describes gathering high-frequency tasks that need speed and low cost more than advanced reasoning, running their evaluations, and switching only the prompts where the candidate performs well. The evaluation assets and model-selection logic are part of the product’s intellectual property, alongside prompts and application code.

The stakes are particularly clear for language coverage. Sachs reports that 60% of Notion Enterprise users are non-English-speaking, while all Notion AI engineers speak English. Rigorous multilingual evaluation lets the team inspect experiences its own everyday usage would otherwise underrepresent.

The Q&A clarifies that a unique judge for one thumbs-down example can coexist with judges that cover an entire dataset. Notion has built layers over the Braintrust SDK to help specialists author these prompts. A dataset row also need not represent a complete interaction: the team can extract one call from a sequence of five LLM calls and evaluate that stage alone. Automated prompt optimization has been tried, but Sachs has not found that it solves most of Notion’s problems, even though she has heard other teams report success.

Historical feedback and current evaluation scores answer different questions. Some alpha users authorize evaluation access through submitting feedback. A thumbs-up is not a consistent quality label, although internal positive examples sometimes seed fine-tuning data. A thumbs-down from September 2023 identifies a request the system handled poorly then; it does not prescribe the score a new implementation should receive today. The durable asset is the request and available workspace context. The application, prompt, model, and resulting answer can all change when that case is rerun.

Sachs believes Notion’s judge scores use a scale that is not particularly well calibrated. In practice, the team collects cases below a threshold and treats them as failures for human inspection. With thousands of samples, another LLM summarizes failure themes and differences between experiments into a report for the engineer. More elaborate calibration has not been a priority for this workflow. Pairwise comparisons are useful when choosing between methods or protecting a production control; they are less common during exploratory development or alpha testing, where the desired experience is still being discovered.

A passing score only covers the criteria the judge was asked to check. Notion initially used graders to catch regressions in its special Markdown format, bullet usage, and topic coverage. Those graders could approve an answer that had switched to the wrong language. As thousands of criteria accumulate, they can acquire authority without anyone examining what they miss. Sachs recommends either separating tasks such as formatting and language following, or keeping a small set of judges whose failures the team commits to reviewing.

A question such as “When is our next offsite?” exposes a second source of evaluation instability: the search index changes. Index freshness and the date of evaluation can alter the answer before generation begins. Freezing an entire index is expensive, especially when the relevant documents are not known in advance. Notion instead separates retrieval evaluation from downstream generation evaluation.

Retrieval has its own evaluation framework around vector databases and Elasticsearch. For generation, the task assumes the captured results were retrieved and executes the remaining steps. If a sample cannot be evaluated because the answer was absent from the index, the team drops it. Reconstructing historical indexes also raises privacy problems: an object’s permissions may have changed. Heavy internal usage helps supply fresh, natural cases without repeatedly reconstructing old workspace states.

Managing prompts becomes an operational problem during provider outages. Notion has hundreds of prompts, including model-specific variants. Moving all GPT-4o mini traffic to Claude Sonnet 3.7 is not a uniform fallback strategy: an expensive replacement might be acceptable for one feature and untenable for frequent database operations. For database Autofill, Sachs describes a fallback that could cost 12 times more and create millions of dollars in additional expense during an outage. That is her workload-specific example; the traffic, duration, and pricing assumptions are not specified.

The practical solution is ownership. Each product or prompt owner preselects an acceptable fallback in code or configuration so the on-call engineer can switch traffic without rediscovering the trade-offs during an incident. Retrofitting that configuration required gathering everyone who had written a prompt. Sachs wishes the team had established it from the beginning.

Carlos Esteban and Doug Guthrie turn the case study into a workshop: first the playground, then the TypeScript SDK, production logging, human feedback, and finally remote evaluations. The questions remain product questions: did a change cause a regression, is a cheaper model sufficient, does the output match the brand, and can failures be diagnosed? Prompt versioning, automated evaluations, and production observability give technical and nontechnical collaborators a common place to answer them.

An evaluation is a structured test with three components:

• Task: the prompt or code being tested, from a single model call to a complete agent workflow.
• Dataset: the examples supplied to the task.
• Scorer: the logic that assesses each output against a chosen criterion.

The workshop uses normalized scores from 0 to 1, displayed as percentages. Offline evaluations run predefined examples during development. Online evaluations score live application outputs, helping identify failures and capture cases for the next offline dataset.

Before optimizing the application, compare the score with your own inspection:

Agreement makes a score useful. Disagreement is evidence that the measurement itself needs attention, rather than an instruction to optimize blindly toward it.

In the playground, a task can combine a system prompt with Mustache-templated user input. Extra messages package a conversation—including assistant responses and tool calls—into one evaluation context. Tools pushed to the Braintrust library become available to playground users and experiments. The agent feature shown in the recording chains prompts so one output becomes the next input; it is described as beta, with branching still forthcoming.

An expected value need not be a fully written answer: for RAG, it can specify facts the output should include. Start with five or ten rows, then improve them using logs, internal testing, synthetic examples, and expert review. In medicine or law, establishing ground truth may require a specialist rather than a general reviewer.

Use code for exact conditions and LLM judges for qualitative interpretation. A deterministic formatting check and a completeness judge can assess different aspects of the same answer. Carlos recommends trying a stronger model to judge a smaller one, keeping each judge focused, and testing the judge against human annotations with the same care used for the application prompt.

The playground is for quick iteration; Experiments preserves a durable history. Save a playground configuration as an experiment when the comparison matters. Evaluations run through the SDK or CI also appear there, allowing comparisons across prompt versions, models, and weeks of development.

The workshop application, Unreleased AI, takes a GitHub repository and summarizes commits since its latest release, helping developers understand what might arrive next. The demonstrated setup uses OpenAI, a Braintrust organization, and a project named Unreleased AI. A project can correspond to a product feature. For this checkout, Doug recommends Node 20 or 22 and warns against Node 23; Git and TypeScript are also prerequisites. Other cloud or custom model providers can be configured.

After cloning the example, the setup sequence is:

1. Copy .env.local.example to .env.local.
2. Add the Braintrust and OpenAI API keys. An optional GitHub token helps avoid GitHub rate limits.
3. Run pnpm install.
4. Inspect the prompts, scorers, and dataset created by the post-install resource push.

The public repository’s current main branch differs from the workshop checkout, including its prompt setup and package scripts. The recording’s two-prompt exercise should therefore be followed as the demonstrated configuration, rather than assumed to match today’s checkout exactly.

The prompt’s Mustache variables map to the dataset’s structure. The completeness judge checks whether the summary covers breaking changes and new features, with rubric categories mapped to normalized scores. A separate formatting heuristic returns a binary result. That separation lets the team distinguish missing content from an answer that contains the right information in the wrong structure.

A small TypeScript formatting scorer can make that boundary explicit. For a changelog format requiring an executive summary followed by categorized changes, the check can enforce the headings and their order without attempting to judge completeness:

type ChangelogResult = { output: string };

export function ChangelogFormat({ output }: ChangelogResult) {
  const headings = [...output.matchAll(/^#{1,6}\s+(.+?)\s*$/gm)]
    .map((match) => match[1].trim().toLowerCase());

  const summary = headings.indexOf("executive summary");
  const changes = headings.indexOf("categorized changes");

  return {
    name: "ChangelogFormat",
    score: summary >= 0 && changes > summary ? 1 : 0,
  };
}

These heading requirements are an illustrative contract for the workshop’s formatting-check pattern. A separate judge must still determine whether the commit summary contains the important changes.

Doug loads the two changelog prompts, selects the dataset and scorers, and runs the evaluation. Rows execute in parallel. The useful next step is inspection: open a judge’s rationale, compare output diffs, and examine how the base task compares with the alternative on accuracy and other criteria. Saving the setup as an experiment preserves the comparison. Grouping experiments by model then helps examine quality alongside cost. Substituting GPT-4.1 illustrates that decision process; no numeric quality-versus-cost result is reported.

The next demonstration previews an assistant that uses the playground configuration and previous evaluation results to propose prompt changes or new dataset rows. Its interface presents a diff for acceptance, then reruns the scores to assess the revised task. Selected failing rows can focus the requested improvement. Doug describes the feature as something the team hopes to release soon and says Braintrust uses its own evaluations to assess it. The proposed edit and the measured result remain separate steps: generating a plausible prompt revision is not evidence that the revision improves the task.

Custom models can be added through provider settings when they expose an OpenAI-compatible interface. Custom objectives need more thought. An audience member asks how to score a trading system whose profit has no natural upper bound. The presenters do not supply a general transformation from unbounded profit to a normalized score; they leave the choice of floor and ceiling to the application developer. The score’s meaning must come from the task, not from the fact that the interface displays a percentage.

A long conversation does not gain unlimited memory merely by being placed in the extra-messages field. In an agent chain, each prompt makes its own call. These are different execution structures, even if both ultimately produce one user-facing answer.

Doug illustrates span-level diagnosis with a pipeline that first combines chat history and the latest message to rephrase the user’s question, then determines intent. If the rephraser fails, later stages may faithfully process the wrong question. Scoring individual spans makes it possible to identify that earlier failure instead of treating every bad final answer as a generation problem. For voice evaluation, Carlos points to the voice-agent cookbook; the workshop does not demonstrate a real-time speech-to-speech evaluation.

Deployment boundaries matter too. Carlos describes a hybrid model in which Braintrust manages the control plane and the customer runs the data plane in its own VPC, with the two meeting in the browser. Asked about LangSmith, he cites customer feedback about the interface and attributes Braintrust’s performance and scale to its in-house Brainstore database, full-text search, and the founding engineers’ SingleStore/MemSQL experience. These are vendor positioning claims; the workshop supplies no comparative test against LangSmith. The exercises are also presented as available for later practice with a free tier.

The SDK connects code to Braintrust in two distinct ways. Pushing code-defined assets makes prompts, scorers, and tools available in the library. Running an evaluation creates an experiment. The latter can happen locally or in CI around a pull request; its results can also feed a custom reporting process, such as Notion’s LLM-generated summaries of failures.

Assets are defined with APIs such as project.prompts.create and project.scores.create. Teams can push source-controlled prompt definitions into Braintrust or pull prompts into their application, depending on where they manage them. In either case, versioning makes changes inspectable. The SDK evaluation retains the same three-part structure as the playground: dataset, task, and scorers.

In the demonstrated repository, resources.ts holds prompts and scores, while changelog.eval.ts defines the evaluation using Eval from the Braintrust SDK. Naming files with .eval.ts also supports discovery when passing a directory to the CLI. Doug runs the repository’s pnpm eval command, which invokes braintrust eval for the evaluation file. The terminal returns an experiment link, and the results enter the same history used by playground experiments. Model, duration, completion-token, and score views help inspect the consequences of a change.

The SDK Q&A establishes several support boundaries for the recording: image evaluation is said to work in the UI, while support for a non-LLM grounding model is left unresolved. Python and TypeScript are the main SDKs discussed. Java, Kotlin, and Ruby are described as Stainless-generated API wrappers, and a Go SDK as work in progress.

Production logging makes quality changes visible, supports alerts, and lets engineers inspect the functions and model calls behind a response. It also supplies new evaluation cases. Capturing feedback without changing prompts or adding relevant examples to datasets leaves that loop unfinished.

Instrumentation has several levels:

• Wrap the model client. wrapOpenAI, or the corresponding Vercel AI SDK wrapper, captures model-call metrics such as tokens, latency, and cost.
• Trace application functions. wrapTraced and the tracing interfaces expose work beyond the model call itself.
• Record task-specific context. span.log adds inputs, outputs, and metadata needed for diagnosis and scoring; OpenTelemetry integration offers another instrumentation path.

Logger initialization authenticates the application and selects the destination project. The workshop’s generate/route.ts ties these pieces together.

Online scoring reuses offline scorers on production traces. Low-scoring cases can then be inspected and promoted into a dataset. Tags support comparisons between live variants, while sampling controls how much traffic is scored. Carlos recommends starting with a low sampling rate and increasing it as the team gains confidence in the metrics. Saved views preserve filters and sorting so each team can return directly to the cases it needs to inspect.

Doug starts the application with pnpm dev at localhost:3000 and submits a GitHub repository. The app generates a categorized summary of commits. In the code, wrapAISDKModel provides model metrics, while explicit tracing defines meaningful inputs, outputs, and metadata. That extra structure matters because the scorer needs to receive the same kind of data it was designed to evaluate.

The resulting log contains a top-level Generate request trace with the repository input and generated output. Nested spans expose operations such as obtaining commits and the latest release. The selected frame shows the trace beside an output headed Executive Summary and Categorized Changes. This is the point where the application’s final answer becomes inspectable together with the work that produced it.

To add online scoring, Doug opens Configuration → Online scoring, creates a rule, and selects the scorers and target. The demo applies the selected scorers to the overall trace at 100% sampling. A subsequent request produces the same trace structure with scores and judge rationales attached. He also describes an automation that could react when a score falls below a chosen threshold.

The next step is a saved view filtering for completeness below 50%. That threshold defines an inspection queue, not a reported failure rate. Reviewers can open the relevant inputs and outputs, then add useful spans to an existing or new dataset. Production monitoring now feeds the development process with cases selected for a reason.

Aggregate scores are available as an overall progress indicator. For SaaS online scoring, the scorers must be accessible in Braintrust because Braintrust runs the grading compute. Publishing assets also lets playground collaborators use prompts and tools authored in code, but a code-only evaluation workflow does not require publishing every asset. Built-in evaluators from Autoevals provide another starting point for playground and online scoring, with less customization than a task-specific scorer.

Human review and user feedback serve different purposes. In healthcare, finance, or legal applications, specialists can identify consequential errors, establish expected answers, and audit edge cases. Their labels also provide a reference for evaluating the LLM judge. User feedback instead reveals how the product is serving people in practice: repeated thumbs-down responses around an edge case can justify a targeted dataset and another round of prompt work.

In api/feedback/route.ts, the application uses logger.logFeedback to associate a reaction with the relevant span. The user_feedback score is binary, with an optional comment and metadata. A TypeScript helper following that pattern keeps the span ID explicit:

import { initLogger } from "braintrust";

const logger = initLogger({ projectName: "Unreleased AI" });

export function recordUserFeedback(
  spanId: string,
  helpful: boolean,
  comment?: string,
) {
  logger.logFeedback({
    id: spanId,
    scores: { user_feedback: helpful ? 1 : 0 },
    ...(comment ? { comment } : {}),
    metadata: { source: "changelog-feedback" },
  });
}

The application must retain the ID of the response being reviewed so a later reaction attaches to the right evidence. Metadata then supports filters and saved views.

Doug submits feedback through the app’s thumbs controls and comment field, then returns to Braintrust to inspect the comment and user_feedback column. Those fields can drive a review queue, where an annotator decides whether the case belongs in a dataset. The feedback does not automatically become ground truth merely because it was submitted.

Human-review configuration supports options, sliders, and free-form text. Reviewers receive a simplified interface showing the span’s inputs and outputs alongside the configured questions, rather than needing to navigate the full trace interface. They can score a case and promote it into a dataset. Doug reports that the demonstration began with seven dataset rows and added two through the feedback and review process. Those additions are then available for subsequent offline evaluations.

Reviewers are added to the relevant project, and comments or tags on datasets, experiments, or traces direct them to the work. The application team defines the review form in advance: a completeness judgment, a thumbs response, or a text field for an ideal answer. Assignment and form design make the human contribution part of a repeatable process.

A final versioning question exposes a distinction between prompts and tools. Carlos demonstrates saving a new prompt version; adding a tool to a prompt also changes that prompt’s version. The audience asks about independently versioned tools and mixing tool versions with prompt versions during evaluation. The project has no tool with which to demonstrate that case, and Carlos’s answer remains tentative. The demonstrated prompt history should not be taken as proof of the requested independent tool-version behavior.

The final activity addresses a limit of the playground shown in the recording. A prompt chain can connect model calls, but it cannot represent every intermediate code step or directly access every private service an application uses. Remote evaluations let the team retain that execution environment while exposing an evaluation interface to PMs, subject-matter experts, and other playground users. Custom tools, changing research code, and complex scorers can stay with the application instead of being continually pushed as separate assets.

The execution sequence is:

1. The playground initiates an evaluation request to the configured server.
2. That server runs the task, including intermediate code and service calls.
3. The scorer logic runs in the same environment.
4. Outputs, scores, and metadata return to populate the playground.

The demonstrated CLI mechanism is braintrust eval pointed at an evaluation file with --dev. It starts a development server; localhost and configurable network binding are discussed. Current documentation also covers newer sandbox execution, which is separate from the local-server workflow shown here.

Doug’s remote changelog task exposes parameters for detail level, whether to include authors, and target audience. He starts the server using a repository-specific package command; the selected frame shows the development server running at localhost:8300 below the parameter definitions. The stable mechanism to recognize is the underlying evaluation command with --dev, rather than a package-script name that may differ between checkouts.

Back in the playground, Doug selects the remote changelog generator and opens its exposed parameters. The complex implementation remains in the codebase, while the UI provides a place to adjust the task and configure scoring. He selects scores and switches to the grid layout to inspect outputs; the demonstration establishes the connection and configuration workflow without reporting a measured improvement.

Carlos connects this to a possible next step for Notion. Its evaluations depend on many services and tools, so the team runs them through the SDK. A remote task could let someone isolate one failing row and rerun it in the playground while retaining those dependencies. He proposes this as a way to reduce repeated work and cost, and as a feature to discuss with Notion—not a workflow Notion has already adopted or a measured savings result.

The last audience question is operational: must the remote server have a public IP? Carlos answers with public reachability or localhost, and Doug notes that an unreachable private endpoint would not work. That exchange does not establish the request’s origin, authentication requirements, or a blanket need to expose an evaluation server publicly. Before using the pattern beyond a local development session, the team must establish which component connects to the endpoint and how access is controlled. The useful boundary is clear: make the experiment accessible to collaborators while keeping execution in an environment that can safely reach the task’s dependencies.