Shipping complex AI applications | Braintrust & Trainline

How do you take an AI application that works on a laptop and make it dependable enough to serve customers? This hands-on workshop brings together Giran Moodley from Braintrust and Trainline engineers Oussama Hafferssas and Mayank Soni. Their backgrounds span mission-critical enterprise systems and mathematics, platform engineering and computer vision, and language-model research predating today’s large models, including BERT. The workshop moves from operational context into a working application, then through evaluation and remediation. Its public Slack channel and persistent cheat sheet support participants across software engineering, applied AI, ML and infrastructure.

A few successful demonstrations establish that an application can produce a plausible answer. They do not establish how it behaves across customer inputs, organizational policies or changing dependencies. Giran identifies a familiar cycle: ship a promising prototype, encounter a production failure, patch the prompt and wait for the next failure. Without a record of what changed and a way to test its consequences, each patch adds uncertainty. Logs help record events; observability must also expose the behavior and execution path that produced them.

The operational unit is a change you can inspect and evaluate. That means separating responsibilities, tracing their execution and checking whether a fix closes a gap without opening another. Giran borrows the monolith-to-microservices analogy: break a large prompt’s responsibilities into explicit stages. The goal is progressive coverage of meaningful failure modes, rather than a promise of complete coverage. An initial golden dataset gets the process started; real production cases keep it relevant.

Braintrust provides the shared environment for that workflow. Giran describes a company approaching its third anniversary, expanding in Europe with customers including Trainline, Lovable and Doctolib. He reports an $80 million Series B at an $800 million valuation; the funding announcement confirms the round and names ICONIQ, Andreessen Horowitz and Greylock, but does not state the valuation.

The platform’s rationale comes from Ankur Goyal’s experience building evaluation tooling at Impira and Figma before founding Braintrust. Production traces contain changing, semi-structured data: model requests, tool calls, nested operations and application-specific metadata. Giran presents Brainstore as storage designed around that workload, with Braintrust positioned to work across agent frameworks and model providers.

There are two entry points into the same loop. For a new application, create an evaluation set that expresses the behavior you need. For an existing application, instrument it and collect actual failures. In either case, identify a failure mode, remediate it, ship the change and monitor what happens next. Each pass should leave behind better evidence for the following change.

Rail travel crosses fragmented carrier systems. Trainline brings journeys such as Paris–London and Lyon–Milano into one booking application. Oussama describes a large customer base and ticket-sales operation: the financial figure is approximately £6.3 billion in FY2026 net ticket sales value, not a count of tickets or company revenue, as clarified by the pre-workshop trading update. The reported 27 million active customers are customers transacting during the preceding 12 months, a definition supplied in the subsequent annual results. The travel assistant is available from a ticket’s view in the app and, according to the presenters, can handle refunds and train changes.

Trainline builds both conventional ML models and agentic applications. Its disruption predictor resembles a weather application for trains: historical data supports predictions about delays and disruption. The travel assistant operates further along the customer journey, finding alternative trains, handling refunds and handing over to human support when needed. Those actions depend on the state of a journey and the rules attached to a ticket.

Agentic applications combine deterministic and probabilistic components, so Trainline combines the corresponding quality practices. Offline evaluation happens before release; online evaluation checks production behavior. For the disruption model, the relevant question is whether its prediction matches the train’s actual state.

Oussama describes Trainline as an unpaid, satisfied Braintrust customer. Its travel-assistant evaluations cover tone of voice and practical helpfulness, including ticket-specific reasoning. Refund eligibility can depend on lateness and whether the ticket is a return or an advance fare. A fluent response that ignores those distinctions is still a poor answer.

Cost and token consumption create pressure to change models. Mayank describes evaluating a candidate offline against the current system, then checking online that the expected effect survives contact with production traffic. The same process gives the team confidence when releasing new assistant features. Tool-call and token-level traces help explain failures before and after deployment.

Shared access changes the collaboration model, too. Product managers and other nontechnical colleagues can inspect data and insights themselves rather than asking an engineer to download and send logs. Trainline uses these capabilities alongside its own systems, rather than treating Braintrust as its entire stack. The presenters close their case study with an invitation to join Trainline’s AI engineering team.

The hands-on environment uses TypeScript, Node and pnpm. To follow the workshop:

1. Obtain the repository and cheat sheet pinned in the public AI Engineer Europe 2026 Braintrust Workshop Slack channel.
2. Create a Braintrust account and obtain an OpenAI API key. Giran suggests plus-addressing for an isolated workshop account; staff offer session-specific keys to participants who cannot provision one.
3. Use mise for Node and pnpm, or install the workshop’s Node 22 runtime directly.
4. Use Make as the command wrapper, or inspect package.json and run the corresponding pnpm scripts when Make is unavailable.

The runtime is deliberately pinned for the workshop. Coding assistants are welcome for exploring the code, and the instructions remain available after the session.

The application is a fictitious support-triage system intended for teaching, not production deployment. A ticket enters a staged workflow that gathers context, decides how to classify the issue, checks policy, writes a reply and packages the result. There are five overall stages, with model calls concentrated in three specialist stages:

Severity can trigger a tool call for human escalation. The resulting information is structured for downstream systems, rather than stopping at a conversational answer.

Braintrust is then added around this application: first end-to-end tracing, then evaluations, then managed prompts and tools. The repository includes published slides and cumulative, runnable checkpoints. If a participant gets stuck, checking out a later checkpoint and running make install and make setup restores the corresponding stage without rebuilding every earlier step. This structure keeps the focus on operationalizing an agent: inspect it, establish a baseline, move shared assets into managed infrastructure and turn a failure into a tested fix.

The first checkpoint has one function, one model call and structured output. A system prompt supplies instructions, the user message supplies the ticket, and the application receives JSON. This is a useful baseline because it gives later changes something concrete to improve upon.

The implementation uses the OpenAI SDK directly, without an agent framework. Giran runs make ticket, enters a password-reset request and accepts the interactive defaults. The environment selects GPT-5 mini. The output looks plausible, but the single call has little basis for handling organization-specific exceptions. make demo exercises the same function using predefined JSON tickets. The demonstrated command path uses Make and pnpm; npm and Yarn are mentioned as untested alternatives.

The next checkpoint adds local tools for help-desk information, account context and escalation. An account migration, for example, might explain a backend problem and provide a reason to escalate it. In this teaching application, the tools are deterministic code checked into the repository. Real integrations could retrieve context through vector search, MCP or a CLI. Each added dependency also creates another possible failure point, making traces more valuable.

Giran repeats the interactive test with a password-reset request that also says the account is locked. The response becomes more detailed because the model now has tool-derived context. The checkpoints remain cumulative: moving ahead to a later tag includes these earlier capabilities.

Tool context improves the information available to the model. Specialist stages make its responsibilities easier to inspect. Context collection, triage, policy review, reply writing and finalization become separate functions with their own prompts where needed. The advantage is local diagnosis: a wrong classification, an inappropriate reply and a missing escalation no longer have to be debugged as one undifferentiated output.

The TypeScript application connects these functions asynchronously. A compact way to express the same dependency structure is to make every handoff explicit:

export type SupportStages<Ticket, Context, Triage, Policy, Reply, Result> = {
  collectContext(ticket: Ticket): Promise<Context>;
  triage(ticket: Ticket, context: Context): Promise<Triage>;
  reviewPolicy(context: Context, triage: Triage): Promise<Policy>;
  writeReply(
    ticket: Ticket,
    context: Context,
    triage: Triage,
    policy: Policy,
  ): Promise<Reply>;
  finalize(triage: Triage, policy: Policy, reply: Reply): Promise<Result>;
};

export async function runSupportWorkflow<T, C, D, P, R, O>(
  ticket: T,
  stages: SupportStages<T, C, D, P, R, O>,
): Promise<O> {
  const context = await stages.collectContext(ticket);
  const triage = await stages.triage(ticket, context);
  const policy = await stages.reviewPolicy(context, triage);
  const reply = await stages.writeReply(ticket, context, triage, policy);
  return stages.finalize(triage, policy, reply);
}

Each boundary is also a natural place to attach a trace span or evaluate an intermediate result.

The live example changes from account access to billing: a customer wants to upgrade from Pro to Enterprise, but the website returns HTTP 500 errors. Giran supplies customer tier two, the billing category and the designated account. Because the specialist model calls run sequentially, the response takes longer than the single-call baseline. The resulting triage treats the blocked upgrade as high priority because of its revenue impact, supplies an escalation reason and produces both internal and customer-facing text. It also includes a confidence score; tool facts from other systems provide additional evidence for the decision.

The final answer cannot explain which stage introduced a mistake or consumed most of the time. Instrumentation needs to capture the full execution path: stages, nested functions, model calls and tools. Useful fields include latency, cost, input and output token counts, time to first token, inputs, outputs and application metadata. Metadata makes traces filterable and supports targeted monitoring or alerts.

The workshop uses TypeScript, while Giran also discusses SDK use in Python, Go and Ruby, with tentative mention of .NET users. The essential design is language-independent: one parent span represents an interaction or conversation; child spans represent the work inside it. Recording each call separately loses the relationships needed to understand a multi-stage or multi-turn workflow.

The tracing checkpoint introduces a helper that wraps the provider SDK and creates parent and child spans around application operations. Python can express similar instrumentation with decorators. The stage structure already established in the application now becomes the structure visible in its trace, letting an investigator move from a failed result to the operation that produced it.

Connecting the application requires two credential paths. Generate a Braintrust API key from the profile settings and store it in .env or a key vault. Configure the OpenAI key under AI providers at organization or project scope; the latter lets teams or environments separate their configuration. That provider credential will be needed for managed execution and online scoring later in the workshop.

Running make setup and then make demo or make ticket sends executions into the workshop project’s Logs tab. Giran describes non-blocking submission and rapidly available reads through Brainstore, and reports customers submitting tens of millions of traces over short periods; he does not provide a timed throughput test.

The UI shows recent logs first. Opening a trace reveals the root invocation’s duration, token usage, cost and metadata. Expanding the triage specialist reaches its model input and output; later spans expose the escalation decision. The waterfall timeline answers a different question: which stage accounts for the delay? The tree explains dependencies, while the timeline makes sequential waiting and expensive stages visible.

A new application has a cold-start problem: there may be no production history from which to learn what goes wrong. Start by defining release criteria. For this support agent, the category should be correct, blocking issues should not receive low severity, escalation should respect policy and SLAs, and the output should have a valid structure. A change must improve its target behavior without regressing other cases.

An evaluation combines data, a task and scoring functions. The golden dataset supplies cases that represent important expected behaviors and edge conditions. Running the application against those cases creates a repeatable baseline, replacing anecdotal confidence with recorded results. It remains a starting set that can grow as failures are discovered.

Use deterministic checks when the requirement can be expressed directly. Reserve an LLM judge for criteria that need interpretation, such as whether a customer-facing reply follows a nuanced style rubric. This keeps simple requirements cheap and explicit while allowing more subjective qualities to be evaluated separately.

The next checkpoint runs make seed_dataset to upload the initial cases. The seeded dataset contains ten inputs, each accompanied by expected output and metadata. Categorization makes it possible to inspect related kinds of failures rather than relying only on an aggregate result.

The code includes deterministic checks for category, schema and the presence of an escalation reason when required, plus an LLM-based customer rubric. For example, the escalation-reason requirement can be expressed without a judge:

type SupportResult = {
  shouldEscalate: boolean;
  escalationReason?: string | null;
};

export function scoreEscalationReason(output: SupportResult): number {
  if (!output.shouldEscalate) return 1;
  return typeof output.escalationReason === "string" &&
    output.escalationReason.trim().length > 0
    ? 1
    : 0;
}

This checks whether a required reason is present. Whether the escalation decision itself is correct requires a separate criterion.

Running make eval evaluates the application against the dataset. The workshop uses flat-file JSON, though the inputs could also come from a database. Results appear in the terminal and in Experiments, where individual inputs, outputs and scores retain their nested execution traces. Subsequent experiments can therefore be compared both at the score level and at the level of the operations that produced a changed answer.

Local files are convenient while one engineer builds the application. Collaboration adds new requirements: a shared point of reference, version history, change authorship and reproducibility. A product manager or domain expert should be able to propose prompt changes without asking an engineer to perform every edit. Giran connects this need for traceable change to his experience in regulated banking environments, including data-deletion and system-exit concerns.

The managed checkpoint packages prompts, tools and scoring functions and uploads them to Braintrust. The application then refers to those managed assets while preserving its execution trace. Local mode is the workshop default; managed mode changes which execution path the commands use. After setup, the UI exposes the three specialist prompts. A prompt slug provides a stable code reference, and interpolation supplies runtime values. A stable slug does not freeze its content: prompt versioning treats the reference and the selected version separately.

A managed baseline-model parameter makes the collaboration concrete. Giran changes the model in the UI, saves a version with a comment and reruns setup to synchronize the assets. He then submits an account-unlock ticket in managed mode. The execution follows the managed model selection without an application-code edit, providing another configuration that can be evaluated against the baseline.

Managed editing still needs source-control discipline. Giran explicitly recommends retaining version control and a centralized representation of prompts, tools and parameters, with automation to synchronize them. Shared UI access solves collaboration friction; it does not remove the need to know which configuration is running or why it changed.

Offline cases establish an initial level of confidence. Online scoring applies quality checks to the traffic the application actually receives. In Braintrust, automations select incoming logs and execute managed scorers against them. Giran recommends beginning with a higher LLM-judge sampling rate to establish a baseline, then reducing sampling to 5–10% once satisfied with behavior. That is cost-management guidance, not a quality threshold: criticality and traffic volume affect the appropriate rate. Cheap deterministic checks can run on every eligible execution.

An audience question asks what an automation rule actually does. During the UI walkthrough, Giran catches up on a skipped managed-tools checkpoint by using the cumulative branch and uploading the tools and scorers. He then shows the triage judge and its online-quality rule. A rule can operate on an individual span or an entire trace, and metadata can restrict it to relevant executions. The demonstration rule uses 100% sampling.

Two kinds of automation are involved, but they perform different jobs. Setup automation packages assets and bootstraps the environment. An online scoring rule executes against incoming logs. Uploading a scorer alone does not define which production executions should receive that score.

A second audience question exposes a harder issue: what can the judge evaluate when production data has no labeled ground truth? Giran proposes capturing the case in a dataset, beginning the feedback loop and replaying data through the playground. That provides a route for investigating examples, but the exchange leaves the judge’s precise properties and calibration unspecified. Dataset replay does not itself define what a correct judgment means.

The final exercise supplies a concrete priority failure. A customer says the issue is not urgent, but their CFO cannot export invoices before a board meeting. A model that follows the explicit urgency wording can miss the business impact of a blocked financial workflow and an approaching deadline. Calm language does not make the underlying problem low priority.

The repair procedure connects the earlier pieces:

1. Replay the failure so its execution can be inspected.
2. Establish an evaluation for the behavior that should change.
3. Tighten the relevant prompt.
4. Rerun the targeted case.
5. Rerun the broader test set to check for regressions.

The workshop separates the failure and remediation states into different branches. This makes the change itself available for comparison, rather than relying on memory of an earlier answer.

In managed mode, Giran replays five JSON regression cases, including the invoice-export scenario. The traces now include managed tool execution and online scoring. He then runs an evaluation for the calm-wording, high-impact scenario and opens Experiments to inspect the degradation and compare it with existing runs.

The remediation branch contains a revised prompt. During publication, Giran identifies a workshop-specific setup requirement: set BRAINTRUST_IF_EXISTS=replace so the updated assets replace the existing managed definitions. Once uploaded, the UI exposes change authorship and the prompt diff. He calls out the inclusion of tool facts in the revised prompt—the additional evidence the triage decision needs to consider.

Giran reruns the remediated evaluation and reports an upward recovery in the experiment comparison, without giving numerical before-and-after scores. The demonstrated outcome is a visible comparison attached to a specific prompt change. That connection matters: the team can inspect what changed, which cases improved and whether other cases deteriorated, rather than treating a better-looking reply as sufficient evidence.

By the end, the original single-call agent has become a five-stage workflow with tool calls, nested traces, a golden dataset, managed assets and online scoring. The remediation exercise connects them into an operating process: inspect a failure, change the relevant prompt and compare the new execution with the old one. Explicit stages introduce more places where something can fail, but also make those failures easier to locate.

Giran treats production tracing as a baseline requirement. Before production data exists, plausible edge cases are useful; once real failures arrive, they should become recurring test cases. Start with one existing, important workflow rather than instrumenting an entire application suite at once. Add tracing, collect its edge cases, implement deterministic scores and keep routing new evidence into the evaluation set. The value comes from shortening the interval between discovering a failure and learning whether a fix worked.

For implementation follow-through, Giran points to the documentation and its search assistant, then suggests giving the Braintrust cookbook to Cursor or Codex as context for adding SDK tracing. He also announces CLI-assisted instrumentation; the current CLI guide is a follow-on resource rather than the command interface demonstrated in this recording. Account-team and Discord support remain available as teams apply the workflow to their own systems.