Simulation-Maxxing: How Nubank ships agents 20× faster with simulations

How do you improve customer-support agents quickly when every production experiment puts a real customer’s experience at stake? At Nubank, the setting is substantial: Aman Gupta reports 135 million customers across Brazil, Mexico and Colombia, with US expansion still ahead at the time of the talk. Quarterly revenue had crossed five billion in Q1 2026. AI agents handle routine requests end to end, while human experts concentrate on difficult and long-tail cases.

Gupta attributes a 20× faster agent shipping cycle to generating evaluation data in simulation instead of waiting for production data. That is a reported operational result, without a defined timing baseline or aggregation method in the talk. The customer-quality evidence is separate: the opening chart tracks average TNPS, a customer-satisfaction measure, across five production agents. Initially weak results improved over months of work and several quarters of effort, approaching the expert-human benchmark.

Gupta says the displayed results were already stale and that several agents had subsequently exceeded human quality. He also points to a KDD paper. The related Building Customer Support AI Agents at 100M-User Scale documents Nubank’s evaluation-driven framework; its earlier results place most agents below expert-human TNPS, so the later superiority claim remains Gupta’s update rather than a result established by that paper.

Reaching those results requires an evaluation system that supports repeated experiments. Evaluation has two inputs: metrics and data. For metrics, Rajpal describes an established, if demanding, process: use LLM-as-a-judge classifiers, align their decisions with human judgments and task rubrics, then improve them through automated optimization and prompt tuning. A judge becomes useful when its scores reflect the behavior the team actually wants.

The harder bottleneck is obtaining the data those judges score. Structured machine-learning examples and single-turn question answering can often be represented as independent rows. A multi-turn agent example is a trajectory: user messages, agent responses, internal tool calls and the state those calls read or modify. Generating one example means making that entire sequence coherent; annotating it means understanding what happened across the sequence. Repeating that work over a useful sample set compounds the cost.

Teams commonly obtain evaluation trajectories through manual authoring or production traces. Each imposes a different constraint:

Manual authors must anticipate how the conversation unfolds and keep synthetic state consistent across every tool call. Production traces arrive as a by-product of serving customers, but testing changes through those traces means exposing customers to the changes. That makes many parallel production experiments difficult.

A conventional release starts with a change to the agent harness—the prompts, tools and surrounding logic that control the language model. Gupta describes implementation taking hours in the best case, followed by days of offline evaluation on hand-curated data. Production A/B testing then introduces another wait: customer feedback forms produce sparse, noisy observations, and establishing a statistically significant improvement can take much longer.

Gupta reports shortening a weeks-long cycle to less than a day, sometimes hours or minutes. Simulation accelerates the evaluation signal, not the implementation work. Harness changes still take hours. What changes is the ability to generate conversations with a simulated user immediately, rather than wait for people to curate new examples or for enough customers to submit feedback. The resulting offline signal helps predict whether a candidate will perform well in production.

For agent evaluation, simulation means exercising the real agent while controlling the customer and the external environment it encounters. Rajpal describes Snowglobe as providing that infrastructure by pointing its SDK at an agent without changing the agent’s code. The current tutorial makes the integration work explicit through wrappers, tool declarations and decorators; the talk’s description should not be read as a universal promise of zero integration code.

The pipeline separates configuration from the conversations it produces:

1. Identify tools to mock. Determine which external dependencies the simulator must answer for the agent.
2. Configure the scenarios. Supply personas, use cases and the situations the simulation should explore.
3. Run conversations. Exercise the real agent over multi-turn interactions, supplying mocked tool responses as needed.
4. Judge the trajectories. Apply evaluation judges to obtain per-turn information about agent behavior, then feed it into the existing evaluation pipeline.

The output is a dataset of complete conversations, potentially numbering in the thousands, rather than a collection of isolated prompts. The mocked environment makes those conversations usable for evaluating behavior that depends on tool calls and state.

The concrete example is Maria Souza, an explicitly synthetic customer. She is a designer, a first-time credit-card customer, and wants to order a card. The simulator gives her more than a biography: it generates a fake address and credit-card data that remain consistent throughout the agent’s execution. Those facts provide the grounding needed to test whether the agent handles her request correctly.

Maria also has a configured interaction style. Her short, curt, one-line messages follow that persona rather than the verbose, cooperative dialogue a generic simulated user might produce. When the agent calls a tool, the simulator returns valid mocked information identifying Maria and supplying her address for verification. Account data grounds the conversation as well. The test therefore exercises two coupled requirements: the user must behave consistently with her intent and style, and the tools must describe the same customer throughout.

A compact JSON record makes those requirements explicit. It captures the demonstrated scenario and the consistency checks a test fixture needs; the generated address and account values would be shared across the mocked responses that use them.

{
  "persona": {
    "name": "Maria Souza",
    "occupation": "designer",
    "first_time_credit_card_customer": true,
    "intent": "order a credit card",
    "message_style": "curt, one-line messages"
  },
  "grounding_requirements": {
    "identity": "same customer across tool responses",
    "address": "same generated address throughout the conversation",
    "credit_card": "consistent generated card data",
    "account": "consistent generated account data"
  }
}

The critical property is coherence across calls: a tool response that supplies an unrelated address would undermine the test even if each individual response looked plausible.

An agent team faces two uncertainties at once: whether the agent is good, and whether the evaluations can tell. Nubank’s improvement loop starts by shipping and observing an agent, then building robust evaluations. Gupta describes investing in automated prompt optimization, including GEPA, to improve those evaluations.

The team then runs simulations and passes both simulated conversations and real production conversations through its evaluations. Those measurements provide the signal for optimizing the agent harness. After verifying that an optimization is good, the team ships again. Repeating this loop supports many experiments across multiple agents: production observation informs the evaluation system, while simulation supplies additional conversations for testing candidate changes.

Simulation is useful only if its results say something about real behavior. Nubank compares quality measured by evaluations on simulated conversations with quality measured on production conversations. Gupta reports high correlation between the two, without giving a numerical coefficient.

Gupta reports that 80% of domain-expert labels confirmed that simulation data was usable. The talk does not provide the review sample size or detailed labeling rubric. He describes usefulness for both mature agents and greenfield agents, allowing the team to improve an existing system or bring up a new one before it has accumulated a large production history.

Simulation also catches changes that should never reach a live experiment. Gupta describes one regression detected before production and, in a separate agent, an issue that could have reduced the self-service rate. These are different failure modes: a candidate can harm customer experience, or it can make support less efficient by resolving fewer requests without human help.

Gupta reports that one agent’s TNPS doubled through simulation, robust evaluations and systematic improvement work together. No baseline scores are supplied, and the result is attributed to the combined process rather than simulation alone. The team now tries ideas in Snowglobe and waits until it is satisfied with simulation output before launching a candidate.

His illustrative comparison is a team that might otherwise run ten A/B tests in a quarter. It could screen multiple ideas in simulation and move a selected candidate into one live test, avoiding several preliminary experiments. That is a workflow example, not a measured reduction in test count. Production A/B testing remains the next validation step.

Customer satisfaction and self-service must be evaluated together. Higher TNPS can come at the cost of lower self-service if the agent hands more work to humans. Gupta says Nubank improved self-service rather than sacrificing it for satisfaction. Gupta reports a 4% self-service-rate improvement in one case. The talk does not specify whether that means a relative percentage increase or percentage points.

The same infrastructure supports model selection. Nubank places different open-source models inside its agent harnesses, runs them through simulation and evaluates which versions work well for its tasks. Keeping the harness, simulation and evaluation setup available makes it practical to explore more candidates before committing to online experiments.

Gupta reports that simulation saved multiple weeks of effort in screening open-source models. The accompanying slide makes the experiment scale visible: it describes parallel screening of model variants and time saved in online experiments. At the time of the talk, production A/B tests were still running to establish whether the selected open-source models matched frontier models. The demonstrated outcome is a faster shortlist, not established production parity.

Generating evaluation data on demand removes the need to wait for production feedback on every candidate. But faster generation pays off only when the team closes the sim-to-real gap: compare simulation with observed production behavior through offline metrics, online metrics and human review. That comparison establishes when the simulation deserves trust.

For an enterprise agent, continuous improvement rests on two concrete capabilities: metrics aligned with the outcomes that matter, and a reliable way to generate data on which those metrics produce useful signal. Together, they support a repeated loop of feedback, harness changes, verification and release. The destination is an agent that can improve from those signals; the foundation is an evaluation system that makes each proposed improvement worth testing.