How a Trading Firm Ships New AI Models the Same Day - Noyan Tokgozoglu, Optiver

A change rolled out to Optiver’s traders and quantitative researchers can show up in profit and loss within hours. That short interval gives an AI platform an unusually direct incentive to make useful capabilities available quickly. Noyan Tokgozoglu, Optiver’s head of AI engineering at the time of the recording, starts with the scale behind that pressure: more than ten million trades a day, more than a million instruments priced, petabytes of data processed daily, and more than ten locations worldwide. The firm trades with its own capital, so the consequences of its decisions land on its own books.

The contrast is with a product business that first watches engagement over days or weeks, then revenue over months or quarters. At Optiver, the daily P&L provides a much earlier signal. Some instruments still have longer horizons, so the first day does not reveal all the value of a decision. But it can reveal enough to make waiting expensive: an opportunity tied to a particular market event may be gone by the time a slow deployment process finishes.

That creates two requirements at once. The platform must move quickly enough to capture an opportunity, and it must measure changes well enough to decide whether they deserve production use. Speed without measurement risks moving the wrong thing into a consequential workflow. Measurement without a fast delivery path leaves useful capabilities waiting.

Modularity is Optiver’s way of keeping that delivery path open. Models, products, and agent architectures keep changing—“loops,” “swarms,” and “graphs” in Tokgozoglu’s quick succession. A useful new approach should be measurable and replaceable without requiring the entire platform to be rebuilt around it. External benchmark scores help identify candidates, but Optiver’s internal tasks have a different data distribution. The decisive question is how a candidate performs on the firm’s own work.

Self-service introduces a separate tension. Traders are capable, motivated users who will build their own workflows, but unrestricted configuration can produce chaos. The platform therefore combines user agency with good defaults. AI engineers also work directly alongside trading teams: they learn how the teams trade, teach where AI can help, and bring those observations back into the shared platform. This gives platform decisions a concrete source of requirements rather than relying only on what engineers imagine users need.

The infrastructure reflects the firm’s regulatory and intellectual-property constraints. Optiver runs its own on-prem GPUs and infrastructure; third-party model inference is the stated exception to keeping work inside that environment, covered by agreements such as zero data retention. Agent safety matters because an agent acting unpredictably against the internal stack could have serious consequences. These constraints shape the platform even as AI use spreads across the company.

The platform overview answers a useful architectural question: where do model access and agent coordination belong? At the bottom, Optiver uses third-party models, self-hosted models, and open-source models. An inference gateway routes requests among them, measures spending, and assigns budgets. That common access layer also provides an alternative when a provider becomes unavailable: the platform can switch providers instead of making every consuming workflow handle the outage itself.

An agent gateway serves a different purpose. It maintains a shared ledger of agents, supports communication and orchestration, and brings humans and agents into similar identity-management arrangements. The agent platform registers capabilities such as skills, plugins, and tools, while individual teams can bring their own knowledge and memory systems. Keeping the gateways distinct separates the choice of a model backend from the discovery and coordination of the agents that use models.

The stack ends with interfaces people can actually use. A web interface exposes sessions; Agent Workbench lets teams build and share profiles for how they use agents; and a personal assistant is available throughout the company. These interfaces are part of the platform’s effectiveness, not an optional finishing layer. A capable agent that people do not know how to reach contributes little to their work.

Optiver’s goal is to make a newly public frontier model usable in production flows on the same day—if it adds value. The qualification matters. Trading teams contribute benchmarks to an internal evaluation framework, creating golden sets against which a new model can be run. The resulting metrics help teams answer whether a public release is useful for their particular cases, rather than treating a higher external benchmark score as sufficient reason to switch.

Several kinds of work support that goal:

• Model integration: Make the model available through the inference gateway, with the appropriate APIs and interfaces.
• Cost accounting: Set up cost management so the new capability enters the same spending controls as existing models.
• Harness delivery: Update and distribute the software around the model when its use requires harness changes.
• Operational support: Maintain regional support rotations for round-the-clock coverage across a global firm.
• User guidance: Share benchmark results, usage instructions, and a recommendation so users can make an informed choice.

This makes same-day adoption an operational capability rather than merely a fast API integration. A model has to be evaluated, reachable, accounted for, supported, and explained. The recording presents this as the platform’s operating goal; it does not provide a release-by-release record showing that every new model met it.

Portability has practical reasons beyond avoiding a favorite vendor. Model availability varies by region and cloud provider. Some workloads must run in an isolated on-prem zone, for both isolation and cost reasons. Optiver therefore wants to remain agnostic to both the model and the harness—the software environment around it that shapes how an agent works.

Third-party and on-prem model interactions pass through the inference gateway. Those interactions are logged and automatically sent to the evaluation system. Above the supported harnesses sits a meta harness: a configuration layer through which teams define their experience and retain some portability between harnesses. Team configurations connect to their knowledge bases, and traces provide material from which additional knowledge can be derived over time.

Which choices can change independently? The diagram separates two branches: a team can expose agents as managed deployments or through the portal/IDE interface, while model requests can reach third-party or on-prem backends through the same inference gateway. Logging feeds evaluation rather than forming an extra user-facing deployment step. The shared configuration layer offers portability across these choices; it does not imply that all harnesses or models behave identically.

The spending chart asks a different question: what happened when powerful new models met growing internal use? Tokgozoglu describes a sharp rise around the summer’s frontier-model releases, followed by spending brought back under control after optimizations. The chart is a rough illustration with no numerical scale, so it supports the direction of the change, not a savings percentage or a measured return on investment.

One concrete failure was visible before an agent had done much work. Some fresh sessions began with 200K tokens of context by default, sometimes more. Users had accumulated MCP-connected tools, knowledge sources, and other material in their setups. Self-service made it easy to add useful capabilities individually; their combined cost became an unreasonable starting state. This was the earlier defaults problem arriving as a token bill.

The first corrections targeted individual sessions rather than requiring a sophisticated routing system:

• Tool access: Add MCP tool search and give agents the option of using a CLI that wraps several MCPs. Tokgozoglu reports the CLI approach as more efficient, without specifying its token accounting.
• Cache behavior: Investigate when caches become invalid, how long cache-related parameters should be set, and whether those settings should vary dynamically. Repeatedly losing reusable context can undermine the benefit of caching.
• Prompt placement: Work with teams to move suitable material out of massive system prompts and into knowledge bases, where it can be retrieved when relevant instead of accompanying every interaction.

These changes address how much context a session carries and how often that context must be paid for. Avoiding unnecessary tokens becomes more valuable as token prices rise. The useful surprise is how much room remained in ordinary session configuration, even before model routing or sub-agent optimization entered the discussion. Shipping the changes also took time across Optiver’s global regions; the point of deployment and the point of full effect were not necessarily the same.

A second set of controls changed what teams could see and choose:

• Spending visibility: Show teams their usage and let them set alerts and limits. Many teams set reasonable limits themselves once they had that information and control.
• Open-source alternatives: Offer models that are competitive on token cost for appropriate tasks. Optiver found useful replacements in some cases, not a universal substitute for frontier models.
• Routing experiments: Explore sending prompts to cheaper models, or using a frontier model to generate a plan whose implementation is divided among smaller models.
• Selective retrieval: Improve knowledge bases so agents retrieve the context they need rather than simply retrieving more context.

The aim is still substantial AI use. Tokgozoglu’s formulation is “more ROI out of the same tokens”: keep investing where the work is valuable, while removing context waste, exposing costs, and matching model expense to the task. Some interventions were already in use; dynamic routing and frontier-planner/smaller-implementer arrangements were described as approaches being explored.

Before the shared portal, users had already solved access for themselves. Different harnesses came with bespoke arrangements for using tmux on development machines, and some people built improvised web interfaces. The platform team treated those workarounds as evidence of demand: build a better version of what people were already assembling, then make it available across the company.

The portal brought session management, chat windows, and artifacts into one place. An internal implementation fit Optiver’s infrastructure and let the team iterate at its own pace. Traders also began contributing code: a useful workflow improvement could arrive as a pull request and become a shared feature. Collaborative document editing is one example of what this contribution path produced.

The next direction was “agent multiplayer”: rooms bringing together one person’s agents, another person’s agents, and other deployed agents. A Slack client also provided another way to access the system. The product principle is deliberately practical. Users with enough agency will build around a missing interface anyway; a shared platform can turn that scattered effort into a better-supported experience.

The questions begin by probing the knowledge layer that supports both agent quality and lower context costs. Optiver uses a hybrid approach that leaves room to explore different methods. Hierarchical Markdown and vector search are both discussed as options with places where they work, rather than as a settled winner. Tokgozoglu is blunt about the state of the work: he is not happy with the solution.

Graph databases are a direction he wants to investigate further, not a demonstrated improvement in this platform. That leaves an important distinction intact: better retrieval can reduce unnecessary context, but choosing and maintaining the knowledge system that delivers it remains an open engineering problem.

A question about hand-built systems versus Codex brings the infrastructure decision into focus. Tokgozoglu describes a historical mismatch: tools did not necessarily support managed agents on Optiver’s own infrastructure in the way the firm needed when it began building. At the time of the recording, he says those options were improving and Optiver was working with the provider. The internal platform’s origin therefore reflects the capabilities and constraints encountered during its development, not a claim that commercial tools can never fit.

The more persistent concern is dependence on someone else’s release schedule. With a previous inference-gateway provider, a newly needed capability could turn into a conversation with a go-to-market team and a request to join the roadmap. That delay conflicts directly with same-day adoption. Deep investment in a provider’s model or surrounding system can make the platform move only as fast as that provider allows.

Building internally also rests on confidence in Optiver’s engineering team and its own managed-agent roadmap. That is a condition of the decision, not a free benefit of owning code. Tokgozoglu explicitly limits the recommendation: this approach will not work well for everyone. The firm’s combination of infrastructure requirements, urgency, and engineering capacity makes the tradeoff attractive in its circumstances.

The next question exposes a limitation beneath the evaluation pipeline. Optiver logs tool-call inputs and outputs and sends them into its evaluation system. But consider the support bot it runs in Slack. To ask whether a different model would make that bot better, the team needs to rerun work involving tools whose answers depend on external state. By the time the evaluation runs again, that state may have changed.

The problem is causal, not merely archival. If a replacement model receives different tool results because the surrounding system changed, an outcome difference cannot be cleanly attributed to the model. Logging the original input and output preserves what happened in that invocation, but rerunning an agent against its tools also requires a way to recover the relevant historical environment.

The feasibility depends on the tool:

• Trading backtest database: Point-in-time access is already part of how the system works, so an evaluation can target the historical state.
• Slack and other changing systems: Reconstructing what was available at the original moment is harder. Tokgozoglu raises the possibility of recreating a point-in-time Slack environment, while noting that some systems offer no comparable answer.

Optiver’s partial solution is to record a confidence value for how well a particular tool invocation can be reconstructed at a point in time. During an evaluation rerun, a proxy around the MCPs lets the team request a particular snapshot. This makes historical-state fidelity an explicit part of the evaluation rather than silently assuming it. The mechanism is imperfect: where the original environment cannot be recovered faithfully, a model comparison remains confounded by changes in tool state.

The final question follows a successful agent into a larger use case. One team had an agent for coding and analysis scoped to its repository. Extending it to work with another team sounded incremental, but it immediately introduced multiple repositories and different authentication requirements. The initial experiment was the obvious one: give a new agent the union of both sets of functionality. In practice, that approach often produced technical debt and poor behavior.

Optiver distinguishes small capability increments from a large expansion of responsibility. Agents are versioned, so a modest addition can ship as a new version. For broader growth, the preferred direction is to keep agents scoped to tasks they do well, can be measured on, and can have their lifecycle managed around. A second task can belong to a second agent rather than becoming another responsibility of the first.

The agent gateway makes that separation usable. Agents share a ledger through which they can find, communicate with, and spin up one another. A narrowly scoped agent can therefore call on another agent’s capability instead of absorbing its tools, repositories, and responsibilities into one growing implementation. This turns expansion into a multi-agent orchestration problem, which still needs engineering, but preserves more manageable units of behavior.

Measurability is the reason to accept that orchestration work. When one large agent does many things, a change can improve some functions while worsening others, making diagnosis and tuning difficult. Smaller task-focused agents keep the affected behavior easier to isolate. The ending returns to the condition behind same-day shipping: changing the system quickly is useful only while the team can still tell what the change did.