Xinghong Fang

The LLM gateway has become a critical piece of infrastructure at Optiver. Engineers, researchers and agents rely on it daily for a variety of tasks, from analysing large volumes of market news and research to running automated workflows. When the gateway goes down, the impact is expensive and quickly noticed. Keeping LLM traffic running reliably across multiple regions and providers has been challenging, with many of the failures outside our control.

In this talk, we share our experience building and scaling a multi-region LLM gateway. We walk through the key design decisions behind protocol translation, scaling, provider failover, observability and multi-region high availability, as well as the tradeoffs and footguns we encountered along the way.