Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons — Kanish Manuja, Twilio

“Something went wrong. Please try again” is a familiar ending to an AI interaction. Kanish Manuja, a principal engineer at Twilio, opens with that message because the system behind it can be much more carefully engineered than the message suggests. A gateway may route around provider failures and still reach a point where it cannot recover a response already underway.

An LLM gateway sits between applications and model providers. It handles routing, authentication, fallback, rate limits and governance. Those responsibilities bring four goals into the same request path: availability, latency, guardrails and cost. During degradation, improving one can worsen another. Sending a second model request may preserve availability while adding latency and expense; continuing without a failed guardrail may preserve service while weakening protection.

The gateway needs controls that callers can choose for their use cases. One recovery policy cannot express every application’s willingness to wait, spend or accept a weaker check. Productionizing the gateway means making those choices available before an incident forces them.

With one model provider, its availability limits the application’s availability: its outage becomes your outage. Conventional dependency handling starts with retries, exponential backoff and jitter, then opens a circuit breaker after enough failures. For a fast, inexpensive API, another attempt can absorb a transient fault. A slow, expensive model call consumes the request’s remaining time and spend much faster.

The basic recovery example uses providers A and B. The gateway sends a request to A. If A fails, it sends that request to B. The observable change is that A’s failure no longer immediately becomes an application failure: another provider gets a chance to answer. This per-request fallback avoids spending the entire recovery effort repeatedly calling A while B remains healthy. 3:00

Two dispatch strategies expose different tradeoffs:

• Sequential fallback: Try A, then B after A fails. The request pays for the time spent on A before B can begin.
• Parallel requests: Start both providers together when latency matters enough to justify duplicate work. Manuja describes this as doubling cost: both model calls run even when the primary might have succeeded.

Circuit breaking still has a role. Once A has been failing for some time, remove it from the request path, let it cool down and try reintroducing it after a few minutes. Per-request fallback handles the current failure; cooldown keeps later requests from repeatedly discovering the same unhealthy dependency.

Where should the failure counts live?

• Instance-local counters: Each serving instance keeps its own history in memory. Changing the deployment size changes how failures accumulate across instances, so the same configuration can produce different failover behavior.
• Fleet-wide counters: Shared infrastructure collects failure history across the deployment. This can help the fleet recognize an outage and fail over quickly.

A successful HTTP exchange with B is only part of a successful fallback. Providers can differ in tool-calling schemas, token limits and stop reasons even when they expose an OpenAI-compatible API format. A normalization layer can reconcile those differences, but the fallback path still needs testing against the application’s actual expectations. Similar request shapes do not guarantee interchangeable behavior.

Now change the A-to-B example by enabling streaming. A begins generating, and the gateway forwards its output to the client. The user sees progress instead of waiting thirty seconds for a wall of text. If A fails after that output arrives, the gateway cannot transparently replace the response with B’s answer: the client already has part of A’s response, and those tokens cannot be recalled. In this streaming design, the continuing response is committed to A. 5:14

Where does the recovery path change? The diagram separates a failure before output reaches the client from one after streaming begins. The first path can move to B; the second leaves an interrupted response because switching providers cannot undo the text already delivered.

That is the mechanism behind the opening error message. Streaming improves perceived speed by giving away a recovery lever. The tradeoff can be worthwhile—some use cases require streaming—but it must be part of the application’s failure design.

B also needs enough capacity to be a real last line of defense. Teams often provision and test the primary thoroughly while giving the backup less attention. Manuja recommends even more throughput, capacity or headroom for the fallback provider. Once the primary is unavailable, an inadequate backup can turn a recoverable provider failure into an application outage.

Availability failures announce themselves with errors and pages. Latency failures can stay quiet: a request remains in progress while the customer waits. Mixed workloads make that problem harder to recognize. Manuja’s examples include embedding and classification requests taking less than a second, chat requests taking three seconds and reasoning requests taking much longer. These illustrate different workload expectations rather than universal performance targets.

A gateway-wide latency number blends those expectations. It cannot tell you whether a chat route has become unusually slow or whether the gateway simply served more long-running reasoning requests. Track P99 per model, per route instead. P99 describes the slow tail of a particular request population; choosing the population is as important as choosing the percentile. As Manuja puts it, “A reasoning model’s normal is actually a chat model’s outage.” 7:24

Timeouts need the same specificity: set them per model class and route. Without a timeout, the gateway can continue treating an outstanding request as healthy even when it is no longer serving the application usefully. Manuja identifies missing timeouts as his leading cause of silent outages. A route-specific timeout gives the gateway a point at which waiting becomes failure and recovery can begin.

Reasoning and router models add another source of variation. Manuja reports production behavior where the same prompt could take two to sixty seconds and P99 suddenly rose to sixty seconds without an identified explanation. That experience concerns the systems he encountered, rather than a universal range for reasoning models. His first recommendation is to fix the reasoning level per route. Some controls may be unavailable: in many cases, he notes, temperature cannot be set to zero. Router models also choose underlying models behind an abstraction, so constrain the request settings you can control to avoid adding unnecessary variation.

Tail hedging offers a different response to a slow primary. Start the primary normally, then launch another request after the primary has spent enough time outstanding to trigger a delay threshold. Unlike starting two requests immediately, this duplicates work only after waiting. Unlike failure-triggered fallback, it acts while the primary is still running. The aim is to reduce the P99 tail, while accepting the cost of additional model work.

Guardrails check for prompt injection, personally identifiable information and toxic output—including the practical problem of a model swearing at customers. Those checks add services to the system, and those services can fail too. Their failure forces an explicit choice:

• Fail open: Serve the request despite the unavailable check. Availability continues with reduced protection.
• Fail closed: Block the request because the check cannot run. Protection takes priority over serving it. 10:15

There is no universal answer across checks and applications. A toxicity-filter outage might be tolerable for one use case, allowing requests to continue; another use case may require blocking them. Choose the worst case you can live with as the default. That decision concerns the consequence of an unavailable check, rather than an assumption that the check will always be available.

Give each guardrail a specific time budget and timeout so it cannot become the component that determines how long the entire request takes. Manuja’s intended design keeps the LLM as the rate-determining step. Guardrails can also have fallback arrangements: secondary providers, secondary checks or cached decisions can preserve service when the primary guardrail provider is down.

Placement changes both latency and what the check can inspect:

• Pre-hook: Check the input before the model runs. Manuja considers this probably the safest placement, but its runtime adds serial latency before generation.
• Parallel check: Run the guardrail concurrently with model work. This overlaps their runtimes and is his preferred latency-saving option for structured output, with a recommendation to avoid streaming that output.
• Post-hook: Inspect output after generation. This placement suits output monitoring and auditing.

Why does parallel placement save time, and why is streaming awkward? The diagram shows the input splitting into two concurrent paths. The check no longer has to finish before model work starts. But streaming can let delivery get ahead of the checking decision—the same irreversible delivery problem that limited provider fallback.

Routing around provider and guardrail failures does not remove the gateway’s own failure modes. It is another dependency in the request path. Shared limits are one way it can spread trouble: a noisy tenant can consume capacity needed by unrelated requests. Segregate API keys as granularly as possible by route and use case so those workloads do not unnecessarily share the same limits.

A retry storm creates a different problem. Repeated attempts pile more work onto an already stressed service, and scaling out alone is insufficient as a recovery strategy. The gateway needs load shedding: the ability to stop accepting more work under overload. Its web-server queues must be bounded, rather than letting requests accumulate without a limit. 13:44

Traffic prioritization can determine which work survives that pressure. Under load, the most important use cases should continue receiving service rather than competing indiscriminately with every retry. Put load shedding into runbooks and game days so operators know the gateway supports it and can use it during an incident.

The final architectural question is whether the entire company needs one central gateway deployment at all. A common request path can become a single point of failure. Before putting every application behind it, identify the requirement that motivated the design. Manuja’s experience is that teams often want centralized governance: consistent cost tracking, rate-limit management and related controls. 14:34

Those controls can live in plugins or shared custom code while gateways remain decentralized. One team can manage the gateway system without operating one company-wide deployment. Even a deployment distributed across instances can remain a shared dependency for every caller; distributing instances does not by itself separate the applications’ request paths. Explore separate deployments with shared governance before making all company traffic depend on the same gateway deployment.

The ending brings those decisions back to the customer. Manuja asks the audience to “prevent one incident.” The next step is small enough to act on: find one route without a timeout, one backup without enough headroom or one gateway queue that can grow without a bound, and fix the failure before customers encounter it.