# Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons — Kanish Manuja, Twilio

Kanish Manuja, Twilio

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/zrZ1amZBSPw-productionizing-llm-gateways-architecture) · [Video](https://www.youtube.com/watch?v=zrZ1amZBSPw)

## Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons

Kanish Manuja explains how provider fallback, streaming, route-specific timeouts and guardrail placement change an LLM gateway’s failure behavior—and why centralized governance need not mean one company-wide traffic path.

### The system behind “Something went wrong”

[0:13](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=13s)

“Something went wrong. Please try again” is a familiar ending to an AI interaction. [Kanish Manuja](https://www.linkedin.com/in/kanish-manuja-a99bb923/), a principal engineer at Twilio, opens with that message because the system behind it can be much more carefully engineered than the message suggests. A gateway may route around provider failures and still reach a point where it cannot recover a response already underway.

An **LLM gateway** sits between applications and model providers. It handles routing, authentication, fallback, rate limits and governance. Those responsibilities bring four goals into the same request path: availability, latency, guardrails and cost. During degradation, improving one can worsen another. Sending a second model request may preserve availability while adding latency and expense; continuing without a failed guardrail may preserve service while weakening protection.

The gateway needs controls that callers can choose for their use cases. One recovery policy cannot express every application’s willingness to wait, spend or accept a weaker check. Productionizing the gateway means making those choices available before an incident forces them.

### Use the healthy provider before exhausting the unhealthy one

[1:50](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=110s)

With one model provider, its availability limits the application’s availability: its outage becomes your outage. Conventional dependency handling starts with retries, exponential backoff and jitter, then opens a circuit breaker after enough failures. For a fast, inexpensive API, another attempt can absorb a transient fault. A slow, expensive model call consumes the request’s remaining time and spend much faster.

The basic recovery example uses providers A and B. The gateway sends a request to A. If A fails, it sends that request to B. The observable change is that A’s failure no longer immediately becomes an application failure: another provider gets a chance to answer. This **per-request fallback** avoids spending the entire recovery effort repeatedly calling A while B remains healthy. [3:00](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=180s)

Two dispatch strategies expose different tradeoffs:

- **Sequential fallback:** Try A, then B after A fails. The request pays for the time spent on A before B can begin.
- **Parallel requests:** Start both providers together when latency matters enough to justify duplicate work. Manuja describes this as doubling cost: both model calls run even when the primary might have succeeded.

Circuit breaking still has a role. Once A has been failing for some time, remove it from the request path, let it cool down and try reintroducing it after a few minutes. Per-request fallback handles the current failure; cooldown keeps later requests from repeatedly discovering the same unhealthy dependency.

Where should the failure counts live?

- **Instance-local counters:** Each serving instance keeps its own history in memory. Changing the deployment size changes how failures accumulate across instances, so the same configuration can produce different failover behavior.
- **Fleet-wide counters:** Shared infrastructure collects failure history across the deployment. This can help the fleet recognize an outage and fail over quickly.

### Fallback stops being transparent when output escapes

[4:35](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=275s)

A successful HTTP exchange with B is only part of a successful fallback. Providers can differ in tool-calling schemas, token limits and stop reasons even when they expose an OpenAI-compatible API format. A normalization layer can reconcile those differences, but the fallback path still needs testing against the application’s actual expectations. Similar request shapes do not guarantee interchangeable behavior.

Now change the A-to-B example by enabling streaming. A begins generating, and the gateway forwards its output to the client. The user sees progress instead of waiting thirty seconds for a wall of text. If A fails after that output arrives, the gateway cannot transparently replace the response with B’s answer: the client already has part of A’s response, and those tokens cannot be recalled. In this streaming design, the continuing response is committed to A. [5:14](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=314s)

Where does the recovery path change? The diagram separates a failure before output reaches the client from one after streaming begins. The first path can move to B; the second leaves an interrupted response because switching providers cannot undo the text already delivered.

That is the mechanism behind the opening error message. Streaming improves perceived speed by giving away a recovery lever. The tradeoff can be worthwhile—some use cases require streaming—but it must be part of the application’s failure design.

B also needs enough capacity to be a real last line of defense. Teams often provision and test the primary thoroughly while giving the backup less attention. Manuja recommends even more throughput, capacity or headroom for the fallback provider. Once the primary is unavailable, an inadequate backup can turn a recoverable provider failure into an application outage.

The same provider failure has two recovery paths
Before delivery, fallback can give B a chance to answer. After A’s output reaches the client, a transparent switch cannot retract it.
- Request to provider A — The primary provider starts handling the request.
- A fails before delivery — The request can follow the sequential fallback path.
- Try provider B — A second provider gets a chance to serve the request.
- A’s tokens reach the client — Streaming makes progress visible; delivered output cannot be recalled.
- A fails mid-stream — The response is interrupted; transparent provider switching is unavailable.
- Request to provider A → A fails before delivery (Failure before delivery)
- A fails before delivery → Try provider B (Sequential fallback)
- Request to provider A → A’s tokens reach the client (Forward streaming output)
- A’s tokens reach the client → A fails mid-stream (Later failure cannot undo delivered tokens)

### A reasoning model’s normal can be a chat model’s outage

[6:30](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=390s)

Availability failures announce themselves with errors and pages. Latency failures can stay quiet: a request remains in progress while the customer waits. Mixed workloads make that problem harder to recognize. Manuja’s examples include embedding and classification requests taking less than a second, chat requests taking three seconds and reasoning requests taking much longer. These illustrate different workload expectations rather than universal performance targets.

A gateway-wide latency number blends those expectations. It cannot tell you whether a chat route has become unusually slow or whether the gateway simply served more long-running reasoning requests. Track **P99 per model, per route** instead. P99 describes the slow tail of a particular request population; choosing the population is as important as choosing the percentile. As Manuja puts it, “A reasoning model’s normal is actually a chat model’s outage.” [7:24](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=444s)

Timeouts need the same specificity: set them per model class and route. Without a timeout, the gateway can continue treating an outstanding request as healthy even when it is no longer serving the application usefully. Manuja identifies missing timeouts as his leading cause of silent outages. A route-specific timeout gives the gateway a point at which waiting becomes failure and recovery can begin.

Reasoning and router models add another source of variation. Manuja reports production behavior where the same prompt could take two to sixty seconds and P99 suddenly rose to sixty seconds without an identified explanation. That experience concerns the systems he encountered, rather than a universal range for reasoning models. His first recommendation is to fix the reasoning level per route. Some controls may be unavailable: in many cases, he notes, temperature cannot be set to zero. Router models also choose underlying models behind an abstraction, so constrain the request settings you can control to avoid adding unnecessary variation.

**Tail hedging** offers a different response to a slow primary. Start the primary normally, then launch another request after the primary has spent enough time outstanding to trigger a delay threshold. Unlike starting two requests immediately, this duplicates work only after waiting. Unlike failure-triggered fallback, it acts while the primary is still running. The aim is to reduce the P99 tail, while accepting the cost of additional model work.

### Guardrails need failure policies and their own time budgets

[9:45](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=585s)

Guardrails check for prompt injection, personally identifiable information and toxic output—including the practical problem of a model swearing at customers. Those checks add services to the system, and those services can fail too. Their failure forces an explicit choice:

- **Fail open:** Serve the request despite the unavailable check. Availability continues with reduced protection.
- **Fail closed:** Block the request because the check cannot run. Protection takes priority over serving it. [10:15](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=615s)

There is no universal answer across checks and applications. A toxicity-filter outage might be tolerable for one use case, allowing requests to continue; another use case may require blocking them. Choose the worst case you can live with as the default. That decision concerns the consequence of an unavailable check, rather than an assumption that the check will always be available.

Give each guardrail a specific time budget and timeout so it cannot become the component that determines how long the entire request takes. Manuja’s intended design keeps the LLM as the rate-determining step. Guardrails can also have fallback arrangements: secondary providers, secondary checks or cached decisions can preserve service when the primary guardrail provider is down.

Placement changes both latency and what the check can inspect:

- **Pre-hook:** Check the input before the model runs. Manuja considers this probably the safest placement, but its runtime adds serial latency before generation.
- **Parallel check:** Run the guardrail concurrently with model work. This overlaps their runtimes and is his preferred latency-saving option for structured output, with a recommendation to avoid streaming that output.
- **Post-hook:** Inspect output after generation. This placement suits output monitoring and auditing.

Why does parallel placement save time, and why is streaming awkward? The diagram shows the input splitting into two concurrent paths. The check no longer has to finish before model work starts. But streaming can let delivery get ahead of the checking decision—the same irreversible delivery problem that limited provider fallback.

Parallel guardrails overlap checking with generation
Concurrent work removes the serial wait before generation. Streaming can expose output while the guardrail decision is still pending.
- Request input — Starts both paths in the parallel placement.
- Guardrail check — Runs within a specific time budget; failure invokes the chosen open or closed policy.
- Model generation — Runs concurrently with the guardrail.
- Guardrail decision — The check can complete after generation has begun.
- Streamed client output — Delivery can precede completion of the concurrent check; delivered tokens cannot be recalled.
- Request input → Guardrail check (Start checking)
- Request input → Model generation (Start generation concurrently)
- Guardrail check → Guardrail decision (Complete check)
- Model generation → Streamed client output (Streaming may precede check completion)

### The gateway becomes another dependency

[12:59](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=779s)

Routing around provider and guardrail failures does not remove the gateway’s own failure modes. It is another dependency in the request path. Shared limits are one way it can spread trouble: a noisy tenant can consume capacity needed by unrelated requests. Segregate API keys as granularly as possible by route and use case so those workloads do not unnecessarily share the same limits.

A retry storm creates a different problem. Repeated attempts pile more work onto an already stressed service, and scaling out alone is insufficient as a recovery strategy. The gateway needs **load shedding**: the ability to stop accepting more work under overload. Its web-server queues must be bounded, rather than letting requests accumulate without a limit. [13:44](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=824s)

Traffic prioritization can determine which work survives that pressure. Under load, the most important use cases should continue receiving service rather than competing indiscriminately with every retry. Put load shedding into runbooks and game days so operators know the gateway supports it and can use it during an incident.

### Centralize governance without concentrating all traffic

[14:34](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=874s)

The final architectural question is whether the entire company needs one central gateway deployment at all. A common request path can become a single point of failure. Before putting every application behind it, identify the requirement that motivated the design. Manuja’s experience is that teams often want centralized governance: consistent cost tracking, rate-limit management and related controls. [14:34](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=874s)

Those controls can live in plugins or shared custom code while gateways remain decentralized. One team can manage the gateway system without operating one company-wide deployment. Even a deployment distributed across instances can remain a shared dependency for every caller; distributing instances does not by itself separate the applications’ request paths. Explore separate deployments with shared governance before making all company traffic depend on the same gateway deployment.

The ending brings those decisions back to the customer. Manuja asks the audience to “prevent one incident.” The next step is small enough to act on: find one route without a timeout, one backup without enough headroom or one gateway queue that can grow without a bound, and fix the failure before customers encounter it.

## Key takeaways

- Prefer per-request provider fallback to blind retries, while using cooldown to keep a repeatedly failing primary out of later request paths.
- Streaming commits delivered output to the selected provider. A mid-stream failure cannot be recovered through a transparent provider switch; provision the backup thoroughly for requests that can fail over.
- Measure P99 per model and route, set corresponding timeouts and constrain available reasoning settings. Tail hedging launches another request after waiting for a slow primary.
- Treat guardrails as fallible services: choose fail-open or fail-closed behavior, bound their runtime, consider fallback checks and place them deliberately relative to generation.
- Protect the gateway with granular API keys, bounded queues, load shedding and traffic priorities. Shared governance can coexist with separate gateway deployments.

## Resources

- [Kanish Manuja’s restaurant voice-agent prototype lessons](https://www.linkedin.com/posts/kanish-manuja-a99bb923_conversationrelay-activity-7339233014209462272-NbbD) — A separate account of a restaurant order-taking prototype comparing a speech-to-text → LLM → text-to-speech pipeline with an end-to-end realtime audio approach. Useful for extending the talk’s latency, cost and control tradeoffs to voice applications; its conclusions reflect that prototype and the models used then.

## Related talks

- [The State of Model Routing — NVIDIA, Cognition, OpenRouter](https://ai.engineer/talks/QHBjufYK8TA-state-model-routing-nvidia-cognition-openrouter) — Continues the model-routing subject introduced by provider fallback and router-model latency.
- [$1 AI Guardrails: The Unreasonable Effectiveness of Finetuned ModernBERTs – Diego Carpintero](https://ai.engineer/talks/YZHPEkfy2kc-1-ai-guardrails-unreasonable-effectiveness) — A companion on guardrail models for readers considering the cost and implementation of checks placed around generation.

## Slides

### Productionizing LLM Gateways

Kanish Manuja introduces productionizing LLM gateways against the backdrop of provider outages. The opening diagram shows a gateway router connected to a primary provider and two alternatives.

[0:18](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=18s) [Title slide introducing Kanish Manuja and showing a gateway routing to a primary and two other providers.](https://assets.ai.engineer/sites/aiecode2025/sha256/9de645ada25b5d576e1c8b7aa4039c7f394f1145f7efcf62a74249f7fa05fea6.webp)

Productionizing LLM Gateways

Architecture, tradeoffs, and hard lessons from the trenches

Kanish Manuja
Principal Engineer, twilio

Gateway
router
Primary
Provider B
Provider C

2026 Twilio Inc. All rights reserved

### What is an LLM gateway?

An LLM gateway sits between apps and model providers, giving them one API and a central place for routing, credentials, fallback, limits, guardrails, and observability.

[0:49](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=49s) [Apps connect through an LLM gateway to three providers; six boxes list gateway functions.](https://assets.ai.engineer/sites/aiecode2025/sha256/6fec08434fcbce4bb75dd9130ec399d87744caca2ad61190e5380027fdc06f99.webp)

First: what is an LLM gateway?
One entry point between your apps and the model providers behind them

App / service A
App / service B

LLM Gateway
one API · one place for policy

Provider A
Provider B
Provider C

WHAT A GATEWAY CONSOLIDATES

ROUTING
Pick the model or provider per request.

AUTH & KEYS
One credential boundary; rotate keys centrally.

FALLBACK
Reroute when a provider fails or slows.

RATE LIMITS & COST
Throttle callers; attribute spend per workload.

GUARDRAILS
Run input / output safety checks in one place.

OBSERVABILITY
Tokens, latency, and errors tracked per model.

2026 Twilio Inc. All rights reserved.

### A gateway is where four things fight

A gateway puts availability, latency, guardrails and cost on the critical path. During degradation, users must choose their tradeoffs; designers should give callers those choices.

[1:08](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=68s) [A gateway sits between availability, latency, guardrails and cost, with guidance for gateway users and designers.](https://assets.ai.engineer/sites/aiecode2025/sha256/6ba41178a11a02f8dc35f16558c38d1adb05018a6cef665891f59fd79e4236b3.webp)

A gateway is where four things fight

Availability, latency, security, and cost. Every safeguard buys you one and charges you another. You cannot max all four at once.

IF YOU USE A GATEWAY
You’ll know which axis to trade away for your use case.

IF YOU DESIGN A GATEWAY
You’ll know which levers to provide so callers can choose.

Availability
fallbacks

Latency
timeouts

Gateway
critical path

Guardrails
input, output

Cost
limits

2026 Twilio Inc. All rights reserved

### Availability: fallback done right

A single model provider’s outage becomes your outage. Favor a primary with the strongest guarantees, then fall back to another provider only when it fails.

[1:45](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=105s) [Section slide recommends a strong primary provider and fallback when it fails.](https://assets.ai.engineer/sites/aiecode2025/sha256/d9ed7bd768961f48f538bc13435a39477280a857554be5ff2e146c2aed81e2c5.webp)

Availability: fallback done right

01

One provider means one ceiling. Favor a primary with the strongest guarantees, then fall back only when it fails.

2026 Twilio Inc. All rights reserved.

### The standard playbook: retries + circuit breaker

Retries consume latency and cost, while a tripped circuit breaker can fail a request even when another provider is available.

[2:00](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=120s) [The slide defines retries and circuit breakers and lists three limitations for LLM calls.](https://assets.ai.engineer/sites/aiecode2025/sha256/6f3d4a2d600bb68460114e73b61f065c54eae90abe3a5940119020696e9491e2.webp)

The standard playbook: retries + circuit breaker

You already know these. They were built for stateless, homogeneous, low-latency calls.

RETRIES
Retry the same provider on a transient error.

CIRCUIT BREAKER
Trip after N failures; stop calling a dead dependency.

Three reasons that isn’t enough for LLMs →

PROBLEM 1
Retrying the same provider during its outage just burns your latency budget.

PROBLEM 2
A tripped breaker fails the request even though a second provider was available the whole time.

PROBLEM 3
Calls are slow and expensive, so blind retries multiply both cost and tail latency.

2026 Twilio Inc. All rights reserved.

### Per-request fallback and a breaker that ejects the primary

An open per-model breaker removes the primary from the request path and routes directly to the secondary. Fleet-wide state needs shared infrastructure; local state requires each instance to fail first.

[2:58](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=178s) [A breaker routes requests to a secondary when the primary is open, with notes on per-request decisions and breaker state.](https://assets.ai.engineer/sites/aiecode2025/sha256/d62be73ec3b73d202f9e17b31602a35c592b852973f276d6ab3d56c6d614abbb.webp)

Better: per-request fallback + a breaker that ejects the primary
When the breaker trips, the primary leaves the path, so requests route straight to secondary

Request

Breaker
per-model state

Primary (open)
ejected, not even tried

Secondary
direct route while open

Response

PER-REQUEST
Decided on every call. Try providers in sequence, or fire in parallel for speed (faster, but you pay for every call).

EJECT, DON’T RETRY
Open breaker removes the primary from the path entirely, with no wasted attempt.

WHERE STATE LIVES
Fleet-wide trips once for everyone (needs shared infra); local needs no infra but each instance must fail first.

2026 Twilio Inc. All rights reserved.

### Fallback and streaming trade-offs

Provider differences call for normalization; once streaming starts, a failure reaches the user without a clean retry. Manuja also urges teams to provision and test fallback capacity, which often gets less attention.

[4:33](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=273s) [Three considerations: provider differences, streaming limitations, and fallback provider capacity.](https://assets.ai.engineer/sites/aiecode2025/sha256/7dfe344dff1efed9494c993b5e8509bbe0974dce74e0fc9831ee56b8d496f72e.webp)

From the trenches: what the clean diagram doesn’t show you

01 Fallback is not transparent
Falling from one provider to the next means different tool-calling schemas, token limits, and stop reasons. Without a normalization layer, the response shape shifts under your application.

02 Streaming trades away your levers
Once the first token is out, you can’t fall back, retry, or rewrite. A mid-response failure is one the user sees. Stream only when latency demands it; if you don’t, you keep every lever to fix a bad response before you ship it.

03 Fallback Provider capacity
Teams are cognizant of the primary provider capacity but not the secondary provider for the throughput.

2026 Twilio Inc. All rights reserved.

### Latency across mixed workloads

In the illustrative chart, p99 ranges from 0.3s for embeddings to 38s for reasoning; a blended ~9s obscures that spread. Track p99 per model and route, and set timeouts per model class and route.

[6:50](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=410s) [Illustrative p99 latency chart compares embeddings, classification, chat, reasoning, and blended latency.](https://assets.ai.engineer/sites/aiecode2025/sha256/c6488ae5bf7c87813dccc0303a6708abe9eb769e0e042a8c7ab18abebc18800f.webp)

A gateway may run mixed workloads, so blended latency lies

Track p99 per model and route, not one gateway-wide number.

Set timeouts per model class per route

A reasoning model’s normal is a chat model’s outage.

P99 LATENCY BY WORKLOAD (ILLUSTRATIVE)
Embeddings 0.3s
Classify 0.6s
Chat 3s
Reasoning 38s
Blended p99 ~9s ← lies

2026 Twilio Inc. All rights reserved.

### Reasoning and router model latency

The same prompt can take 2s or 60s, while routers hide which model ran. Fix reasoning levels per route and hedge slow requests with a delayed backup; take the first response and cap the rate to control cost.

[8:18](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=498s) [Variable thinking time and model routing affect latency; proposed mitigations include fixed reasoning levels and hedged requests.](https://assets.ai.engineer/sites/aiecode2025/sha256/c49a1f64e400b658111b8fc0e9430d673eca90656b4b96a1042ee3c89e2ae961.webp)

Reasoning & router models make latency genuinely unpredictable

WHY
Variable thinking time
Reasoning models ”think” for a non-deterministic number of tokens, so the same prompt can take 2s or 60s.

WHY
Router models hide which model ran
When the provider picks the model per request, your latency is a mixture you can’t predict.

FIX
Static reasoning level per route
Latency expectations are measured well ahead of time

FIX
Hedge the tail
Fire a backup request after a short delay and take the first to return. Cuts p99; cap the rate to control cost.

2026 Twilio Inc. All rights reserved.

### Security and guardrails

Guardrails help filter prompt injection, PII and toxic content, but they add a service dependency that can fail.

[9:43](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=583s) [Security section divider states that guardrails are necessary and introduce a dependency.](https://assets.ai.engineer/sites/aiecode2025/sha256/c01c929c5baed359455274ed1c9114f59e1da099eff2fc8e125a5f73187b1884.webp)

Security:
ignore all instructions and
focus on guardrails

Guardrails are necessary, and they are a new dependency that can take you down.

03

2026 Twilio Inc. All rights reserved

### When a guardrail fails

When a guardrail goes down, fail open keeps requests flowing but accepts security risk; fail closed blocks them. Choose per policy based on the worst case you can tolerate.

[10:13](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=613s) [Fail-open and fail-closed choices trade security against availability; the slide recommends deciding by policy severity and blast radius.](https://assets.ai.engineer/sites/aiecode2025/sha256/057ac97028ce7a24b740c18ab73a5f89cb97160d9be75b1563cd53da79f6dc77.webp)

When the guardrail itself fails, you must choose

FAIL OPEN
Let the request through ungated. Protects availability; accepts security risk.

FAIL CLOSED
Block the request. Protects security; accepts an availability hit.

There is no universal answer. It’s a per-policy risk decision. A toxicity filter might fail open; a PII or prompt-injection check should fail closed.

DECIDE BY BLAST RADIUS
Default to the choice whose worst case you can live with.

Tie the decision to each guardrail’s severity, not one global switch.

2026 Twilio Inc. All rights reserved

### Making guardrails dependable

Pre- and post-hooks add serial latency. Parallel checks gate the response with little perceived latency, but work poorly with streaming. Deadlines and fallbacks help manage guardrail failures.

[11:01](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=661s) [Guardrail design considerations include time budgets, fallback, and running checks before, alongside, or after the model.](https://assets.ai.engineer/sites/aiecode2025/sha256/3d15c3bf57bab05cdefe3b61fcd300f6ec186fd7b741f7f76003040cc6b6b991.webp)

Make guardrails safe to depend on

TIME BUDGET
Give each guardrail a hard deadline. A slow check must never become your app’s latency.

FALLBACK
Guardrails are services too. Have a secondary check or cached decision when one is down.

PLACEMENT →
When you run them decides how much latency users feel.

WHEN TO RUN GUARDRAILS

PRE-HOOK (BEFORE THE MODEL)
Safest for inputs; serial latency on every call.

IN PARALLEL (ALONGSIDE THE MODEL) ★
Run concurrently, gate the response. Virtually no perceived latency. The preferred default.

POST-HOOK (AFTER THE MODEL)
Needed to inspect output; serial on the response.

2026 Twilio Inc. All rights reserved.

### Shared limits and load shedding

Where permissible, separate keys by team or workload to contain noisy tenants. Bound queues and shed excess requests with a fast 429; unbounded queues can grow during retry storms and make scaling out difficult.

[12:57](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=777s) [Slide contrasts shared rate limits and unbounded queues with isolated usage and bounded queues that shed excess load.](https://assets.ai.engineer/sites/aiecode2025/sha256/5092e684eb05c80c433516278e103c18267627d91f3cfffc2714dfef19879a4d.webp)

Shared limits are a trap, shared infra is a risk

SHARED RATE LIMITS
One noisy tenant starves the rest. Where permissible, isolate usage with separate keys per team or workload. This is the simplest path to cost attribution and blast-radius control.

LOAD SHEDDING
Under resource pressure, shed early with bounded queues. Never queue unboundedly.

UNBOUNDED QUEUE + RETRY STORM = Disaster

UNBOUNDED
queue grows → can’t scale out

BOUNDED + SHED
shed past limit → fast 429, stable, scalable

2026 Twilio Inc. All rights reserved.

### A central gateway is a single point of failure

Centralizing fallback, guardrails and routing lets one bad deploy or shared breaker state affect every workload. Manuja recommends local resilience and a shared control plane for cost and guardrail events.

[14:32](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=872s) [Slide recommends decentralized gateways with local resilience and centralized governance.](https://assets.ai.engineer/sites/aiecode2025/sha256/eca7ebca839223856835eba6bb4249f60505cdafdb41a0066c34e043c778fe46.webp)

A central gateway is a single point of failure

Centralizing fallback, guardrails, and routing concentrates blast radius: one bad deploy or shared breaker state can take down every workload at once.

Decentralize the gateway. Each service owns and maintains its own fallback, timeouts, and load shedding.

Centralize governance, not traffic. Integrations for cost attributions and guardrail-failure events, etc without the SPOF.

Decentralized gateway, centralized governance.

Keep resilience local; emit cost and guardrail events to a shared control plane.

2026 Twilio Inc. All rights reserved

### AI Engineer World's Fair

[16:15](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=975s) [AI Engineer World's Fair logo on a black background, with the AI Engineer website below.](https://assets.ai.engineer/sites/aiecode2025/sha256/e3c9c767202d6914647b3070fb4145c6db47d0e6ba189981bd0de2523dcf5e7e.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:01](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=1s) [music]

[0:13](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=13s) I'm Kanesh Manuja. I'm a principal

[0:15](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=15s) engineer at Twilio.

[0:18](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=18s) Let's start with a quick show of hands.

[0:20](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=20s) Who here has seen the message, something

[0:23](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=23s) went wrong. Please try again.

[0:27](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=27s) Well, we have a few lucky ones and a few

[0:30](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=30s) that have had a good lunch. Um, so

[0:34](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=34s) behind that simple message is actually a

[0:37](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=37s) system that is very complex

[0:40](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=40s) that serves you that message despite the

[0:42](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=42s) model providers being down.

[0:45](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=45s) And that's what we're going to

[0:46](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=46s) productionize today or discuss

[0:48](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=48s) productionizing today.

[0:50](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=50s) So what is an LM gateway? An LLM gateway

[0:53](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=53s) is an entry point or a middleware

[0:55](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=55s) between your apps and the model

[0:57](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=57s) providers behind them. It does a bunch

[1:00](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=60s) of things. Routing, authentication,

[1:02](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=62s) fallback, rate limits, all kinds of

[1:04](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=64s) governance that you can think of.

[1:08](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=68s) And right at the heart of the gateway is

[1:11](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=71s) a fight between four things. It's

[1:13](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=73s) availability, latency, your guardrails

[1:17](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=77s) and costs.

[1:18](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=78s) In case of a degradation, you cannot

[1:21](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=81s) maximize all four. You need to pick what

[1:24](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=84s) you want. So with this talk, if you use

[1:29](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=89s) an LLM gateway, I want you I want to

[1:32](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=92s) help you to make that trade-off for your

[1:34](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=94s) use case. And if you design a gateway, I

[1:37](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=97s) want you to design or provide those

[1:39](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=99s) levers to your callers and customers u

[1:43](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=103s) so that your customers are happy.

[1:46](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=106s) Let's start with availability.

[1:50](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=110s) If you have a single model provider,

[1:54](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=114s) their ceiling is your ceiling. Their

[1:57](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=117s) outage is your outage.

[2:03](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=123s) So in typical software engineering, the

[2:06](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=126s) way you tackle unreliable dependency is

[2:08](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=128s) by retrying.

[2:11](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=131s) Retrying with exponential backoffs, with

[2:14](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=134s) jitters. And when all of that fails, you

[2:17](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=137s) have a circuit breaker that trips after

[2:19](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=139s) you've seen sufficient failures and you

[2:21](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=141s) stop calling the damn thing.

[2:24](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=144s) This is not enough for LLMs. LLMs are

[2:27](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=147s) very different compared to your fast

[2:29](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=149s) cheap APIs that you retry on. Retrying

[2:33](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=153s) an LLM API eats into your latency budget

[2:37](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=157s) really fast. And also tripping over a

[2:40](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=160s) circuit breaker when you have another

[2:43](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=163s) perfectly fine model provider to route

[2:45](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=165s) to doesn't make sense. You should use

[2:47](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=167s) the second model provider. And third, as

[2:51](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=171s) I said, the calls are slow and

[2:53](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=173s) expensive. So blind retries just

[2:56](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=176s) multiply your cost and your tail

[2:57](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=177s) latencies.

[3:00](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=180s) So what is a better idea here? It is

[3:03](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=183s) actually a per request fallback. What

[3:06](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=186s) that means is you can actually try model

[3:08](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=188s) provider A and then in sequence try

[3:11](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=191s) model provider B if your request to

[3:13](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=193s) model provider A fails. Another option

[3:16](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=196s) to consider here is you can fire

[3:18](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=198s) requests to both the providers in

[3:19](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=199s) parallel. But that's only if you're

[3:21](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=201s) highly highly obsessed with latencies

[3:25](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=205s) because that's just going to double your

[3:26](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=206s) cost.

[3:28](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=208s) Some of the similar circuit breaking

[3:30](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=210s) patterns apply here to LMS as well. If

[3:34](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=214s) you know that your primary has been

[3:37](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=217s) failing for some time, it doesn't make

[3:39](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=219s) sense to try it again. You put it, you

[3:42](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=222s) take it out of the load balancer or your

[3:45](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=225s) request path and put it in a cool down

[3:48](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=228s) and then after a few minutes have

[3:50](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=230s) passed, try putting that back again.

[3:53](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=233s) One interesting choice that you have to

[3:56](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=236s) make here is where your failure counts

[3:58](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=238s) live.

[4:00](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=240s) You can decide to have the failure

[4:01](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=241s) counts live in memory on the instances

[4:04](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=244s) that are serving your traffic or you can

[4:07](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=247s) have shared infra where your failure

[4:11](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=251s) counts are shared across the fleet.

[4:13](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=253s) There are trade-offs.

[4:15](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=255s) If you want quick failovers, then

[4:18](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=258s) fleetwide helps. And with instance uh

[4:22](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=262s) with local state counters the issue that

[4:25](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=265s) you run into is whenever you change your

[4:26](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=266s) deployment size your configuration and

[4:29](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=269s) your expectations change. So something

[4:32](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=272s) to consider.

[4:35](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=275s) What that clean diagram did not really

[4:37](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=277s) show you are some of the other gotchas

[4:39](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=279s) that I'm going to discuss. So fallbacks

[4:41](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=281s) are not transparent.

[4:43](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=283s) While the industry is converging on an

[4:45](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=285s) OpenAI API compatible format, I would

[4:49](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=289s) say there are still nuances. So you need

[4:50](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=290s) to really test your fallbacks well. They

[4:53](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=293s) can have differences in your tool

[4:55](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=295s) calling schemas, token limits, stop

[4:57](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=297s) reasons and what have you. So with LM

[4:59](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=299s) gateways, you can have a normalization

[5:02](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=302s) layer that can ensure that you can do

[5:05](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=305s) cross provider fallbacks as well.

[5:08](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=308s) Another thing is streaming

[5:14](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=314s) it.

[5:16](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=316s) So essentially nobody wants to wait for

[5:20](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=320s) 30 seconds to have a wall of text appear

[5:22](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=322s) in front of them. So there are use cases

[5:24](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=324s) where streaming is absolutely required.

[5:27](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=327s) But it comes as at a cost. You trade

[5:29](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=329s) away your levers. You cannot once you

[5:31](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=331s) have decided to go with provider A, you

[5:34](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=334s) have to continue going with provider A.

[5:37](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=337s) You cannot mid-stream change the

[5:39](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=339s) providers. Whatever has been sent to the

[5:42](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=342s) client, it's done. And that's where the

[5:45](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=345s) something uh went wrong message, that's

[5:47](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=347s) the one that you see. It's not because

[5:50](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=350s) of laziness. It's by design uh that you

[5:52](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=352s) see that and it's one of the trade-offs.

[5:55](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=355s) I would like to call out one other thing

[5:57](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=357s) where I've seen teams trip over and over

[6:00](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=360s) again. They really provision and test

[6:03](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=363s) their primary providers really well, but

[6:07](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=367s) they the second provider, the fallback

[6:09](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=369s) provider doesn't necessarily get the

[6:10](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=370s) same level of love. And I would argue

[6:13](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=373s) that your throughputs or your capacity

[6:15](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=375s) or your headroom should be even higher

[6:18](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=378s) for the second provider or the fallback

[6:21](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=381s) provider because that's your last line

[6:23](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=383s) of defense. If that goes down, your

[6:25](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=385s) application goes down.

[6:30](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=390s) Let's discuss latencies.

[6:32](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=392s) Availability failures are right in your

[6:34](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=394s) face. They fail. You get alarmed. You

[6:38](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=398s) get paged. But high latencies can be the

[6:42](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=402s) quiet ones. And they need to receive

[6:45](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=405s) more love um than I would say tuning

[6:47](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=407s) your services for just availability.

[6:54](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=414s) One thing to call out, a gateway may run

[6:58](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=418s) mixed workloads

[7:00](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=420s) and you can have embedding embedding

[7:02](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=422s) requests that takes just less than a

[7:04](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=424s) second. You can have classification

[7:06](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=426s) requests that take less than a second.

[7:08](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=428s) Uh you have chat requests taking 3

[7:10](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=430s) seconds and reasoning requests taking a

[7:13](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=433s) long time.

[7:15](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=435s) Quick show of hands. If you measure

[7:18](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=438s) your aggregate latency for your entire

[7:20](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=440s) service.

[7:22](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=442s) Well, that was a trick question. Sorry.

[7:24](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=444s) You shouldn't. It doesn't make sense.

[7:25](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=445s) It's a lie. You should be tracking your

[7:28](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=448s) P99 per model per route, not a gateway

[7:32](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=452s) wide number. Gateway wide number doesn't

[7:34](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=454s) make sense, especially if you're running

[7:36](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=456s) mixed workloads. And I hope you're not u

[7:38](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=458s) for those who raise your hand. Another

[7:41](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=461s) thing that can really I cannot emphasize

[7:44](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=464s) this enough is for you to set timeouts

[7:47](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=467s) on per model class per route.

[7:50](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=470s) That's where that's the number one root

[7:52](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=472s) cause of your silent outage. If you

[7:54](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=474s) don't have a timeout, your gateway

[7:57](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=477s) thinks you're hap your request is being

[7:58](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=478s) happily served while it is not. And I'll

[8:02](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=482s) leave you with this message for for

[8:03](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=483s) latencies. Um, specifically a reasoning

[8:07](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=487s) models normal is actually a chat models

[8:10](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=490s) outage. So you definitely need to track

[8:12](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=492s) latency per route.

[8:17](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=497s) Okay, this is the most painful or this

[8:19](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=499s) the slide that has given me the most

[8:20](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=500s) scarse which is reasoning and router

[8:24](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=504s) models. So this is where truly the

[8:28](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=508s) latency is unpredictable

[8:30](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=510s) and reasoning models they do not give

[8:34](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=514s) you

[8:36](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=516s) they they're highly undeterministic more

[8:38](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=518s) deterministic undeterministic than your

[8:40](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=520s) normal models. You cannot set the

[8:42](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=522s) temperature to zero in many cases and

[8:44](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=524s) the same prompt can take somewhere from

[8:47](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=527s) 2 seconds to 60 seconds and we've seen

[8:49](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=529s) that in production where P99 suddenly

[8:51](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=531s) popped to 60 seconds for no good reason.

[8:54](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=534s) So that's

[8:56](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=536s) while there's no magical solution to it.

[8:59](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=539s) I would recommend that you at least

[9:01](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=541s) start with fixing the reasoning level

[9:03](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=543s) per route. So with router models, they

[9:07](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=547s) hide that abstraction behind you. Like

[9:09](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=549s) they pick which models to run and I

[9:12](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=552s) would highly recommend that you at least

[9:15](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=555s) make as much uh you make requests as

[9:18](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=558s) determinist deterministic as possible

[9:20](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=560s) with an undeterministic system.

[9:24](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=564s) Another idea is hedging the tail. You

[9:28](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=568s) can have a you can fire another request

[9:30](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=570s) if your primary request actually

[9:32](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=572s) consumed let's say P90 of your latency

[9:35](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=575s) budget.

[9:37](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=577s) This can hedge the t this can really

[9:39](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=579s) hedge the P99 tail u for for your

[9:43](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=583s) services.

[9:45](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=585s) All right. This is one of my favorite

[9:47](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=587s) ones. Um

[9:49](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=589s) to keep your model secure you need to

[9:52](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=592s) have guardrails.

[9:54](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=594s) And with that, guardrails are necessary

[9:57](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=597s) for preventing your services from prompt

[9:59](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=599s) injection attacks, keeping PII filters

[10:03](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=603s) in place, having toxicity filters,

[10:05](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=605s) keeping the LMS to stop swearing at your

[10:08](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=608s) customers, all those good things. But

[10:12](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=612s) just like a model provider, there are

[10:14](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=614s) trade-offs, too. Guardrails are just

[10:17](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=617s) like another service that can go down

[10:20](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=620s) that can be unreliable and that's where

[10:23](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=623s) you need to choose do you fail open or

[10:26](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=626s) do you fail close when I say fail open

[10:29](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=629s) you can still serve the request even if

[10:31](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=631s) your guardrails are down fail close you

[10:34](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=634s) block the request and say hey I'm not

[10:36](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=636s) available that's the trade-off between

[10:38](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=638s) availability and security to certain

[10:40](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=640s) extent while there's a no universal

[10:42](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=642s) answer it really depends on your use

[10:45](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=645s) case you can decide like for example a

[10:48](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=648s) toxicity filter if it's not up and

[10:50](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=650s) running you can still serve that

[10:52](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=652s) request.

[10:54](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=654s) So the default choice should be the

[10:57](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=657s) worst case that you can live with.

[11:03](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=663s) There are a few things that you can

[11:05](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=665s) actually do to improve the behavior of

[11:09](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=669s) your systems in face of uh you know

[11:12](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=672s) guardrails being down and and managing

[11:14](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=674s) just unreliability of the guardrails

[11:16](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=676s) themselves. So the first is time budget.

[11:21](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=681s) Your request should never be bound by

[11:24](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=684s) your guardrail timing. It should always

[11:27](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=687s) be the LM that is the rate determining

[11:29](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=689s) step. So make sure that you have

[11:32](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=692s) timeouts in place and those guardrails

[11:35](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=695s) run with a specific time budget.

[11:38](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=698s) Another important thing is fallback.

[11:41](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=701s) You've heard, you probably know and I've

[11:43](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=703s) talked about it. We always discuss

[11:45](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=705s) fallbacks with regards to model

[11:47](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=707s) providers, but guardrails are critical

[11:50](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=710s) services too where you can consider

[11:53](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=713s) fallbacks, have secondary provider,

[11:55](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=715s) secondary checks, cache decisions uh to

[11:58](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=718s) keep your service available when a

[12:01](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=721s) guardrail provider is down.

[12:04](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=724s) Another interesting choice that pops up

[12:06](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=726s) with regards to guardrails is the

[12:08](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=728s) placement of the guardrails.

[12:11](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=731s) Typically, you can place the guardrail

[12:14](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=734s) in three ways. You can have a pre- hook

[12:17](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=737s) that runs where the guardrail actually

[12:19](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=739s) runs on the input. You can and that's

[12:22](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=742s) probably the safest uh but it does add

[12:24](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=744s) serial latency uh to your requests.

[12:28](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=748s) Another one is in parallel. This is one

[12:30](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=750s) of my favorites, but just to call out,

[12:33](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=753s) streaming wouldn't work well here with

[12:35](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=755s) with parallel. So if you're specially

[12:38](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=758s) producing structured output, please

[12:39](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=759s) don't stream them. Uh try to save your

[12:41](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=761s) latencies and run run these guardrails

[12:44](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=764s) concurrently for your structured

[12:45](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=765s) outputs. Another one is post hooks. The

[12:48](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=768s) these are best for um output monitoring,

[12:52](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=772s) auditing your outputs and and so forth.

[12:58](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=778s) So, so far we've all I've discussed all

[13:02](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=782s) the things that can go wrong with

[13:04](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=784s) regards to our dependencies.

[13:07](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=787s) We haven't discussed that we are

[13:09](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=789s) actually adding another dependency in

[13:11](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=791s) the request path itself which is the

[13:13](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=793s) central or which is the LM gateway

[13:15](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=795s) itself. There are a few things where we

[13:17](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=797s) have been bitten by u and we've learned

[13:19](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=799s) some lessons that I want to share with

[13:21](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=801s) you. If you're working on an LLM gateway

[13:24](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=804s) or using one, one is shared limits.

[13:29](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=809s) Make sure that your API keys are

[13:31](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=811s) segregated per route, per use case to

[13:35](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=815s) the most granular possible uh to the

[13:39](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=819s) most granular thing that you can

[13:40](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=820s) imagine. U

[13:44](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=824s) having a noisy tenant can be one of the

[13:47](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=827s) biggest problems here.

[13:49](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=829s) Another thing is load shedding. This is

[13:52](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=832s) a feature that you should uh as part of

[13:54](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=834s) your runbooks, game days, uh make sure

[13:56](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=836s) that the gateway that you're using

[13:58](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=838s) supports load shedding because when you

[14:01](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=841s) have a retry storm, it becomes really

[14:03](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=843s) hard to just scale out. You cannot

[14:05](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=845s) simply scale out services that is under

[14:07](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=847s) a retry storm and all these web servers

[14:11](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=851s) they have an internal queue and they're

[14:13](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=853s) configurable. Make sure that they're

[14:15](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=855s) bounded and they cannot request they

[14:18](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=858s) cannot accept requests that are

[14:19](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=859s) unbounded. And if you want to have some

[14:21](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=861s) custom logic, you can even have traffic

[14:24](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=864s) prioritization here as well to make sure

[14:26](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=866s) under load your most important use cases

[14:29](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=869s) get served. Well,

[14:34](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=874s) last thing that I wanted to discuss is

[14:37](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=877s) the whole idea of a central gateway

[14:38](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=878s) itself. It is a single point of failure.

[14:41](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=881s) So if you're thinking of having a

[14:43](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=883s) central gateway for your entire company

[14:45](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=885s) for to LLMs, I would recommend rethink

[14:49](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=889s) that and see what are the reasons that

[14:51](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=891s) you want it. What I've noticed is that

[14:54](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=894s) in most scenarios, it's not the central

[14:56](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=896s) gateway that they want. They want

[14:58](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=898s) centralized governance.

[15:00](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=900s) And there is a path forward where you

[15:02](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=902s) can actually decentralize the gateway

[15:05](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=905s) and still centralize government

[15:07](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=907s) governance. So do not try to centralize

[15:11](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=911s) your traffic but you can have plugins,

[15:14](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=914s) you can have custom code that can

[15:16](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=916s) centralize your governance. Uh

[15:18](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=918s) governance can be in the form of cost

[15:20](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=920s) tracking, rate limit managing management

[15:23](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=923s) and there are other solutions possible.

[15:25](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=925s) So explore those before you chart on

[15:28](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=928s) having one central gateway for your

[15:30](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=930s) entire company. It can be managed by a

[15:33](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=933s) single team, but I wouldn't recommend

[15:35](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=935s) deploying it as a single deployment for

[15:39](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=939s) the entire company even though it's

[15:41](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=941s) distributed.

[15:43](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=943s) With that said, I want to end this talk

[15:46](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=946s) on a personal note. So, it is my son's

[15:49](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=949s) birthday today and I'm here talking to

[15:52](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=952s) strangers about circuit breaking. So the

[15:56](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=956s) least you can do for me is please go and

[15:58](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=958s) prevent one incident for me and for your

[16:01](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=961s) customers. Thank you. If you have any

[16:04](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=964s) questions. Yeah.

[16:21](https://www.youtube.com/watch?v=zrZ1amZBSPw&t=981s) >> [music]
