AI Engineer World's Fair 202616:23
Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons — Kanish Manuja, Twilio
Kanish Manuja discusses production LLM gateways as a tradeoff among availability, latency, guardrails and cost. He recommends tested per-request fallbacks, model-specific latency measurements and timeouts, explicit guardrail failure policies, and protection against overload. He also argues that centralized governance can be implemented without routing all…
Kanish Manuja