[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 13200,
    "text": "I'm Kanesh Manuja. I'm a principal"
  },
  {
    "startMs": 15679,
    "text": "engineer at Twilio."
  },
  {
    "startMs": 18240,
    "text": "Let's start with a quick show of hands."
  },
  {
    "startMs": 20880,
    "text": "Who here has seen the message, something"
  },
  {
    "startMs": 23600,
    "text": "went wrong. Please try again."
  },
  {
    "startMs": 27439,
    "text": "Well, we have a few lucky ones and a few"
  },
  {
    "startMs": 30720,
    "text": "that have had a good lunch. Um, so"
  },
  {
    "startMs": 34719,
    "text": "behind that simple message is actually a"
  },
  {
    "startMs": 37200,
    "text": "system that is very complex"
  },
  {
    "startMs": 40399,
    "text": "that serves you that message despite the"
  },
  {
    "startMs": 42640,
    "text": "model providers being down."
  },
  {
    "startMs": 45200,
    "text": "And that's what we're going to"
  },
  {
    "startMs": 46320,
    "text": "productionize today or discuss"
  },
  {
    "startMs": 48160,
    "text": "productionizing today."
  },
  {
    "startMs": 50399,
    "text": "So what is an LM gateway? An LLM gateway"
  },
  {
    "startMs": 53600,
    "text": "is an entry point or a middleware"
  },
  {
    "startMs": 55680,
    "text": "between your apps and the model"
  },
  {
    "startMs": 57840,
    "text": "providers behind them. It does a bunch"
  },
  {
    "startMs": 60320,
    "text": "of things. Routing, authentication,"
  },
  {
    "startMs": 62559,
    "text": "fallback, rate limits, all kinds of"
  },
  {
    "startMs": 64640,
    "text": "governance that you can think of."
  },
  {
    "startMs": 68560,
    "text": "And right at the heart of the gateway is"
  },
  {
    "startMs": 71520,
    "text": "a fight between four things. It's"
  },
  {
    "startMs": 73920,
    "text": "availability, latency, your guardrails"
  },
  {
    "startMs": 77200,
    "text": "and costs."
  },
  {
    "startMs": 78960,
    "text": "In case of a degradation, you cannot"
  },
  {
    "startMs": 81520,
    "text": "maximize all four. You need to pick what"
  },
  {
    "startMs": 84640,
    "text": "you want. So with this talk, if you use"
  },
  {
    "startMs": 89119,
    "text": "an LLM gateway, I want you I want to"
  },
  {
    "startMs": 92000,
    "text": "help you to make that trade-off for your"
  },
  {
    "startMs": 94479,
    "text": "use case. And if you design a gateway, I"
  },
  {
    "startMs": 97439,
    "text": "want you to design or provide those"
  },
  {
    "startMs": 99439,
    "text": "levers to your callers and customers u"
  },
  {
    "startMs": 103280,
    "text": "so that your customers are happy."
  },
  {
    "startMs": 106720,
    "text": "Let's start with availability."
  },
  {
    "startMs": 110560,
    "text": "If you have a single model provider,"
  },
  {
    "startMs": 114479,
    "text": "their ceiling is your ceiling. Their"
  },
  {
    "startMs": 117600,
    "text": "outage is your outage."
  },
  {
    "startMs": 123439,
    "text": "So in typical software engineering, the"
  },
  {
    "startMs": 126079,
    "text": "way you tackle unreliable dependency is"
  },
  {
    "startMs": 128800,
    "text": "by retrying."
  },
  {
    "startMs": 131360,
    "text": "Retrying with exponential backoffs, with"
  },
  {
    "startMs": 134560,
    "text": "jitters. And when all of that fails, you"
  },
  {
    "startMs": 137520,
    "text": "have a circuit breaker that trips after"
  },
  {
    "startMs": 139280,
    "text": "you've seen sufficient failures and you"
  },
  {
    "startMs": 141040,
    "text": "stop calling the damn thing."
  },
  {
    "startMs": 144400,
    "text": "This is not enough for LLMs. LLMs are"
  },
  {
    "startMs": 147440,
    "text": "very different compared to your fast"
  },
  {
    "startMs": 149760,
    "text": "cheap APIs that you retry on. Retrying"
  },
  {
    "startMs": 153360,
    "text": "an LLM API eats into your latency budget"
  },
  {
    "startMs": 157280,
    "text": "really fast. And also tripping over a"
  },
  {
    "startMs": 160239,
    "text": "circuit breaker when you have another"
  },
  {
    "startMs": 163680,
    "text": "perfectly fine model provider to route"
  },
  {
    "startMs": 165760,
    "text": "to doesn't make sense. You should use"
  },
  {
    "startMs": 167760,
    "text": "the second model provider. And third, as"
  },
  {
    "startMs": 171440,
    "text": "I said, the calls are slow and"
  },
  {
    "startMs": 173760,
    "text": "expensive. So blind retries just"
  },
  {
    "startMs": 176080,
    "text": "multiply your cost and your tail"
  },
  {
    "startMs": 177760,
    "text": "latencies."
  },
  {
    "startMs": 180800,
    "text": "So what is a better idea here? It is"
  },
  {
    "startMs": 183040,
    "text": "actually a per request fallback. What"
  },
  {
    "startMs": 186000,
    "text": "that means is you can actually try model"
  },
  {
    "startMs": 188640,
    "text": "provider A and then in sequence try"
  },
  {
    "startMs": 191680,
    "text": "model provider B if your request to"
  },
  {
    "startMs": 193360,
    "text": "model provider A fails. Another option"
  },
  {
    "startMs": 196159,
    "text": "to consider here is you can fire"
  },
  {
    "startMs": 198159,
    "text": "requests to both the providers in"
  },
  {
    "startMs": 199599,
    "text": "parallel. But that's only if you're"
  },
  {
    "startMs": 201920,
    "text": "highly highly obsessed with latencies"
  },
  {
    "startMs": 205040,
    "text": "because that's just going to double your"
  },
  {
    "startMs": 206480,
    "text": "cost."
  },
  {
    "startMs": 208560,
    "text": "Some of the similar circuit breaking"
  },
  {
    "startMs": 210319,
    "text": "patterns apply here to LMS as well. If"
  },
  {
    "startMs": 214080,
    "text": "you know that your primary has been"
  },
  {
    "startMs": 217360,
    "text": "failing for some time, it doesn't make"
  },
  {
    "startMs": 219920,
    "text": "sense to try it again. You put it, you"
  },
  {
    "startMs": 222879,
    "text": "take it out of the load balancer or your"
  },
  {
    "startMs": 225120,
    "text": "request path and put it in a cool down"
  },
  {
    "startMs": 228080,
    "text": "and then after a few minutes have"
  },
  {
    "startMs": 230239,
    "text": "passed, try putting that back again."
  },
  {
    "startMs": 233840,
    "text": "One interesting choice that you have to"
  },
  {
    "startMs": 236080,
    "text": "make here is where your failure counts"
  },
  {
    "startMs": 238879,
    "text": "live."
  },
  {
    "startMs": 240400,
    "text": "You can decide to have the failure"
  },
  {
    "startMs": 241760,
    "text": "counts live in memory on the instances"
  },
  {
    "startMs": 244560,
    "text": "that are serving your traffic or you can"
  },
  {
    "startMs": 247200,
    "text": "have shared infra where your failure"
  },
  {
    "startMs": 251040,
    "text": "counts are shared across the fleet."
  },
  {
    "startMs": 253680,
    "text": "There are trade-offs."
  },
  {
    "startMs": 255840,
    "text": "If you want quick failovers, then"
  },
  {
    "startMs": 258639,
    "text": "fleetwide helps. And with instance uh"
  },
  {
    "startMs": 262400,
    "text": "with local state counters the issue that"
  },
  {
    "startMs": 265199,
    "text": "you run into is whenever you change your"
  },
  {
    "startMs": 266880,
    "text": "deployment size your configuration and"
  },
  {
    "startMs": 269520,
    "text": "your expectations change. So something"
  },
  {
    "startMs": 272240,
    "text": "to consider."
  },
  {
    "startMs": 275360,
    "text": "What that clean diagram did not really"
  },
  {
    "startMs": 277040,
    "text": "show you are some of the other gotchas"
  },
  {
    "startMs": 279120,
    "text": "that I'm going to discuss. So fallbacks"
  },
  {
    "startMs": 281440,
    "text": "are not transparent."
  },
  {
    "startMs": 283360,
    "text": "While the industry is converging on an"
  },
  {
    "startMs": 285280,
    "text": "OpenAI API compatible format, I would"
  },
  {
    "startMs": 289120,
    "text": "say there are still nuances. So you need"
  },
  {
    "startMs": 290880,
    "text": "to really test your fallbacks well. They"
  },
  {
    "startMs": 293680,
    "text": "can have differences in your tool"
  },
  {
    "startMs": 295840,
    "text": "calling schemas, token limits, stop"
  },
  {
    "startMs": 297520,
    "text": "reasons and what have you. So with LM"
  },
  {
    "startMs": 299840,
    "text": "gateways, you can have a normalization"
  },
  {
    "startMs": 302400,
    "text": "layer that can ensure that you can do"
  },
  {
    "startMs": 305440,
    "text": "cross provider fallbacks as well."
  },
  {
    "startMs": 308960,
    "text": "Another thing is streaming"
  },
  {
    "startMs": 314800,
    "text": "it."
  },
  {
    "startMs": 316320,
    "text": "So essentially nobody wants to wait for"
  },
  {
    "startMs": 320320,
    "text": "30 seconds to have a wall of text appear"
  },
  {
    "startMs": 322320,
    "text": "in front of them. So there are use cases"
  },
  {
    "startMs": 324479,
    "text": "where streaming is absolutely required."
  },
  {
    "startMs": 327280,
    "text": "But it comes as at a cost. You trade"
  },
  {
    "startMs": 329039,
    "text": "away your levers. You cannot once you"
  },
  {
    "startMs": 331759,
    "text": "have decided to go with provider A, you"
  },
  {
    "startMs": 334479,
    "text": "have to continue going with provider A."
  },
  {
    "startMs": 337520,
    "text": "You cannot mid-stream change the"
  },
  {
    "startMs": 339680,
    "text": "providers. Whatever has been sent to the"
  },
  {
    "startMs": 342000,
    "text": "client, it's done. And that's where the"
  },
  {
    "startMs": 345520,
    "text": "something uh went wrong message, that's"
  },
  {
    "startMs": 347680,
    "text": "the one that you see. It's not because"
  },
  {
    "startMs": 350080,
    "text": "of laziness. It's by design uh that you"
  },
  {
    "startMs": 352800,
    "text": "see that and it's one of the trade-offs."
  },
  {
    "startMs": 355680,
    "text": "I would like to call out one other thing"
  },
  {
    "startMs": 357600,
    "text": "where I've seen teams trip over and over"
  },
  {
    "startMs": 360320,
    "text": "again. They really provision and test"
  },
  {
    "startMs": 363520,
    "text": "their primary providers really well, but"
  },
  {
    "startMs": 367199,
    "text": "they the second provider, the fallback"
  },
  {
    "startMs": 369199,
    "text": "provider doesn't necessarily get the"
  },
  {
    "startMs": 370800,
    "text": "same level of love. And I would argue"
  },
  {
    "startMs": 373520,
    "text": "that your throughputs or your capacity"
  },
  {
    "startMs": 375680,
    "text": "or your headroom should be even higher"
  },
  {
    "startMs": 378560,
    "text": "for the second provider or the fallback"
  },
  {
    "startMs": 381280,
    "text": "provider because that's your last line"
  },
  {
    "startMs": 383520,
    "text": "of defense. If that goes down, your"
  },
  {
    "startMs": 385919,
    "text": "application goes down."
  },
  {
    "startMs": 390479,
    "text": "Let's discuss latencies."
  },
  {
    "startMs": 392639,
    "text": "Availability failures are right in your"
  },
  {
    "startMs": 394560,
    "text": "face. They fail. You get alarmed. You"
  },
  {
    "startMs": 398319,
    "text": "get paged. But high latencies can be the"
  },
  {
    "startMs": 402080,
    "text": "quiet ones. And they need to receive"
  },
  {
    "startMs": 405360,
    "text": "more love um than I would say tuning"
  },
  {
    "startMs": 407919,
    "text": "your services for just availability."
  },
  {
    "startMs": 414319,
    "text": "One thing to call out, a gateway may run"
  },
  {
    "startMs": 418080,
    "text": "mixed workloads"
  },
  {
    "startMs": 420080,
    "text": "and you can have embedding embedding"
  },
  {
    "startMs": 422639,
    "text": "requests that takes just less than a"
  },
  {
    "startMs": 424639,
    "text": "second. You can have classification"
  },
  {
    "startMs": 426080,
    "text": "requests that take less than a second."
  },
  {
    "startMs": 428479,
    "text": "Uh you have chat requests taking 3"
  },
  {
    "startMs": 430319,
    "text": "seconds and reasoning requests taking a"
  },
  {
    "startMs": 433039,
    "text": "long time."
  },
  {
    "startMs": 435120,
    "text": "Quick show of hands. If you measure"
  },
  {
    "startMs": 438479,
    "text": "your aggregate latency for your entire"
  },
  {
    "startMs": 440639,
    "text": "service."
  },
  {
    "startMs": 442880,
    "text": "Well, that was a trick question. Sorry."
  },
  {
    "startMs": 444479,
    "text": "You shouldn't. It doesn't make sense."
  },
  {
    "startMs": 445919,
    "text": "It's a lie. You should be tracking your"
  },
  {
    "startMs": 448720,
    "text": "P99 per model per route, not a gateway"
  },
  {
    "startMs": 452000,
    "text": "wide number. Gateway wide number doesn't"
  },
  {
    "startMs": 454240,
    "text": "make sense, especially if you're running"
  },
  {
    "startMs": 456000,
    "text": "mixed workloads. And I hope you're not u"
  },
  {
    "startMs": 458720,
    "text": "for those who raise your hand. Another"
  },
  {
    "startMs": 461680,
    "text": "thing that can really I cannot emphasize"
  },
  {
    "startMs": 464479,
    "text": "this enough is for you to set timeouts"
  },
  {
    "startMs": 467280,
    "text": "on per model class per route."
  },
  {
    "startMs": 470880,
    "text": "That's where that's the number one root"
  },
  {
    "startMs": 472880,
    "text": "cause of your silent outage. If you"
  },
  {
    "startMs": 474960,
    "text": "don't have a timeout, your gateway"
  },
  {
    "startMs": 477120,
    "text": "thinks you're hap your request is being"
  },
  {
    "startMs": 478800,
    "text": "happily served while it is not. And I'll"
  },
  {
    "startMs": 482160,
    "text": "leave you with this message for for"
  },
  {
    "startMs": 483840,
    "text": "latencies. Um, specifically a reasoning"
  },
  {
    "startMs": 487120,
    "text": "models normal is actually a chat models"
  },
  {
    "startMs": 490080,
    "text": "outage. So you definitely need to track"
  },
  {
    "startMs": 492800,
    "text": "latency per route."
  },
  {
    "startMs": 497120,
    "text": "Okay, this is the most painful or this"
  },
  {
    "startMs": 499360,
    "text": "the slide that has given me the most"
  },
  {
    "startMs": 500960,
    "text": "scarse which is reasoning and router"
  },
  {
    "startMs": 504160,
    "text": "models. So this is where truly the"
  },
  {
    "startMs": 508080,
    "text": "latency is unpredictable"
  },
  {
    "startMs": 510400,
    "text": "and reasoning models they do not give"
  },
  {
    "startMs": 514640,
    "text": "you"
  },
  {
    "startMs": 516240,
    "text": "they they're highly undeterministic more"
  },
  {
    "startMs": 518159,
    "text": "deterministic undeterministic than your"
  },
  {
    "startMs": 520000,
    "text": "normal models. You cannot set the"
  },
  {
    "startMs": 522080,
    "text": "temperature to zero in many cases and"
  },
  {
    "startMs": 524800,
    "text": "the same prompt can take somewhere from"
  },
  {
    "startMs": 527680,
    "text": "2 seconds to 60 seconds and we've seen"
  },
  {
    "startMs": 529519,
    "text": "that in production where P99 suddenly"
  },
  {
    "startMs": 531920,
    "text": "popped to 60 seconds for no good reason."
  },
  {
    "startMs": 534720,
    "text": "So that's"
  },
  {
    "startMs": 536399,
    "text": "while there's no magical solution to it."
  },
  {
    "startMs": 539200,
    "text": "I would recommend that you at least"
  },
  {
    "startMs": 541040,
    "text": "start with fixing the reasoning level"
  },
  {
    "startMs": 543680,
    "text": "per route. So with router models, they"
  },
  {
    "startMs": 547519,
    "text": "hide that abstraction behind you. Like"
  },
  {
    "startMs": 549920,
    "text": "they pick which models to run and I"
  },
  {
    "startMs": 552720,
    "text": "would highly recommend that you at least"
  },
  {
    "startMs": 555120,
    "text": "make as much uh you make requests as"
  },
  {
    "startMs": 558320,
    "text": "determinist deterministic as possible"
  },
  {
    "startMs": 560880,
    "text": "with an undeterministic system."
  },
  {
    "startMs": 564320,
    "text": "Another idea is hedging the tail. You"
  },
  {
    "startMs": 568000,
    "text": "can have a you can fire another request"
  },
  {
    "startMs": 570959,
    "text": "if your primary request actually"
  },
  {
    "startMs": 572720,
    "text": "consumed let's say P90 of your latency"
  },
  {
    "startMs": 575360,
    "text": "budget."
  },
  {
    "startMs": 577920,
    "text": "This can hedge the t this can really"
  },
  {
    "startMs": 579680,
    "text": "hedge the P99 tail u for for your"
  },
  {
    "startMs": 583279,
    "text": "services."
  },
  {
    "startMs": 585680,
    "text": "All right. This is one of my favorite"
  },
  {
    "startMs": 587519,
    "text": "ones. Um"
  },
  {
    "startMs": 589920,
    "text": "to keep your model secure you need to"
  },
  {
    "startMs": 592800,
    "text": "have guardrails."
  },
  {
    "startMs": 594640,
    "text": "And with that, guardrails are necessary"
  },
  {
    "startMs": 597600,
    "text": "for preventing your services from prompt"
  },
  {
    "startMs": 599519,
    "text": "injection attacks, keeping PII filters"
  },
  {
    "startMs": 603279,
    "text": "in place, having toxicity filters,"
  },
  {
    "startMs": 605920,
    "text": "keeping the LMS to stop swearing at your"
  },
  {
    "startMs": 608640,
    "text": "customers, all those good things. But"
  },
  {
    "startMs": 612560,
    "text": "just like a model provider, there are"
  },
  {
    "startMs": 614720,
    "text": "trade-offs, too. Guardrails are just"
  },
  {
    "startMs": 617839,
    "text": "like another service that can go down"
  },
  {
    "startMs": 620560,
    "text": "that can be unreliable and that's where"
  },
  {
    "startMs": 623120,
    "text": "you need to choose do you fail open or"
  },
  {
    "startMs": 626560,
    "text": "do you fail close when I say fail open"
  },
  {
    "startMs": 629680,
    "text": "you can still serve the request even if"
  },
  {
    "startMs": 631600,
    "text": "your guardrails are down fail close you"
  },
  {
    "startMs": 634320,
    "text": "block the request and say hey I'm not"
  },
  {
    "startMs": 636160,
    "text": "available that's the trade-off between"
  },
  {
    "startMs": 638160,
    "text": "availability and security to certain"
  },
  {
    "startMs": 640720,
    "text": "extent while there's a no universal"
  },
  {
    "startMs": 642959,
    "text": "answer it really depends on your use"
  },
  {
    "startMs": 645040,
    "text": "case you can decide like for example a"
  },
  {
    "startMs": 648320,
    "text": "toxicity filter if it's not up and"
  },
  {
    "startMs": 650240,
    "text": "running you can still serve that"
  },
  {
    "startMs": 652399,
    "text": "request."
  },
  {
    "startMs": 654399,
    "text": "So the default choice should be the"
  },
  {
    "startMs": 657680,
    "text": "worst case that you can live with."
  },
  {
    "startMs": 663680,
    "text": "There are a few things that you can"
  },
  {
    "startMs": 665680,
    "text": "actually do to improve the behavior of"
  },
  {
    "startMs": 669279,
    "text": "your systems in face of uh you know"
  },
  {
    "startMs": 672399,
    "text": "guardrails being down and and managing"
  },
  {
    "startMs": 674880,
    "text": "just unreliability of the guardrails"
  },
  {
    "startMs": 676720,
    "text": "themselves. So the first is time budget."
  },
  {
    "startMs": 681360,
    "text": "Your request should never be bound by"
  },
  {
    "startMs": 684800,
    "text": "your guardrail timing. It should always"
  },
  {
    "startMs": 687200,
    "text": "be the LM that is the rate determining"
  },
  {
    "startMs": 689680,
    "text": "step. So make sure that you have"
  },
  {
    "startMs": 692640,
    "text": "timeouts in place and those guardrails"
  },
  {
    "startMs": 695680,
    "text": "run with a specific time budget."
  },
  {
    "startMs": 698880,
    "text": "Another important thing is fallback."
  },
  {
    "startMs": 701120,
    "text": "You've heard, you probably know and I've"
  },
  {
    "startMs": 703440,
    "text": "talked about it. We always discuss"
  },
  {
    "startMs": 705760,
    "text": "fallbacks with regards to model"
  },
  {
    "startMs": 707680,
    "text": "providers, but guardrails are critical"
  },
  {
    "startMs": 710399,
    "text": "services too where you can consider"
  },
  {
    "startMs": 713040,
    "text": "fallbacks, have secondary provider,"
  },
  {
    "startMs": 715120,
    "text": "secondary checks, cache decisions uh to"
  },
  {
    "startMs": 718079,
    "text": "keep your service available when a"
  },
  {
    "startMs": 721600,
    "text": "guardrail provider is down."
  },
  {
    "startMs": 724160,
    "text": "Another interesting choice that pops up"
  },
  {
    "startMs": 726320,
    "text": "with regards to guardrails is the"
  },
  {
    "startMs": 728639,
    "text": "placement of the guardrails."
  },
  {
    "startMs": 731839,
    "text": "Typically, you can place the guardrail"
  },
  {
    "startMs": 734720,
    "text": "in three ways. You can have a pre- hook"
  },
  {
    "startMs": 737519,
    "text": "that runs where the guardrail actually"
  },
  {
    "startMs": 739200,
    "text": "runs on the input. You can and that's"
  },
  {
    "startMs": 742000,
    "text": "probably the safest uh but it does add"
  },
  {
    "startMs": 744399,
    "text": "serial latency uh to your requests."
  },
  {
    "startMs": 748000,
    "text": "Another one is in parallel. This is one"
  },
  {
    "startMs": 750720,
    "text": "of my favorites, but just to call out,"
  },
  {
    "startMs": 753519,
    "text": "streaming wouldn't work well here with"
  },
  {
    "startMs": 755440,
    "text": "with parallel. So if you're specially"
  },
  {
    "startMs": 758320,
    "text": "producing structured output, please"
  },
  {
    "startMs": 759920,
    "text": "don't stream them. Uh try to save your"
  },
  {
    "startMs": 761920,
    "text": "latencies and run run these guardrails"
  },
  {
    "startMs": 764160,
    "text": "concurrently for your structured"
  },
  {
    "startMs": 765839,
    "text": "outputs. Another one is post hooks. The"
  },
  {
    "startMs": 768720,
    "text": "these are best for um output monitoring,"
  },
  {
    "startMs": 772320,
    "text": "auditing your outputs and and so forth."
  },
  {
    "startMs": 778959,
    "text": "So, so far we've all I've discussed all"
  },
  {
    "startMs": 782480,
    "text": "the things that can go wrong with"
  },
  {
    "startMs": 784399,
    "text": "regards to our dependencies."
  },
  {
    "startMs": 787920,
    "text": "We haven't discussed that we are"
  },
  {
    "startMs": 789519,
    "text": "actually adding another dependency in"
  },
  {
    "startMs": 791440,
    "text": "the request path itself which is the"
  },
  {
    "startMs": 793279,
    "text": "central or which is the LM gateway"
  },
  {
    "startMs": 795360,
    "text": "itself. There are a few things where we"
  },
  {
    "startMs": 797760,
    "text": "have been bitten by u and we've learned"
  },
  {
    "startMs": 799920,
    "text": "some lessons that I want to share with"
  },
  {
    "startMs": 801680,
    "text": "you. If you're working on an LLM gateway"
  },
  {
    "startMs": 804560,
    "text": "or using one, one is shared limits."
  },
  {
    "startMs": 809120,
    "text": "Make sure that your API keys are"
  },
  {
    "startMs": 811680,
    "text": "segregated per route, per use case to"
  },
  {
    "startMs": 815440,
    "text": "the most granular possible uh to the"
  },
  {
    "startMs": 819440,
    "text": "most granular thing that you can"
  },
  {
    "startMs": 820639,
    "text": "imagine. U"
  },
  {
    "startMs": 824480,
    "text": "having a noisy tenant can be one of the"
  },
  {
    "startMs": 827680,
    "text": "biggest problems here."
  },
  {
    "startMs": 829839,
    "text": "Another thing is load shedding. This is"
  },
  {
    "startMs": 832240,
    "text": "a feature that you should uh as part of"
  },
  {
    "startMs": 834480,
    "text": "your runbooks, game days, uh make sure"
  },
  {
    "startMs": 836959,
    "text": "that the gateway that you're using"
  },
  {
    "startMs": 838720,
    "text": "supports load shedding because when you"
  },
  {
    "startMs": 841440,
    "text": "have a retry storm, it becomes really"
  },
  {
    "startMs": 843360,
    "text": "hard to just scale out. You cannot"
  },
  {
    "startMs": 845040,
    "text": "simply scale out services that is under"
  },
  {
    "startMs": 847440,
    "text": "a retry storm and all these web servers"
  },
  {
    "startMs": 851040,
    "text": "they have an internal queue and they're"
  },
  {
    "startMs": 853680,
    "text": "configurable. Make sure that they're"
  },
  {
    "startMs": 855839,
    "text": "bounded and they cannot request they"
  },
  {
    "startMs": 858240,
    "text": "cannot accept requests that are"
  },
  {
    "startMs": 859920,
    "text": "unbounded. And if you want to have some"
  },
  {
    "startMs": 861839,
    "text": "custom logic, you can even have traffic"
  },
  {
    "startMs": 864000,
    "text": "prioritization here as well to make sure"
  },
  {
    "startMs": 866160,
    "text": "under load your most important use cases"
  },
  {
    "startMs": 869040,
    "text": "get served. Well,"
  },
  {
    "startMs": 874160,
    "text": "last thing that I wanted to discuss is"
  },
  {
    "startMs": 877199,
    "text": "the whole idea of a central gateway"
  },
  {
    "startMs": 878800,
    "text": "itself. It is a single point of failure."
  },
  {
    "startMs": 881440,
    "text": "So if you're thinking of having a"
  },
  {
    "startMs": 883680,
    "text": "central gateway for your entire company"
  },
  {
    "startMs": 885600,
    "text": "for to LLMs, I would recommend rethink"
  },
  {
    "startMs": 889199,
    "text": "that and see what are the reasons that"
  },
  {
    "startMs": 891199,
    "text": "you want it. What I've noticed is that"
  },
  {
    "startMs": 894079,
    "text": "in most scenarios, it's not the central"
  },
  {
    "startMs": 896800,
    "text": "gateway that they want. They want"
  },
  {
    "startMs": 898480,
    "text": "centralized governance."
  },
  {
    "startMs": 900800,
    "text": "And there is a path forward where you"
  },
  {
    "startMs": 902800,
    "text": "can actually decentralize the gateway"
  },
  {
    "startMs": 905440,
    "text": "and still centralize government"
  },
  {
    "startMs": 907199,
    "text": "governance. So do not try to centralize"
  },
  {
    "startMs": 911440,
    "text": "your traffic but you can have plugins,"
  },
  {
    "startMs": 914399,
    "text": "you can have custom code that can"
  },
  {
    "startMs": 916800,
    "text": "centralize your governance. Uh"
  },
  {
    "startMs": 918560,
    "text": "governance can be in the form of cost"
  },
  {
    "startMs": 920240,
    "text": "tracking, rate limit managing management"
  },
  {
    "startMs": 923040,
    "text": "and there are other solutions possible."
  },
  {
    "startMs": 925360,
    "text": "So explore those before you chart on"
  },
  {
    "startMs": 928160,
    "text": "having one central gateway for your"
  },
  {
    "startMs": 930880,
    "text": "entire company. It can be managed by a"
  },
  {
    "startMs": 933279,
    "text": "single team, but I wouldn't recommend"
  },
  {
    "startMs": 935680,
    "text": "deploying it as a single deployment for"
  },
  {
    "startMs": 939199,
    "text": "the entire company even though it's"
  },
  {
    "startMs": 941199,
    "text": "distributed."
  },
  {
    "startMs": 943680,
    "text": "With that said, I want to end this talk"
  },
  {
    "startMs": 946959,
    "text": "on a personal note. So, it is my son's"
  },
  {
    "startMs": 949120,
    "text": "birthday today and I'm here talking to"
  },
  {
    "startMs": 952399,
    "text": "strangers about circuit breaking. So the"
  },
  {
    "startMs": 956399,
    "text": "least you can do for me is please go and"
  },
  {
    "startMs": 958720,
    "text": "prevent one incident for me and for your"
  },
  {
    "startMs": 961759,
    "text": "customers. Thank you. If you have any"
  },
  {
    "startMs": 964079,
    "text": "questions. Yeah."
  },
  {
    "startMs": 981201,
    "text": ">> [music]"
  }
]