[
  {
    "startMs": 12760,
    "text": "My name is David. I lead AI operations at a company called Revolut. We have, uh, fifty minutes today together. We'll talk about high-performance engineering and self-hosting open source model on dedicated GPU. How does that sound?"
  },
  {
    "startMs": 31860,
    "text": "Cool. So it's gonna be a technical one, so bear with me. Um, so maybe to start off, because I know this is the US and you may not know Revolut, uh, quick words about our company. We're a digital bank in Europe. We have eighty million customers. We're live in forty-five markets. We're the most downloaded financial app, uh, in Europe for the last year. And as of July, we're, uh, Europe's largest private company. Um,"
  },
  {
    "startMs": 62540,
    "text": "so we're doing okay."
  },
  {
    "startMs": 67120,
    "text": "The, the emphasis of Revolut is, you know, shipping a lot of product, and so there's a high focus on engineering culture and, uh, excellence on that demand. And so we'll talk about that. So let's talk about some AI."
  },
  {
    "startMs": 82920,
    "text": "We have a chart here. Uh, you're gonna see two lines. This is a sample of two thousand engineers' performance over the last year. So the green line is gonna be lead time, so it's gonna be when an engineer picks up a feature until a feature is merged and ready to be deployed. The blue line is gonna be coding time. So the green line is in days, the blue line is in hours."
  },
  {
    "startMs": 108440,
    "text": "Just to reset the timeline in case people forget because AI moves really fast. Claude Code came out around February 2025, right? And that's the Claude Code Sonnet 3.7, the one that would delete your product database, right? Not the safe Claude and Codex that we have today. So bear in mind when we talk about adoption."
  },
  {
    "startMs": 129440,
    "text": "So what you see on this chart is, um, January '25, first introduced AI in the company, right? Um, Gemini. First time people were able to use private data within AI. Around February, we started playing around with Claude Code, and then January is when we first launched Claude Code at scale. What you see is, um, Gemini had ten percent improvements in lead time, twenty percent improvement in coding time, so that people no longer needed to look to Stack Overflow and, uh, reading"
  },
  {
    "startMs": 158940,
    "text": "documentation. But then Claude Code, not so much impact at the time. So when we launched it, we were highly skeptical about the cost of AI. We were seeing engineers spend a thousand a month, ten thousand a month with not so much ROI. And so when we launched it, we launched it with, um, strict budget limits. And so we're looking at the next thing, right? So how can we provide unlimited tokens to our engineers so they can use AI guilt-free for a fraction of the cost? And so we"
  },
  {
    "startMs": 188840,
    "text": "looked into open source. So around April 26 is when we started self-hosting open source model. We're a big fan of GLM for coding. So around that time, what you see, uh, April 26, GLM-5.1,"
  },
  {
    "startMs": 204320,
    "text": "we see sixty percent improvement in, uh, coding time, seventy percent improvement in lead time. So dramatic impact. What do we learn from this? Well, there's more impact on the lead time than on the coding time. So what we find is the bottleneck for engineering is not necessarily coding. You can save a lot of time not having to dig through your inbox or messing with Jira and Confluence. And so when you have unlimited token, you're free to do that. You no longer have to save your tokens for"
  },
  {
    "startMs": 234600,
    "text": "the end of the quarter or, uh, the most difficult task."
  },
  {
    "startMs": 239560,
    "text": "So where do we stand? Well, open source is now a priority at Revolut, and yet economics are favorable. The billing is flat. We don't have any bill shock any of the month. And the frontier is the exception. So obviously, we keep using Codex, Claude Code. The latest model are fantastic. Uh, Gemini are going very well as-- very good as well. But, um, we keep them as exception."
  },
  {
    "startMs": 265020,
    "text": "So how do we get there?"
  },
  {
    "startMs": 268200,
    "text": "Some metrics that matter when you're self-hosting model. So to serve two thousand engineers, we roughly need fifty-six V200, and that's to, uh, host GLM-5.3. GLM is a, a big model, right? Eight hundred billion parameters."
  },
  {
    "startMs": 284640,
    "text": "What you want to aim for is sub three-second time to first token. This is empirical evidence. As soon as you exceed five second, my Slack inbox starts to blow up, and so we know that's the limit."
  },
  {
    "startMs": 299380,
    "text": "In terms of throughput, doesn't really matter as much, right? So we serve in terms of interactivity, two hundred token per second. People are fine. You don't need to go higher. Um, the way people work has changed. We no longer look at the stream of thinking tokens. People run task in parallel and check in when the task is done. But if you wanna have high throughputs and you're an engineer at Revolut, you just connect after five PM when no one is on GPU, and you can get eight hundred TPS, and then, you know, it's a dream."
  },
  {
    "startMs": 328940,
    "text": "In terms of cost, in terms of impact, uh, self-hosting GLM, our blended cost of token is thirty cents per million. And so our invoice is one-tenth of what we were spending on Anthropic and, um, OpenAI."
  },
  {
    "startMs": 346140,
    "text": "So there's a lot of tuning that goes into running open source models. If you just get GPUs and upload weights on a GPU and you start doing some inference, you're gonna get terrible performance. It's because your workload is gonna be very dependent Um, on the parameters that you set. So if you look all the way to the left of that chart, the blue line is gonna be demand, so that's requests per second. That's gonna be what's gonna be the bottleneck for your latency for your GPUs."
  },
  {
    "startMs": 377628,
    "text": "And the red line is what we're optimizing for. That's the time to first token. So when we started back in April, you could see requests per second around three, which is high but not so high, and time to first token over ten seconds, which is very slow. But the data that you see is the same number of GPU. We didn't increase GPU count with L56. And by the end, you see in August, requests per second were about over seven and time to first token"
  },
  {
    "startMs": 407668,
    "text": "were around one point six. So we've had eighty-eight percent improvement in performance for seventy-six percent increased demand. So the learning from this is there's a lot of parameters that you can improve on, and you really need to resist scaling if you're looking to self-host a model. So let's talk parameters. There's no secret recipe, but the things you could consider are the following."
  },
  {
    "startMs": 434688,
    "text": "First one, long context is a scam. You really need to be mindful that even though you can serve one million token, two fifty-six, two fifty-six is really what you need."
  },
  {
    "startMs": 446768,
    "text": "Quantization, if you're running Blackwall generation GPU, you don't want to run anything higher than FP4. And the FP4 is best. Roughly going from FP4 to FP8 is a forty percent saving in GPU footprint for zero loss in, uh, precision."
  },
  {
    "startMs": 465388,
    "text": "Other parameters to consider, tensor parallelism. Even though GLM fits on, uh, TP4, you don't have enough key-value cache  and so maybe a TP8 works better depending on how large your workload is. And then on the more esoteric, um, check your cache hit. Coding task needs to be above ninety percent, uh, cache hit. And you can look into optimized spec decoding, uh, sequential prefill, PDD zag, all of these."
  },
  {
    "startMs": 494048,
    "text": "Cool. So now let's talk about, um, production setup. GLM obviously is a text-only model, which is a limitation for some people. The trade-off, I would say, uh, we would make every time is get the smarter model and figure out how to do image recognition another way. You can easily have a local router. OpenCodex is one where you would route any image recognition to Gemini or, uh, Luna and keep GLM for the main task."
  },
  {
    "startMs": 522788,
    "text": "If you really need some, uh, vision model, GLM5G Flash, uh, works really well. We have, uh, our UI designer, uh, making great use of it. Um, turnaround for, uh, UI product has gone a really, really long way."
  },
  {
    "startMs": 540568,
    "text": "And then the last one is gonna be about, uh, the harness. So all of these endpoints are gonna be OpenAI-compatible API with Responses API, and so you can use them inside OpenCode, uh, Claude, Codex. Codex app as well works really well. And so that's what we, uh, we recommend."
  },
  {
    "startMs": 564048,
    "text": "And then, um, last message, and this one is for the CIO in the room. There's been a lot of stigma in using Chinese, uh, open source model. So the thing to consider is you're not using the inference dedicated endpoint that goes to China. You're just getting the weights and putting them on a US-based service. So for data sharing side, uh, this is totally fine."
  },
  {
    "startMs": 591028,
    "text": "Revolut in particular, because we're a EU-regulated bank, we fall under the most, uh, strict regime in terms of compliance. And so what we find that works is all of these coding tasks are not considered autonomous agent. They're always with a human in the loop. And so how do you give, uh, reassurance that you have a human in the loop? You start by creating a sandbox that guarantees that. And the sandbox means now if you've used Codex Claude Code, it has a lot of"
  },
  {
    "startMs": 621047,
    "text": "these features of approve for me. You just have to set the rules for the operations that you don't want to allow. So you can allow grep but not Gradle, for instance. And that's gonna give confidence, uh, to your regulator that it's not an autonomous agent that needs different, uh, supervision."
  },
  {
    "startMs": 639508,
    "text": "The sandbox also protects you from, uh, rogue use, right? You may have bad actor inside your company, and you want a shield, uh, for that. You'd be surprised what you find in, in a large company. And then the last one is prompt injection. Not, not so much a thing, um, now given how smart models are, but always a thing. In terms of data privacy, we spoke about it. So as long as the data sits in one of the countries that are in the Data Equivalence Framework, so US, EU, UK, Israel, you're"
  },
  {
    "startMs": 669388,
    "text": "fine. You can get your GPUs there. All of your inference provider or GPU provider are gonna have zero data retention by default."
  },
  {
    "startMs": 679848,
    "text": "And then the last thing you need is a very strong, uh, open telemetry so that if you ever have an incident, um, you need to be able to go deep, uh, and look at the logs. Also gives you some, uh, yeah, analytics."
  },
  {
    "startMs": 697588,
    "text": "All right. Thanks, everyone. That was all for my talk."
  }
]