[
  {
    "startMs": 12920,
    "text": "My name is Noyan. I'm the head of AI engineering at Optiver, and today I'm gonna talk to you a little bit about how we scale AI with P&L on the line. Like, what does that mean though, right? So first, I think to contextualize, uh, let me tell you a little bit about what we do at Optiver. So we're a market maker. Uh, what that means is that we facilitate trading. Uh, we're a trading firm. We trade with our own money, um, and we don't have clients. Uh, basically, you know, we... i-in the tech sense, we take a lot of data, you know, we build models, and then we make decisions with them,"
  },
  {
    "startMs": 42860,
    "text": "and these decisions need to work in a very high-performance manner. You know, we make ten million plus trades a day, pricing a million plus instruments, and petabytes of data processed daily. And also, we're a pretty global firm. We have ten plus locations across the world, um, and that comes with, you know, certain implications that we'll get into for a second.  So typically, if you look at a traditional tech company, you can think about how they think about, you know, how they measure how much money they make. Uh, it's something along the lines of you implement some change, you roll it out to users, and then users use it"
  },
  {
    "startMs": 72800,
    "text": "for days and weeks, et cetera, and you collect some engagement metrics. Uh, and then over, you know, months and quarters, maybe years, you'll look at like revenue, longtime value, et cetera. I used to be at Apple, um, so I know how this works.  Um, but, um, at Optiver, we implement something, uh, we roll out that change to our traders and our, our quants, and literally within hours you can see the P&L impact. In fact, you know, at the top of every day, we basically zero out the balance sheet and then look by the end of the day how much money we made. So this has certain"
  },
  {
    "startMs": 102180,
    "text": "implications around how, you know, we make decisions around how we want to scale AI and how do we wanna build things.  First of all, the feedback loop is much more immediate, right? Like, when you make a decision, by the end of the day, you'll start seeing the impact of that decision. Of course, like over long term, there are instruments that we trade that are, you know, longer horizon and there's more value to discover over long term, but immediately you'll start seeing the impact of something. But the other side of that coin is that if you miss an opportunity, then that means that opportunity loss is potentially pretty costly, especially"
  },
  {
    "startMs": 132120,
    "text": "if, you know, there's like a special thing going on in the market. Let's say that there's a SpaceX AI IPO or something like that, and you miss out on that, then that's probably, you know, pretty costly. Which means that we need to make decisions pretty quickly. Uh, we need to be able to measure them really effectively and really efficiently, and we gotta have a bias for action. Um, that means like the ceiling of the value and, and importance of a decision, decision you make is really, really high. Um, so in the, in the case of like a service, like a user, uh, facing service business like Spotify or whatever, like"
  },
  {
    "startMs": 161980,
    "text": "fundamentally they're competing for a finite amount of attention of the customers versus with your trading, the more you trade, the more money you make, right? So, you know, the economics are a little bit more different. So to accomplish this, uh, and to be able to scale and operate at this way, we basically needed to build our own AI platform, um, around a few core principles. Move fast. I mean, we talked about that already, but of course, part of moving fast is that being able to be modular because, you know, in the AI world every day"
  },
  {
    "startMs": 191880,
    "text": "there's a new model, there's a new product, there's a new paradigm. We're doing loops. No, we're doing swarms. No, we're doing graphs, et cetera. And we need to be able to account for that. And if that paradigm is valuable, like measure it. And if it is, bring it in because the moment you bring it in, you're gonna start, you know, getting more value. And of course, to be able to understand that we need to be able to measure at every layer. You know, of course like trading means lots of backtesting, et cetera, but also, you know, we need to... When a new model comes out, yes, there's of course external benchmarks as to how this model is performing, but we also need to be"
  },
  {
    "startMs": 221800,
    "text": "able to understand how in our internal benchmarks these things perform because the data distribution of the benchmarks that we have is different from what they have a-available externally. Now, the thing is traders are pretty smart people. They're pretty high agency people, so we want our platform to be very self-service. But of course, anytime you provide something purely self-service, it can be really chaotic, so we also need to be able to provide good defaults. Uh, and one way we kind of have this, um, way of working with them is also being able to forward deploy. So we have a fairly"
  },
  {
    "startMs": 251540,
    "text": "decently sized AI engineering team and we take our people, we have them sit down with the traders, work with them, understand how they trade, teach them how AI can add more value to them, learn and bring it back to the platform. Now, of course, you've all heard the stories of like OpenAI Hugging Face, et cetera. So of course, as we're doing more and more with AI, we also need to make sure that we're doing in a safe manner. Uh, as you can imagine, uh, there's potentially, uh, pretty negative impacts if the agents kind of go haywire and do stuff with our stack. Um, but I do wanna underscore that everybody in the"
  },
  {
    "startMs": 281520,
    "text": "company is using AI. Like everybody's using it a little bit differently, but fundamentally we're very, very, um, AI native I guess is the term that we're using now. Um, and of course, because we're a highly regulated industry and we value our IP deeply, everything that we do is on-prem. So we have our own on-prem GPUs, our on-prem infrastructure, et cetera. The only thing that we phone home outside of our service really is for like third-party model inference. And of course, in those cases we have like things like zero data retention, et cetera, agreements with the companies."
  },
  {
    "startMs": 311220,
    "text": "So here's kind of like a very, you know, shock and awe, like a lot of information on the sl-slide type of view of our AI platform. I don't expect you to, you know, digest all of this, but you know, if you wanna take a photo and talk about it to me afterwards, uh, be my guest. I'll flash the slide again at the end. But broadly, the things that I wanna highlight is that, um, we, uh, have to be accountable for every layer of the stack. At the bottom, you know, we have models from third-party providers. We also self-host, uh, our own models and the open source models. We have an inference gateway that provides"
  },
  {
    "startMs": 341060,
    "text": "things like routing, uh, between these different models. Um, we can, uh, measure how much we're spending on each of these models. We can assign budgets, et cetera. I'll go to that in, in a second. And of course, this gives us resilience. Like if one provider goes down, we can always swap out to a different provider, et cetera, because again, any time that we're down, you know, that's basically we're not making money. Um, and then we have our own agent platform where Uh, agents have, uh, registered, uh, capabilities like skills, plugins, tools. Each agent in each team with"
  },
  {
    "startMs": 371060,
    "text": "using an agent can bring in their knowledge and memory systems. We have an agent gateway, which is basically like a s- single ledger of all of the agents in the company being able to come in together, communicate, orchestrate humans and agents with similar identity management, uh, et cetera. So this kind of goes into things like the agent swarm, et cetera, concepts that you also may have heard about. And finally, what's really important to me, I'm a product guy as well, is that, um, we have the right surfaces for our users to engage with the agents because you can build a great platform, but if nobody wants to, likes to, or knows how to"
  },
  {
    "startMs": 400980,
    "text": "use it, then it doesn't really matter. So we've built our own kind of, um, web interface for people to access like their sessions, et cetera. Uh, we have this thing called our Agent Workbench, which is basically persona management. Teams can kind of build their own profile of how they use agents and share. Uh, we have a personal assistant that everybody in the company, uh, has access to. Um, yeah. So again, I'll get back to this later, uh, but if y'all have questions, we can talk about it afterwards. Right. So how do we move on these principles? Uh, one principle that we have is that,"
  },
  {
    "startMs": 430980,
    "text": "you know, if a new frontier model is made available publicly, uh, we wanna be able to leverage it in the production flows the same day because there's immense value in that if it is valuable. But is it valuable, right? We wanna be able to benchmark it on golden sets. So that means that we need to have a mature evaluation framework, um, and then we partner with key customer trading teams where we collect their benchmarks, we onboard it onto them, and whenever we get a new model, we run the model on these new benchmarks, immediately know the metrics, and can share with the f- folks in the company to see, hey, is this really valuable for our use"
  },
  {
    "startMs": 461000,
    "text": "cases? But additionally, you know, anytime there's a different model, usually that means that we have to make the model available in our gateway. We have the right APIs, interfaces, we have the cost management, et cetera, set up correctly, and oftentimes new models mean, like, the harnesses need to be updated, et cetera, and we also manage the update and delivery of that as well. And then of course, yes, we launch it, but then that also comes with the expectations. We're a global company, so we need to have round-the-clock support. So our team, you know, has, in different regions, uh, support rotations to make sure that the uptime is"
  },
  {
    "startMs": 490700,
    "text": "high. I think some people here in the audience are actually on support today, but anyway. Um, and then of course, like, we run these benchmarks. We also need to be able to have, um, the people in the company understand like, okay, we looked into this model, we benchmarked it, here's how well it works, here's how you use it, here's our recommendation, and kind of keep that communication line open."
  },
  {
    "startMs": 510680,
    "text": "Um, another thing is that because we have optionality between different models, uh, if any model is adding value, we wanna integrate it, uh, and we need to be able to insulate against model availability in, like, different regions, uh, or different, you know, cloud providers, et cetera. Uh, and also we have a requirement that some of our workloads run on a completely DMZ, uh, on-prem zone. Uh, so we also-- There's cost reasons for that. There is, um, reasons of isolation for that. So for those reasons, we wanna be able to be agnostic to the model and harness. So which means that"
  },
  {
    "startMs": 541400,
    "text": "we have the third-party providers, uh, we have on-prem models, and all of it kind of goes through our inference gateway. We log everything. Uh, every interaction going with the models is logged, and we also automatically m- move them to our evaluation system. Uh, we have a bunch of different harnesses that we support, and on top of that, we have a meta harness layer that basically lets teams configure their experience on top of that and have some portability between these different harnesses. Um,  so teams can define their own harness configs in this meta harness layer, and that plugs"
  },
  {
    "startMs": 571360,
    "text": "into the team's knowledge base. Automatically, knowledge is harvested over time. We look at the traces and then derive ways to, uh, get additional knowledge as well. Uh, and then through this meta harness, you can either deploy managed agents on our platform, or you get access to them through the web portal IDE UI that I described earlier. I see some people taking photos, so I'll give it a second. All right.  So, um, this type of chart may be familiar to a bunch of you. Uh, this is kind of a"
  },
  {
    "startMs": 601380,
    "text": "roughly illustrated version of how much we're spending on AI. No numbers on it. Uh, but we are doing more with it. And this is, you know, in December, you see like the Claude Code moment and things go up. By the way, this inflection point here is, like, when I joined the company, so I don't know if I have anything to do with that. Um, and then, um, then we have here, you know, like, the big frontier model releases of the summer. As you can see, like, spend really shot up there. But then we kinda got it back under control because we built some optimizations into the process that helped us really get this thing, uh, in check."
  },
  {
    "startMs": 631560,
    "text": "So, um, the thing is, the more we spend, the more value we can get. We are pretty confident o- that we can, you know, keep getting returns, but we wanna make sure that we're spending wisely, right? So one thing that we discovered was that, um, y- we have a bunch of MCPs, we have a bunch of knowledge bases and information, et cetera, um, and on a f- fresh session, we were seeing that, you know, sometimes people have like 200K tokens worth of context, like, by default minimum, sometimes even more. That's, that's ridiculous, right? Because everybody was kinda dumping a bunch of stuff in there. This is the kind"
  },
  {
    "startMs": 661580,
    "text": "of problem with, like, doing s- things completely self-service, right? We need to have sane defaults. So we implemented a bunch of different changes. Uh, we brought in MCP tool search. Uh, we gave the agents the option to, um, rather than calling MCPs, have their own CLI that's wrapping around a bunch of the MCPs, which is, like, much more efficient. Um, we looked at the caching characteristics of, like, when caches are getting invalidated. How long do we need to set these parameters? Do we need to set them differently? Do we set them dynamically? Um, and then we worked with the individual teams to kind of come up with"
  },
  {
    "startMs": 691500,
    "text": "further optimizations to their harnesses. Um, you know, we- do we put some things into the knowledge base rather than this, like, massive, um, system prompt, uh, et cetera? Uh, and the good thing about these improvements is that, um, these things all scale with token cost, right? So basically, um, if the tokens are more expensive, you're saving more on this. But the point here is that, um, yes, you can dump models out there and make them available to people, and I'm not even g- going into the work that we're doing around things like, you know,"
  },
  {
    "startMs": 721560,
    "text": "model routing, um, or sub-agent optimizations, et cetera. Sheer single session stuff, um, there's almost always value left on the table and we're, you know, ruthless- ruthlessly optimizing towards that."
  },
  {
    "startMs": 735530,
    "text": "Yeah, there we go. And we shipped optimizations at this point in time, but of course, there's always, like, rollout, et cetera. Uh, our tech stack is a little bit interesting when it comes to rolling out across the different global regions. Uh, but then also other things that really helped us spend less was that, um, turns out that if you tell people, like, how much they're spending and, like, give them a little bit more visibility, a little bit more alerting, give them the agency to self-manage, they actually do a pretty good job, you know? We built a system to basically have teams kind of set their own alerting and limits, and we've seen, like, many teams,"
  },
  {
    "startMs": 765670,
    "text": "you know, basically set reasonable limits for themselves. Another option is, you know, exploring open source models. Uh, you know, there are a bunch of open source models that are, on token cost, pretty competitive with the frontier models out there. It's not necessarily right for every use case, uh, but we've found opportunities where they're a good value replacement, and we've been basically making that available internally, uh, and we've seen more and more people, uh, leverage those. Then of course, like, there's things like model routing, et cetera. There's, um... We've been"
  },
  {
    "startMs": 795650,
    "text": "exploring different approaches to dynamically send the prompt to, like, a cheaper model, et cetera, or sometimes have a way where, um, we have the, like, plan generated by, like, a more frontier model and the implementation be broken down by smaller models, et cetera. Um, and finally, you know, I think knowledge bases are underrated. Uh, which, I mean, they're very important, but they also help get spend in control if you do your knowledge base effectively because then you're retrieving less context fundamentally if you just retrieve the right amount of context. Um, so"
  },
  {
    "startMs": 825810,
    "text": "long story short is that we're still investing and using a lot of tokens, but this is just about getting more ROI out of the same tokens."
  },
  {
    "startMs": 834990,
    "text": "And finally, I kinda wanna talk about, uh, meeting the users where they're at. Um, so a few months ago, we were in this place where, like, a bunch of people are using different harnesses. They have all these weird bespoke setups to kind of tmux into their dev boxes, and some people were building, you know, hacky web UIs, et cet- et cetera, whatever. So basically what we said was like, \"Hey, you know, there's a bunch of people doing these types of things. Why don't we just build a nice version of this and then make it available to everybody in the company?\" So that's exactly what we did. In our web portal, we built this basically system where you can manage your"
  },
  {
    "startMs": 864710,
    "text": "sessions, you can have chat windows, you can have, like, uh, artifacts in there. Um, there are similar products out there offered by, you know, you know, some different, uh, harness companies, but, uh, we feel that this worked very well with our infrastructure, um, and we can kind of iterate it at the rate that we would like to. Um, so basically, this was actually, uh, also at a, at a place where different traders in the company are now kinda contributing code. They find something that's valuable to their workflow, and then they, they just send up a"
  },
  {
    "startMs": 894490,
    "text": "PR and we just update it, and then now we have things like, you know, collaborative document editing and things like that. And in fact, we're kind of getting this to a place where, uh, we're doing this, you know, agent multiplayer thing where now I have my sessions in there. We can create rooms with, like, my agents, your agents, like, other deployed agents, and we can kind of have all of that in one place, and this really helps, uh, people be more productive and, uh, you know, we have now a Slack client, et cetera, so you can also do it on the go and whatnot. So"
  },
  {
    "startMs": 924230,
    "text": "basically, um, our users are very high agency. Uh, they will go and build something if we don't build it for them, so we just build a better version of it."
  },
  {
    "startMs": 934290,
    "text": "Yeah. So, uh, basically, this is, um, the overall overarching philosophy of how we did this. Um, we have a platform, very modular, uh, and, um, we continue to iterate on it, but so far we're having a pretty good time. Um, that's the end of the talk, uh, but if you have questions about anything on this or any other area, we can talk. And by the way, we have a networking event, uh, this evening. Uh, please come to our booth and register. It's gonna be fun. We have cool prizes."
  },
  {
    "startMs": 971990,
    "text": "Questions?"
  },
  {
    "startMs": 973750,
    "text": "Can you talk a little bit more about what your knowledge base looks like? Are you using stock and storage vector search?"
  },
  {
    "startMs": 979390,
    "text": "Yeah. So we've, um, definitely been exploring a f- bunch of different options. Um, personally, I'm not happy with any of the options out there. There's trade-offs, but I mean, if you ask, like, some of the frontier labs what you do for knowledge base, they have, like, just huge markdown files, hierarchical markdowns, et cetera. Um, there's places where that works, but, um, there is, um... Vector search has its own benefits and disadvantages. We have a hybrid approach that kind of, uh, allows us to explore these different options."
  },
  {
    "startMs": 1009270,
    "text": "Personally, I want us to spend more time around, like, doing things like perhaps some graph databases, et cetera, that are going to be, uh, more valuable. I mean, every day on Twitter I see something, like, new way of, like, m- handling memory and knowledge. Um, that's a problem that we want to spend more time on. I'm not happy with our solution."
  },
  {
    "startMs": 1030910,
    "text": "Go ahead."
  },
  {
    "startMs": 1031710,
    "text": "Um, what do you think is-- what are your biggest value gains from, like, either hand-built versus Codex solutions? Like, what kind of"
  },
  {
    "startMs": 1041070,
    "text": "Yeah. Good question. Um, value gain from using hand-built solutions versus Codex. So there's a few things. Um, I mean, one kinda boring answer is that because everything is kind of in our own infrastructure, a lot of these tools don't work in the way that you would want them to. Um, un- until recently, Codex didn't really have the option to kind of run managed agents, et cetera, or some of these things on your own infrastructure. Now they've started offering them, and we've been working with them on it actually. But, um, that was one reason why we started this"
  },
  {
    "startMs": 1071130,
    "text": "way. Secondly, I think the, um, most-- many of the places that offer these types of solutions also try to kind of lock you into either their model or some rest of their solution. But the moment we kind of go down that path, then we end up in a place where if our infrastructure is overly invested in one particular provider, we move at the speed that they let us move. Uh, we've actually encountered with, um, that with one of the, uh, inference gateway providers that we used to use. Um, and"
  },
  {
    "startMs": 1101070,
    "text": "basically, we would encounter scenarios where something comes out, we need it, and they're like, \"Okay, uh, let's talk to our GTM team. Let's put you on the roadmap. We'll...\" And it's like, and then by that point it's like, okay, no, we need to actually move faster. Um, so that's also one of the reasons. Uh, and finally, I think the In some of these areas, we've been, um, consistently ahead of the curve. Like talking to some of the frontier labs, um, the, the way that we've been doing managed agent"
  },
  {
    "startMs": 1130682,
    "text": "capabilities, et cetera, like they're asking us to tell them how we did it.  Not to be bragging too much. But I think the, um, we feel pretty confident in our engineering and kind of our roadmap, so that's also, uh, been well. I don't think this approach works well for everybody, but I think we have a unique, um, coalition of circumstances that allow for us to do that. Go ahead."
  },
  {
    "startMs": 1153022,
    "text": "Uh, can you talk a bit about your tool chain in terms of, uh, best practices, like integration to, um, the other part of this platform such as eval or memory?"
  },
  {
    "startMs": 1164622,
    "text": "Um, yeah. So ju- just to make sure I understand your question correctly, um, what are some of the best practices of how we integrate the rest of this into eval and memory?"
  },
  {
    "startMs": 1173002,
    "text": "Uh, culture tool shift. So the scale of function."
  },
  {
    "startMs": 1176542,
    "text": "Right. Um, so I think the-- we started at a place where, you know, we log everything. Like I said, every tool call input, output, et cetera, uh, just hotel logging, and, um, that goes into our, uh, evaluation system. Um, the challenge that we found with that, and we-- it's, uh, kind of an open problem, I would say, is that, um, when you're kind of r- trying to evaluate like a tool call, I mean, we have like, for example, a support bot agent that we run in Slack,"
  },
  {
    "startMs": 1206882,
    "text": "um, and we want to evaluate that agent. Let's say, what if we ran this agent with a different model? Well, the support bot is leveraging a bunch of tools that depend on the state of the world that were different at the time when you ran it, and that's the pretty, like, typical problem. Um, we've been working on basically things like can we snapshot world states with these tools? And depending on the system that it integrates with, it's, you know, there's different answers to that question. If you're looking at some, like, trading backtest database, easy like that. You can always point in time. That's what we do. Um, but if you're looking at some other system,"
  },
  {
    "startMs": 1236902,
    "text": "like if you're looking at Slack, for example, what was the state of Slack when this was originally evaluated? Maybe there are some ways to kind of replace Slack for a particular point in time, but other systems, not really. So we kind of try to assign the confidence value as to how well we can point in time this particular tool invocation, uh, and we still log it as such. And when we're rerunning the eval, we will run the tool with kind of saying, \"Okay, I want kind of this snapshot.\" We have like a proxy around the MCPs. Um, but you know, it's imperfect and,"
  },
  {
    "startMs": 1266762,
    "text": "uh, I don't know if there's a perfect solution other than just record, you know, everything."
  },
  {
    "startMs": 1273702,
    "text": "Yeah. Do we have-- Maybe we have ti- time for one more? Go ahead."
  },
  {
    "startMs": 1278762,
    "text": "So let's say you built some agent for some team, um, and they really like it. Now they want to scale, uh, uh, drastically the use case."
  },
  {
    "startMs": 1291602,
    "text": "Yeah."
  },
  {
    "startMs": 1291902,
    "text": "How do you system design that or how do you think about that?"
  },
  {
    "startMs": 1296342,
    "text": "That's a very classic problem that we've actually run into several times. Uh, you know, we had this one team that ha- built an agent that did like coding and, uh, like analysis with them. Uh, it was kind of scoped to their repo, and it was like, well, what if we also ha- worked with this team and have their thing also come with us and immediately kind of explodes in complexity. Now it's checking out multiple repos at a different auth, et cetera. Like typical problem. Um, so initially our solution was explore, okay, what if we like literally just did that? What if we just made the new agent that just had"
  },
  {
    "startMs": 1326262,
    "text": "a union of the functionality? But oftentimes that ends up, you know, a lot of tech debt, works very poorly, et cetera. I think our current answer to that type of paradigm is, obviously every agent is versioned, et cetera. So you can, if it's small increments, you can push a new version to the agent and make, make new capabilities. But going forward, we would more encourage people to basically leverage our agent gateway, where rather than one agent having like a ten different things that it does, scope your agents to like a task that they're good at, you can easily measure them at, you can easily"
  },
  {
    "startMs": 1356422,
    "text": "manage the life cycle of it. And then there'll be another agent that it can talk to easily so as if it was, you know, kind of operating on itself because we kind of have this construct of agents being able to go through one ledger, spin up each other, et cetera. And that will help us, um, scale these types of functionality more effectively rather than just bloating one agent to do everything. Uh, it becomes kind of like a multi-agent orchestration problem, but, you know, that's-- I think keeping things more measurable, uh, is really"
  },
  {
    "startMs": 1385942,
    "text": "important because if you really bloat the agent, then you can end up in scenarios where like, you know, some of the functionality gets better, some gets worse, and you can't really, you know, tw- tweak it easily."
  }
]