The State of Model Routing — NVIDIA, Cognition, OpenRouter

Talk transcript

1. 0:00

   [upbeat music] Have been really exciting. We've tried to get a bunch of the industry leaders together to talk about some of the problems that are-- we're facing as we try to run more on local.

2. 0:21

   If you guys were here for the first panel, one of the things that we talked about was model routing. We firmly believe that we're in a mo- a multi-model world.

3. 0:29

   I think you heard this from many of the panelists. Anyone who is deploying AI in production and who is doing so locally is seeing that multi-model world. That's why we released these Nemotron models at NVIDIA.

4. 0:40

   Everything is released from the datasets to the weights with recipes so that you can customize them. We do that because we know that people customizing models is gonna be huge.

5. 0:49

   And so this panel is really exciting 'cause we're gonna talk specifically about model routing. So as you are picking which model to use, how does that-- how-- essentially, how does that tooling itself look?

6. 1:00

   Um, do you guys wanna introduce yourselves?

7. 1:03

   Yeah, sure. Um, I'm Walden. I'm the co-founder of Cognition. We build Devin AI software engineer. Uh, in addition to the product, we spend a lot of time partnering with our customers to figure out how they should deploy these models, uh, and these agents.

8. 1:15

   And one of the things they're constantly asking us nowadays is basically, "How do I know the ROI of our models? And how do I know which tasks I can actually let our engineers spend the most expensive models on versus, you know, letting them use a more cost-efficient model?"

9. 1:29

   And so that's why we're, we're also thinking a lot more about multi-model routing nowadays.

10. 1:32

    Totally. Yeah.

11. 1:33

    I'm Carter. You guys heard from me a little bit earlier, but if you weren't here, I'm a developer tech engineer at NVIDIA, and, uh, ultimately, I spend a lot of time thinking about how to get intelligence into as many developers' hands as possible.

12. 1:46

    And, uh, something that is continually becoming a-- not an issue, but something that is top of mind for a lot of developers is, uh, as you use more intelligence and the frontier models get more expensive, uh, it becomes somewhat cost-prohibitive to, to use the best, uh, tools-- what feels like the best tools as much as you would

13. 2:03

    like to use them. And so this has become a recent, uh, you know, focus is how can we, you know, still get the same desired outputs, but actually, uh, both as an individual developer, but also imagine startups and small companies, how can you leverage this incredible tool without totally breaking the bank?

14. 2:21

    Uh, I'm Tanay. I work on model evaluations, uh, both in terms of its accuracies and efficiency and cost understanding of the model. Uh, and then I try and understand those-- uh, implement those learnings and, uh, help build a router.

15. 2:38

    So it's basically, uh, my job is to understand the behavior of, um, of the model on an intimate level and then use those learnings to both improve the model and try and design a system of model that can work together with each other.

16. 2:52

    Totally. Um, yeah, I love a lot of the research that you're doing at NVIDIA as we kind of see the space through. I think what's really interesting is model routing itself is pretty new still.

17. 3:01

    And so what you'll notice is there isn't a very clear solution here. That was something that came up on the first panel, is that there is a lot of space for startups and for companies in the ecosystem to fill in a solution here 'cause we're still figuring out how to best do these patterns.

18. 3:15

    And I think, Walden, I wanna kinda ask you. So, um, Cognition just released Fusion, your guys' model router.

19. 3:21

    Yeah.

20. 3:21

    And when you guys released it, you in your blog said that you're actually getting better performance than Fable, um, than these frontier models. And I feel like that was a very surprising statement to hear because we're thinking that you're getting as good or close enough usually when we're running on Edge, when we're running local in these compute

21. 3:38

    strain, smaller footprint models, but you guys are getting better. Can you explain how? [chuckles]

22. 3:44

    Yeah, absolutely. Um, so I, I, I also wanna be, be clear about something here is like, you know, we're not saying that like we gap above Fable-level performance-

23. 3:50

    Totally. Yeah, yeah

24. 3:51

    ... in the same way that maybe Fable-level performance gaps above other models. I think actually there's this really unintuitive, um, dynamic where smarter models actually get better and better at delegating work.

25. 4:03

    And so one of the philosophies we had with building a model router is we don't want to route people to a dumber model, and then suddenly you're stuck with a model that doesn't know how to do your task, and next thing you know, you're, you're switching yourself back to smarter model anyways and now taking that expensive cost.

26. 4:18

    Uh, in general, we think a lot of the existing model routing systems out there are probably the same ones people have been using like a year ago. And so we really wanted to put out a new framework that actually lets people still feel like and, and still have a frontier model in their system while getting all these

27. 4:33

    like cost benefits. So yeah, we, we're reducing the cost of Fable-level intelligence by forty percent. The way we do that is we allow Fable to still do like the planning and the, the hard decision-making, but delegate a lot of the work to an implementation model.

28. 4:46

    Mm.

29. 4:47

    Um, and the implementation model can be, you know, one of these open source models, be it a, a cheaper mini model. Um, the unintuitive thing is, even though it's cheaper, because you're delegating the work to another model, you can let that model go at the task with much more depth and intensity than you might otherwise.

30. 5:01

    Like, you can spin off like, you know, three sub-agents to go and like explore the code base. And maybe that's actually more, uh, comprehensive than if you had just let Fable explore the code base itself.

31. 5:10

    So you're actually getting-

32. 5:11

    Mm

33. 5:11

    ... this like nice trade-off where it's both more cost efficient and also more comprehensive overall.

34. 5:15

    Interesting. I see. So you're saying, um, by using a bunch of smaller models, you're essentially like, for one example, scouring the code base. You can, you, you can explore it potentially better than if you were to just have one model, I don't know, figure out with what it's limited-- with its limited context, with whatever path it's on.

35. 5:31

    Yeah, totally.

36. 5:32

    But, but also if you think about the budget of if you were to say the frontier model costs this amount per token and the smaller model is this amount per, per token and it's significantly cheaper, then you can use a lot more tokens from the smaller model-

37. 5:45

    Oh, that's right. Yeah, I see

38. 5:45

    ... with still within the budget that it would've been from the frontier model.

39. 5:48

    I would also like to encourage everyone to think, uh, there are jagged capabilities in most models, right? So coding is not one domain. Within, let's say, data visualization, uh, there'll be scikit-learn, there'll be matplotlib, there'll be something else.

40. 6:02

    Uh, it largely come downs-- comes down to the- ... cor- training corpora that, and, uh, that went into each of the models, so, right? So one model, while you're trying to do X type of work, let's say data visualization, and the other type is Y.

41. 6:18

    That means, let's say, model building. Let's say you're trying to have a data science work stream, or you're trying to optimize for some kind of prediction and then visualizing, uh, your results.

42. 6:29

    Within that task, uh, different models will have different strengths. So not nec- it's not necessary that model A, if it scores higher on a coding benchmark, is just plain better at every task there is.

43. 6:42

    So routing is a task of intimately, in- intimately understanding the behavior of and strengths and weaknesses of different models and then applying them thusly, right? I would, I would encourage everyone to think, "Hey, models are strong at different things," rather than like, there's one model to rule them all.

44. 7:02

    I see. And by the way, real quick, thank you, Alex, for joining. Yeah. [laughs]

45. 7:05

    Sorry I'm late.

46. 7:05

    No. Oh, is that still...

47. 7:08

    I might need yours.

48. 7:09

    Sorry I'm late. [laughs] [laughs] I'm Alex from OpenRouter. Um, thanks for having me, Nav.

49. 7:14

    Yeah, of course. Thank you so much. He came right from the airport, so this is perfect. I think, um, Tanay, that's, that's super interesting. So, um, the way that you're thinking through, through model routing, it's not even just, uh, delegating to necessarily a smaller model, but like...

50. 7:26

    And maybe this is kind of what you're saying, is can you, can you put essentially a swarm of agents to accomplish the same task and suddenly routing the task between them is, is much, is a, is a problem to solve in and of itself.

51. 7:36

    Yeah. So if you look at, like, let's say, let's take an easy example. Let's take a science or, like, scientific discovery a- as an example, right? Usually these are one-shot problems.

52. 7:46

    It's ex- incredibly hard. You have, uh, models think throughout this process, right? So in that, you have tons of subdomains, like tons and tons and tons. So in that aspect, if you think about post-training, uh, like the post-training process of a model, they'd be tuned with different teachers, they'd be tuned on different subtasks.

53. 8:09

    Uh, so those, those, um, those overlapping strengths will be readily apparent when you're trying to understand failures of each models on different, different subtasks. Once you understand that, you can orchestrate your system to leverage that arbitrage essentially, and that essentially becomes free.

54. 8:28

    So, uh, I think this is on, uh, LM router bench. There was, there are tons of benchmarks out there, uh, but if you use these techniques, you can get like up to 10% higher accuracy even, right?

55. 8:39

    It depends on the model pool, depends on the task at hand. But I would encourage, uh, to think about the complementary nature of models.

56. 8:48

    I see. Do you see-- So, um, in kind of the way that you were describing the way the task gets broken up, do you see that the, some of the smaller models, because the token cost is cheaper, are they using more tokens?

57. 8:58

    Like is it, are, are you specifically routing so that they do, uh, that they, so that they are chattier?

58. 9:03

    Uh, oh, yeah. I mean, they, they absolutely do use more tokens. Um, I, I actually, I, I wanna, uh, kind of like riff on something that Tanay was saying, which is like, you know, a, a lot of times when you look at these different benchmarks, you'll see that the small models will perform better than like even the

59. 9:17

    frontier models in certain cases. Um, I think a lot of people, they look at this, and they immediately jump to, "Oh, how can we just route like, you know, the task where the smaller models do better just straight to the smaller mo- models?"

60. 9:28

    I, I think that one of the themes we really want to emphasize with our recent, uh, blog post and recent dev infusion was that this kind of like naive, like initial routing to...

61. 9:37

    based on the task type is extremely fragile, uh, especially the more agentic the task you, you work on is. So for example, like a real developer, you might ask your agent first, "Oh, how does this code base work?"

62. 9:48

    And then you, you go deeper and you're like, "Okay, actually, can you implement some features for me?" And then you go deeper, and it's like, "Oh, can you like now go do like a live test of this feature and debug deep cases?"

63. 9:56

    Uh, the complexity changes and the type of task changes over time, and you don't want to be left with some subpar model for the, the task that you're now on.

64. 10:04

    I think this is why people like frontier models so much, is they're, they're just like generally intelligent, and they're capable of shifting between various different domains, even if you can eke out better performance in very specific tasks.

65. 10:15

    Um, and, and so the challenge is: How do you get a small model to know that it's out of its depth, and you need to now like go switch to another model or go like, you know, go to a smart, smarter model?

66. 10:26

    And, and our solution to this is you kind of just always have this like main frontier agent that's watching, even if it's not the one doing the work. It should at least be keeping tabs to figure out, "Okay, wait, like the, the agent I delegated to now is like out of its depth.

67. 10:38

    I need to kind of like move it to, to something else." Um, and, and overall, just the guarantee of always having frontier intelligence present, I think reduces the, the fragility of, of these systems quite a lot.

68. 10:48

    How does the sharing of context between one of those smaller agents who has basically completed up to some level of a task and decides, "Actually, I don't think I'm the right person for this.

69. 10:59

    I need to hand it back to the, the foundational model." Um, of course you don't wanna have the entire trace of that-

70. 11:05

    Yeah

71. 11:05

    ... smaller agent be passed back to the larger model. So how do you get that level of specificity while, uh, basically providing the information it needs, but not more?

72. 11:14

    Yeah, absolutely. So I, I, I think like the, the context here is it's actually very easy to accidentally create a system that's more expensive as soon as you're running like, you know, multiple models together.

73. 11:23

    'Cause, oh no, like, you know, this one file reading, now every, every one of these models is now reading this one file reading, so you, you're getting charged like three times as much.

74. 11:30

    That's-

75. 11:30

    Um, the, the, the trick that we, we spend a lot of our time on is, um, most of the context by default will only be going to like one model.

76. 11:37

    So like most of the context, let's say, will be going to the small model. But the thing you need to then tune very well is, okay, like maybe you still show like what files it's reading.

77. 11:44

    Maybe you show like the high-level thinking of what it's doing back to the main model. Maybe you have the small model, um, you tune its ability to present the context back to the main model.

78. 11:53

    Um, and actually the, a lot of these problems already have been well studied in many domains already, like context compaction is something you already have to solve if you want to do like really long-running agents.

79. 12:03

    And so this problem of taking long context, compacting it in a way that is now understandable is the one that you can also apply to this domain and just kind of give the compacted context back to, back to the main agent.

80. 12:13

    Context compaction is something that, you know, I, I'm familiar with, but I hadn't really thought about as you're doing model routing and as you're trying to share context across now potentially many models, you're, you're expanding the amount of what could be seen as wasteful tokens or redundant tokens just be- just because you have to process that across

81. 12:29

    the many models.

82. 12:30

    Yeah. Yeah. Um, I, I, I think there's-- Uh, the way I describe it is I think we are early in, in the, in the model routing domain. Um-

83. 12:38

    Totally.

84. 12:38

    I, I, I hope that a year from now that even the techniques we kind of used for Devin Fusion, you'll be able to look back on that and like, "Oh, these are some, like, really legacy ideas," and, and now we have, like, much better methods at routing between models.

85. 12:49

    Um, and when people actually start co-designing their models with this in mind, we're, we're gonna ha- be, be in a much better world.

86. 12:55

    Yeah. I, I echo, uh, what you say, right? Um, I think routing

87. 13:02

    will, uh, evolve as the task evolves when, when you start the task, right? So, um, it's more useful to see things in terms of sub-tasks and sessions, uh, than individual problems that you're trying to solve.

88. 13:15

    Because more than likely you're a- when you're working through a problem, you're asking a lot of questions, you're, you're exploring different things. Um, and it, it is imperative that you try-- Like, uh, people who design routers, it's imperative that they try and understand these phases, uh, of different complexities and then try and a-apply some logic, uh, for

89. 13:39

    essentially, uh, sidekicking, uh, uh, tasks or, or, uh, leveraging expertise from other models. Uh, that's, that's pretty on point.

90. 13:49

    Yeah. I, I'd love to hear from the, the, the router guy. [laughs] [laughs] [laughs]

91. 13:53

    Yeah. Um, I, I think those are-- these are, like, important points, and one of the biggest, um, debates I think we have internally is whether that outer model that's doing the orchestration should be the big model or the small model.

92. 14:07

    Yeah.

93. 14:07

    Mm-hmm.

94. 14:08

    Um, you get, like, very different results depending on your choice. Um, and in fa- and it's not even clear what the pricing impact would be. 'Cause if your outer model that's doing the orchestration is the big model, it can leverage its caching to, like, make more of its decisions-

95. 14:22

    Mm.

96. 14:22

    Um, and its caching is going to be, like, a dramatic price savings compared to the small model's caching a lot of the time-

97. 14:29

    Yeah

98. 14:29

    ... um, especially for, like, perform-- for, you know, uh, issues that are on the bright line. Like, zooming out a little bit, I think the-- what, what you want from all the models out there when you do model fusion is to benefit from all the data that is being trained on across all the labs and, and not

99. 14:50

    just the data from one lab, right, or one source. And a, a model is just, like, a combination of, like, the data and its understanding of the data, both its compute and the quality of its RL.

100. 15:02

     So, um, long term, I think you want, you want mo- uh, models where they know that, "Oh, this is, like, in distribution. Like, this is in my data." You can use small models pretty easily, um, and get a cost savings.

101. 15:18

     But if it's out of distribution, small models may actually increase your cost because of how often they'll, like, call tools and how crazy the loops will be. Like, if you run Terminal Bench on Opus and Haiku, like, Opus will do about three times better at one-tenth the cost of Haiku- [laughs] [laughs] ...

102. 15:37

     even though Haiku is significantly cheaper per token. So it really becomes a huge problem if you use a too small of a model, particularly on tasks that are out of domain for the training data.

103. 15:49

     When you're doing something like classifying text, like, "Hey, is this, like, a person's name or is this an organization's name?" That's super in domain, so you would ne- you don't want that kind of task to go to a large model.

104. 16:00

     You want it to go to a s- a small model. Everyone has that in their domain. Um, so d- like, being able to understand in domain, out of domain is a lot of work that we're doing for, for, um, Open Router Fusion and, uh, and then also figuring out, like, what, like, how to orchestrate the outer and

105. 16:16

     inner models, um, for different types of tasks. And, like, um, you know, it's, uh, it's an early industry. It's, like, an early field of research. Uh, most research on model fusion has not been very detailed, not been very, like, you know, optimistic sometimes.

106. 16:32

     Um, it's only, like, just, you know, recently getting, uh, more optimistic, and I think, um, I'm personally very optimistic about it. And, uh, you know, it, like-- We-- We're a very, like, ecosystem-driven collaborative company, and a lot of our-- Like, we work with a lot of partners to try to, like, help improve their orchestration pipelines with good

107. 16:52

     primitives, like the, you know, sub-agent and, like, uh, the advisor tool, um, which is kinda similar to what you were talking about.

108. 16:59

     I'm curious. So, uh, help me understand. Y- When, uh-- If a s-- It makes total sense that, um, a small model, if it's in domain, would be cheaper, but if it's not, then it's gonna thrash around, uh, as it tries to get an answer.

109. 17:12

     When you're describing whether the, uh, con-- like, the main agent should be the lo- the, the local model or the cloud model,

110. 17:22

     does that-- is that d- a decision that's then dependent on whether the task is something that it, that's in domain or not? Does my question kinda make sense?

111. 17:30

     I don't know. Um, I basically, I don't, uh-- It's kinda early to-

112. 17:36

     Yeah

113. 17:36

     ... um, to say. I, I think the, uh, what, what-- The results that we published, which are, uh, a couple weeks ago, which were focused on deep research, not coding-

114. 17:47

     Mm-hmm

115. 17:47

     ... um, we had the smart model be the wrapper model and be the outer model, and we got the best results from doing that. Um, but it, it, uh, for deep research, it works the best.

116. 18:01

     Um-

117. 18:01

     Mm

118. 18:02

     ... for other tasks, it's, like, kind of unclear. Like we-- There-- It's-- Uh, fusion is not super well optimized for coding, and, uh, it might be that, like, a smaller model ends up being, like, a higher, um, efficiency per-- or, you know, fewer dollars, um, per, like, completed, successfully completed task.

119. 18:21

     But it's kinda early to say.

120. 18:22

     Yeah.

121. 18:23

     Yeah.

122. 18:23

     One thing you said earlier is, like, oh, you get the caching benefit from, like, the mainline agent. Um, you actually can get the caching benefit from the side agent.

123. 18:30

     And, um, this is actually one of the key things we talked about with our Devin Fusion launch, is that, um, you kind of are leaving a lot on the table if you do a main agent and sub-agent type system- Um, s- so we don't use sub-agents.

124. 18:43

     We use what we call a sidekick, which is, um, one sub-agent that continually has a running context. So the main agent doesn't need to re-provide a context from earlier.

125. 18:52

     Um, it's all still in the KV cache, right? Like, it's 10 times cheaper on, on all those cache tokens. Um, and then if you want to, like, switch the smart model to be, like, the one on the side or the one, uh, in charge, um, it's actually totally fine, and you can kind of like do, do the

126. 19:06

     swapping back and forth. We're also spending a lot of time right now thinking about

127. 19:11

     how do you train models to actually work collaboratively with other models. I actually, I think there's a lot of literature out there on how you RL one model to do a task end-to-end.

128. 19:20

     How can you RL a model to also be good at collaboration? And when we think about it, we actually try both of these setups where let's RL the model being the orchestrator and the one deciding what gets delegated to other models, see how well that performs.

129. 19:32

     Um, and we also orchestrate it in a way where the model we're training is actually the executor, the sidekick, and see how well is it at executing other models' instructions.

130. 19:42

     Um, and we expect that to, to be a, probably a big lift in this next step of, like, multi-model orchestration is don't just, like, take models as they are and orchestrate them, but, like, can you actually co-design your models with the-

131. 19:52

     Interesting

132. 19:52

     ... orchestration system.

133. 19:53

     Yeah, that makes sense. I mean, with Nemotron and, you know, with all the foundational models, we're, um, we're essentially post-training them for the harnesses that they're getting used.

134. 20:00

     Mm-hmm.

135. 20:00

     If the harness is gonna include a lot of routing, then that makes sense that that makes its way into the post-training.

136. 20:05

     Yeah. Um, are you guys thinking a lot about the kind of like the model training at, at NVIDIA for these kind of purposes?

137. 20:12

     Yeah. So we have a technology called Flextron.

138. 20:16

     Hello?

139. 20:16

     Oh, the mics go.

140. 20:18

     Trying.

141. 20:18

     Hello? [laughs]

142. 20:20

     Testing. Oh, this one works.

143. 20:21

     Oh, wow.

144. 20:21

     All right. [laughs] [laughs]

145. 20:23

     Okay. Uh, so we have a technology called Flextron. So you have, we, we have a setup where, uh, there's a, the, uh, there's a main model, then we distill it, uh, into smaller, uh, footprints.

146. 20:36

     And then based on the, based on the task at hand, you can switch which model does the decoding, right? So there's a lot of fancy stuff you can do, uh, within a model artifact to, uh, to essentially only activate a class of model or a section of weights depending on the task at hand or the complexity at

147. 20:58

     hand. Most, in most cases, you can essentially understand the novelty of a question to a model if you have access to the recipe, uh, with which it was trained, uh, trained.

148. 21:11

     So this works very well for open models, right? Like, or any model you have access to its data for, right? Because you can literally decide if it's in, like, see if it's in distribution or not.

149. 21:21

     Uh, again, if you have, uh, studies from when it was trained, you can also see how much, uh, essentially, uh, how much was your distillation gap across teachers and, uh, the artifact that you trained, right?

150. 21:34

     Because sure, you have domain data from all, uh, all the different domains you're tuning, but it's not guaranteed that it, uh, absorbed all the da- data evenly across the model, right?

151. 21:44

     So, um, it, it becomes, it, it becomes very interesting, uh, to start thinking about, uh, these flexible weights and, uh, flexible model sizes, uh, essentially. There's also... I wanted to add about the context piece, right?

152. 21:59

     So, uh, how do you think about co-, uh, ASTs and, uh, and context compression rep- representations? Uh, compaction in its very nature is lossy, right? So, uh, just like Headroom is there, RTK is there, right?

153. 22:14

     Uh, these, uh, these bases, code bases are usually designed to have representations, uh, that we carry forward through life, and you essentially give models the capability to further expand on them.

154. 22:28

     It's, it's more, more like loss-less-ish compression, which can retain states of models or, or states for agents. What do you think about that?

155. 22:37

     Yeah. It, I think this gets to, like, kind of like a, a fundamental philosophy of how agents and context should work. Um, one, one exercise I like to do is like, you know, as a human, like, how many numbers can you...

156. 22:46

     Like, if I just start spitting out numbers now, right? Like, how many can you remember before, you know, like, you, you start losing track of them. I think it's actually very few, right?

157. 22:54

     So in some ways, you could argue that your context window is actually shorter than these language models. Um, and yet you can actually be very effective-

158. 23:00

     Totally

159. 23:00

     ... at that, right? [laughs] Um, your, your context is, is very lossy. Um, I think one of the nice things that people are starting to realize with agents is like you have a lot of non-lossy systems that you can fall back to.

160. 23:10

     So you have a file system. Like, uh, if in your memory all you remember is that you read some file earlier, you don't need to remember the whole file.

161. 23:16

     You maybe remember the important parts, but you can still have the full version of the file on your system. And that's kind of m- my goal when I'm thinking about how do we build a good context engineered harness is the harness should have everything it needs to find what it needs to have, even if it doesn't have

162. 23:33

     everything immediately available.

163. 23:35

     In that case, uh, do you think that the context sharing problem will become cheaper and less of a problem in future?

164. 23:45

     Yeah, it's de- de- definitely possible as well. I've, I've seen cases where, um, you know, the sidekick agent does a bunch of work. It tells the main model, "Oh, yeah, like, here's, here's all the things I found," instead of dumping the full thing, it just references them by file.

165. 23:58

     And then the main model is actually generally... You find these larger, smarter models, they're actually more token efficient with how they use tools and how they read, and so they actually read the files in a way where they only see the important parts, right?

166. 24:09

     Or they decide that, "Oh, actually I only need to look at a subset of this," or, "Oh, I can run a single command and just know if everything i- is done properly."

167. 24:16

     Um, it, it's, it's actually quite amazing, um, the fact that, you know, these multi-model systems, they actually seem to scale and get better with intelligence, which is, um, not something we should just take for granted, right?

168. 24:27

     It's not obvious that actually more expensive models are actually creating an overall cheaper system.

169. 24:33

     Yeah. Like, uh, the scaling laws, uh, if you have a larger model, it's gonna be more, uh, efficient with its tokens. Uh, smaller models, less efficient with its tokens.

170. 24:43

     Yeah.

171. 24:43

     I guess I had a question for you, Alex. Uh, do you guys, uh, like f- weigh more importance on, uh, the actual infrastructure side of routing? So for instance, KV cache-aware routing or, uh, like is that where most of the business is right now?

172. 24:59

     Or, or the- are you seeing strong, uh, pull of like people actually deploying routers in production?

173. 25:07

     So OpenRouter i-i-is a, like a marketplace for language models. We, we exist at, um, uh, w- like we can't see into the KV caches of models unless we're running them ourselves, which is pretty rare.

174. 25:23

     Um, we, we, we do spend a lot of time optimizing for cache hits, and we, we like pass through cache hits directly to users. Um, but w- like in terms of KV cache optimizations, we, we, we can't do any like specific work there.

175. 25:38

     What we do for, um, model routing is we, we try to like find the best model or best combination of models for the prompt, and then when we see a cache hit, we will, we'll like use up the duration of the cache and send the downstream customer like the f- the full savings of the cache hit.

176. 26:00

     Um, there's more work that we can do here where we could say, "Okay, this looks like something where there's significant benefit to switching the model right now," but you haven't used up the full cache.

177. 26:11

     You still have like two minutes left. Um, and it-- we think it's probably worth switching the model and losing the rest of your cache, um, and letting people kind of like tweak their tolerance for that behavior.

178. 26:25

     Um, we've, we've been doing a little bit of that, but we haven't like exposed it to customers yet.

179. 26:30

     What, what is next for you guys in terms of your, your, your model routing? 'Cause as, as you mentioned, you know, uh, it is kind of like a, a different direction from like the, the marketplace business that exists today.

180. 26:39

     So I, I'd love to hear.

181. 26:41

     So I mean, we've been doing-- like we, we've had a, an auto router for like two years almost. Um, but when we launched it, there was like no adoption of it.

182. 26:51

     It wa-- people really wanted to use specific models. Um, and the, the auto router just had like no real usage. It was-- Uh, we mostly saw it as like a, a discovery play point.

183. 27:02

     Like, "Hey, this is how you discover which model might be good for your prompt." And then, um, a-a-around like this J- January this year, with OpenClau, it exploded. [laughs] And the reason it exploded is because there was this fundamental, um, idiosyncrasy in OpenClau where it sends heartbeats every like ten minutes to your model of

184. 27:27

     choice just to see like if the u-- you know, if the, the, the client was still active. [laughs]

185. 27:32

     Mm-hmm.

186. 27:32

     And that means that if you set Opus to be your default model, it would be like using a lot of tokens on this like heartbeat process. And so, um, this was the very beginning of like a very popular app with two completely different intelligence needs.

187. 27:48

     Completely different. And the models, the open source models have improved to a point where, like it makes sense to segment the market in at, in at least those two areas.

188. 27:57

     And so that's, that was how it got started, and then we saw a lot more segmentation blossom afterwards. And so, um, and now a whole bunch of agents and apps on OpenRouter use the, the different routers that we have, and we have a couple of them.

189. 28:11

     We have Pareto Code, which gives you the Pareto, Pareto optimal model for coding tasks given a certain like threshold that you can tune. Um, we have Fusion, which orchestrates multiple models and gives you a fused result.

190. 28:26

     And, um, and we'll have other experiments in the future. What we wanna do is basically like create good primitives that, uh, developers can use to like get really advanced with how they use model orchestration.

191. 28:39

     Um, kinda like sidecar-

192. 28:41

     Yeah

193. 28:41

     ... sort of like that. Um, but also give people like a really easy thing that they can just like set a slug to that works with all harnesses, um, and just like gets the job done.

194. 28:52

     I feel like it's super interesting how much of a perfect storm there is for model routing right now. Because on one hand, you know, ignore agents, ignore OpenClau for a second, like just to squeeze better performance, it seems like we should be smarter about how we tackle problems.

195. 29:06

     That's obvious, right? If you make a plan, if you make a strategy, that's a, that's a better way to, to go about your day. So I'm not surprised that you're gonna see, uh, you know, better code get written or more performant code get written, less buggy code get written if you break the problem down.

196. 29:19

     And so routing specifically for that, uh, use case makes a ton of sense. But then hearing this, yeah, like the profile of workloads changed with agents, right? They're very-- They-- It went from like I ask questions, I get a response.

197. 29:31

     Then it went to reasoning, where I ask questions, it reasons, and then it comes back. And then it went to, uh, yeah, this heartbeat, right? Like if, um, if my agents are running optimally, uh, there's a token being s- generated every second.

198. 29:43

     Um, and suddenly that is its own need for model routing, and it, it feels like, um, hearing the different solutions, uh, to kind of tackle each of those is very interesting.

199. 29:52

     Even-

200. 29:53

     Well, I was just gonna say, I think that r- yeah, the, the use cases for model routing are, are, there are many of them. And so one could be getting a better answer.

201. 30:02

     One could be, you know, saving money and trying to get the same answer. One that we haven't even talked about yet, which is probably the most relevant maybe even to this crowd, is, uh, when do you wanna actually run a model locally versus when do you actually need something like a frontier model to, to perform that task?

202. 30:18

     And that might be something to the effect of like, uh, for privacy protecting information. You know, when I'm running some local-- Can you detect that my prompt has sensitive information?

203. 30:28

     And if so, do that on, on device. Um, but then maybe even anonymize some of that information to go do the more advanced workloads up on top of that information in the cloud.

204. 30:38

     Um, another example would be, again, for the cost savings, where it's like, "Hey, I bought this DGX Spark-" And I'm n- I know I'm not at 100% utilization, how can I make sure that as part of my workloads, whether it's the heartbeat in OpenCL or what have you, that I'm leveraging that compute, um, to the fullest of

205. 30:54

     its ability because I'm only paying for the electrons that are coming in for my power bill, um, but I'm paying full price for the tokens in the cloud? And I think that's a whole other area of, of model routing that I know that we're doing some work with at NVIDIA, um, that I think will be, will be

206. 31:07

     really cool as it starts to, uh, the hybrid of local and cloud-

207. 31:11

     Totally

208. 31:11

     ... starts to, to really emerge as its own sector.

209. 31:14

     Yeah. I'd be curious to know what you guys take, uh, take is on. Like, like if you self-host a model, the cost dynamics change, right? You have a considerably higher cost at a higher context length because your throughput slows down as the context gets deeper.

210. 31:29

     So rather than switching to a cheaper model, even if you have self-hosted models in data center, you can use compaction, uh, to bring your throughput back up. Uh, have you guys, uh, thought about this paradigm, like compaction versus, uh, just routing?

211. 31:44

     Because one is you have fewer tokens to work with, one is we have cheaper tokens.

212. 31:51

     Y- yeah. I, I think in practice, um, compacting alone doesn't solve the, the cost or, or throughput problems because a lot of times it's just like the, uh, the differential in like model intelligence and cost is just so big where...

213. 32:02

     Also, by the way, when you compact, you're taking a cache miss, so you're actually now like paying 10 times as much for the, for those input tokens if you didn't compact.

214. 32:09

     Um, the main reason we compact is actually intelligence. Um, a- a- all these like model providers, they advertise some like insane context window, like a million tokens. Um, I would like never recommend using like these models past like- [laughs] ...

215. 32:19

     200K tokens. Under 100K if you can. Um, the, the, the intelligence, uh, just kind of like falls off a cliff at some point. Um, sorry, Anthropic, if you're, if you're watching.

216. 32:28

     But [laughs] I, I think that, uh, you know, compaction is like a very useful tool if you are gonna have to take a cache miss anyway, um, one way or another, like when you're routing to another model and you want to just like minimize the, the window there.

217. 32:43

     Do you find that, um, in the sidecar-

218. 32:46

     Yeah

219. 32:47

     ... um, when small models are generating lots of tokens, um, is that like an, is that like one of the best reasons to switch it to a larger model?

220. 32:58

     Like basically, when, when small models generate lots of tokens, I wonder if that's like a-

221. 33:03

     Indicator

222. 33:03

     ... crux of root, you know, the root cause of, of intelligence problems down the road.

223. 33:08

     Mm-hmm.

224. 33:08

     You want your big model to generate the big token chunks, the small models to generate smaller token chunks, right?

225. 33:13

     And on that question, you mentioned like a, a small model essentially needing to flag that it needs help from the larger model.

226. 33:19

     Yeah.

227. 33:19

     What is that mechanism? 'Cause that seems like the... What's the indicator, and then what's the mechanism for it to do so?

228. 33:25

     Yeah, totally. Uh, so there are a lot of mechanisms we, we talk about in our blog post about how, uh, we just detect that we need to like change the model up.

229. 33:33

     Um, to, to... Okay, I guess to answer your question first, how does the small model detect? Um, actually the, the thing that we spend a lot of time on is, um, how do we make sure the small model is like good at detecting it?

230. 33:43

     Unfortunately, there's a lot of cases where you do need the big model to detect it. Um, one thing that we d- don't go into the blog post is, um, you have some kind of cadence on which you're refreshing the cache anyways, 'cause by default there's some like five-minute lifetime on these caches.

231. 33:56

     If you're gonna go refresh a cache anyways, you basically can get a free like big, like frontier model call, uh, if you kind of like ask the right question.

232. 34:04

     So it's at that point where you might say, "Hey, just take a look at what the small model is doing. Does it feel like it's kind of like going into some rabbit hole and, and needs some help now?"

233. 34:12

     Um-

234. 34:12

     What's the need for the five-minute refresh?

235. 34:15

     Uh, it's just like a practical, like you, you have to pay some kind of like cost to like keep these like KV cache-

236. 34:20

     Oh, yeah

237. 34:20

     ... caches warm. Um, and so most caches just get evicted on some kind of cadence.

238. 34:24

     How it works is like, at inference time, you only have so man- so many cache you can keep, uh, kept loaded in the GPU. So once, uh, if a cache is not being used again and again, it's offloaded.

239. 34:35

     I see.

240. 34:35

     So it's lost essentially. So that's why the inference provider asks you for money. Uh, but if you self-host it, you can, uh, get around this problem. You can, you can make it as long as you want based on your big business logic.

241. 34:48

     Well, do, do you see a world where we'll have like much more dynamic cache durations rather than just the five minute, one hour?

242. 34:57

     It depends on who's do, who's deploying the model where, right? So if you have a GPU which has, uh, like a lot of memory, uh, which, like the ratio of, let's say, SMs to memory, um, is memory more, uh, heavily skewed.

243. 35:14

     Or if you're working with unified memory, um, and you have systems like Vera Rubin, uh, you, you have a lot of tricks to play here, right? Uh, the five-minute, uh, window is what a lot of providers right now put, but that's, uh, that's, that's more an operational, uh, o- operational, uh, operational determination rather than a, like a

244. 35:36

     science-based or like a core physics law determination. So you can technically see over time maybe some, uh, some APIs are priced differently. Uh, but, uh, if you de- self-deploy, again, you can, you can get past a lot of this.

245. 35:53

     The, the cost economics really change when you move from self-hosted models to, uh, API providers, uh, because you have a lot more control and, uh, you don't have to guess the shape of your workload.

246. 36:08

     So let's say if your workload is 32K, uh, uh, like on average 32K cache, 1K input, 1K output, um, and someone else is like, let's say 64K, uh, 1K, 1K.

247. 36:22

     Uh, if you use some provider, they are amortizing everyone's use, uh, use case, and then giving you a price, right? And they have optimized, quote-unquote, for general use. If you're self-host, you can optimize specifically for your use, and you'll likely pay much less.

248. 36:39

     Yeah. Uh, and th-this is, like, kind of like the level of like, you know, h-hardware, software frontier that we kind of like thinking about. When, when we started Cognition and we were working on the first agents, uh, I think one reason why no one else worked on agents is they were just extremely expensive.

249. 36:52

     This was before cache tokens was a thing that API providers paid for. Like, if you were sending 100,000 tokens and the same 100,000 tokens, you were paying full price for those tokens-

250. 37:01

     Mm

251. 37:01

     ... uh, back in 2024 when we started. Um, one of the key things that let us build Devin and build these first agents was we actually bought direct c-compute capacity from these providers, and instead of paying on a per token basis, we just paid for the underlying compute, knowing that the economics of the compute was that we

252. 37:19

     were actually paying far less for, for the cache tokens that we'd send over. Um, and, and then nowadays, you know, uh, uh, you know, there's, there's like similar, you know, things I would like about, like, you know, having a, a version of the cache that maybe, like, you can just back out to, like, storage in S3 or

253. 37:32

     something and just, like, hold for much longer.

254. 37:34

     Yeah. Now, this is not extremely relevant to a DGX Spark organization, uh, like setup, but, uh, if anyone's looking to do what you guys are wanting to do, try out Dynamo.

255. 37:44

     Uh, we have a lot of prefix cache optimizations in there.

256. 37:48

     Yeah. Um, and then, yeah, uh, uh, going back to your question, Alex, I think you said like, oh, like you ... When a small model is, like, going off and generating a ton of tokens, is that, like, a interesting time to back off?

257. 37:58

     Um, to be honest, we haven't explored that right yet. So that, that might actually be a very interesting thing to, to take a look at. Um, it, it is weird.

258. 38:05

     I think, like, some small models d-do tend to kind of be less, like, token efficient than others, but, um, they also seem to be trained on, like, their own traces, so maybe in a way it ends up, like, balancing out.

259. 38:15

     A lot of these things I, I feel like we have to be, like, very empirical about, uh, to actually know.

260. 38:20

     So, uh, just to add on, uh, uh, uh, uh, that, um, you have a lot of... Like these days, there are a lot of hallucination probes, so probes that work on either, uh, the internal state, like internal state of the models, uh, uh, directly.

261. 38:35

     So you can have some form of either magnitude analysis done or l-linear probes or just, uh, the n types of probes that you can see. And you can essentially rate, like, how, how much you think is, is tending towards hallucination.

262. 38:51

     Uh, so that kind of gives you a proxy for how lost it is, uh, like how lost a model is in its thinking. Uh, so you can use, like, different kinds of probes to understand, uh, like the perplexity within a model.

263. 39:05

     Oh, that's interesting. So yeah, instead of using the quantity of tokens that, uh, as indicative of a, of a model being lost, it's the, it's hallucinating more. And you're-

264. 39:14

     Yeah. So, uh, so essentially what is cache, right? It's, it's, it's the pref- uh, it's the prefill, uh, states, right? So you c- uh, what is a prefill state?

265. 39:21

     It's just a vector at the end of the day. So you can do tune all kinds of classifiers to understand, uh, different aspects of those, uh, collections of vectors.

266. 39:32

     So, uh, with those kind of probes, you can guesstimate a lot of, uh, states, uh, of a model.

267. 39:41

     I see. One, one question I have is, um, you know, different models behave, uh, differently. And, um, th-that kind of means that these prompts aren't portable. So as you're doing model routing, how do you handle es- essentially if you're, if you're going to a different model architecture, um, what do you have to do to the prompt, and

268. 39:58

     how much is that a factor into either of your guys' model routing solution?

269. 40:03

     Like what, like how, how is the prompt itself a factor-

270. 40:06

     Yeah, like how much-

271. 40:07

     ... into the routing?

272. 40:08

     Yeah.

273. 40:08

     Yeah. Um, well, I think with building agents, there are all kinds of paper cuts and edge cases that are domain specific, and you-- like, the, the value of an agent company, um, like the value of Devin is all these, like, doom loops that you've discovered that a-are across all industries and the best ways to recover from them.

274. 40:32

     Um, and like man- here, I mean, it manifests big time in what the prompts are going to be, both for, like, you know, how the, the advisor model gets called, um, uh, you know, the smart friend, the [laughs]- [laughs] ...

275. 40:47

     the, uh, how the, like, subtask agents get called. And, uh, and the best thing is that, like, anyone can, like, ins- like any engineer or any, like, agent can inspect the traces and, like, adjust the prompt and then see the, like, live accuracy, uh, long time.

276. 41:02

     So I mean, basically, I just think that that's part, the prompt is part of the, the startup building process and, uh, is also really easy to observe and-

277. 41:12

     Yeah

278. 41:13

     ... um, like, and have, and cl- and have, like, multiple people and agents collaborate on them.

279. 41:16

     Yeah. One thing I'd love to do, uh, with, with our Fusion product, and we don't have this yet, and so this is k- maybe a preview of some, some things we work on, is, you know, you can tune it against a data set, but the, the real thing you want when you're building a real agent someone uses

280. 41:29

     is to just, like, tune it against what actual people use it for and, and what actual models they get routed to. And so there's a lot of signals for this.

281. 41:36

     Um, like if, if someone sends a prompt and then Devin is working, and then you see that the user decides themselves to, like, upgrade to a different model, or they decide to downgrade, or the system detects that we originally sent to the wrong one, we now got to replace, like that's actually a really useful stream of signals.

282. 41:51

     And we're actually getting to this world of, like, auto research, where, like, maybe we can just have, like, a constant stream of prompts, what it should have been, what it was instead, and build a system internally that's just capturing all of this and then reiterating on our routing system until it eventually kind of, like, fits the real

283. 42:07

     production data. Um, that's kind of, uh, now that we-- it's, it's, it's public and, and people are using it, this is now something that we're, we're thinking about.

284. 42:15

     Have you guys looked into prompt tuning, and do you find it useful, like, say, Japa?

285. 42:21

     Yeah. So, uh, there are, like, these prompt tuning frameworks from, like, a few years ago that, uh, tried to do some kind of like, like gradient descent type thing.

286. 42:29

     Um, I'm like, I'm actually personally less bullish on, um, these kind of like low-level mechanical prompt tuning harnesses versus just telling, like, a smart model, like- Here is the decision that was made, um, and the context.

287. 42:44

     Uh, figure out why it went wrong. Sometimes you, you can do something as dumb as asking a model, "Why did you do this instead of this?" And cite the, the prompts.

288. 42:51

     Um, and then just have your agent, your dev, and just go and, like, update the prompts, rerun the, the test as a regression, make sure, make sure it changes.

289. 42:58

     Like, it-it's a lot heavier weight of a system, but I, I kind of trust the intelligence of a system like that a lot more.

290. 43:04

     So we're, we're running out of time, so we're gonna wrap up real quick. But I think what's really interesting is just from talking to you guys, we can kind of see how, uh, new this space is, right?

291. 43:13

     How much of this is actually just research. Um, we're starting to see new products come in. I'm, I'm really excited about your guys' solutions as you guys enter the space.

292. 43:20

     The, the ways and the needs that you need routing for. You know, even on a DGX Spark, when you're doing local inference, um, you have, uh, more compute. And if the memory's filled, one w- uh, or if it's, if the memory utilization is high, one thing you need to do is increase the compute utilization.

293. 43:35

     And so one way you can do that is by spawning multiple agents that are working collaboratively. So that collaborative piece is something that not only is optimal for all of these cloud workloads that you guys are doing, but, but specifically that is how you extract more performance out of this edge hardware.

294. 43:49

     And I think, um, you know, a question here is, uh, and maybe s- to, to end on, is a router gonna be something that we see as a product, or is that gonna be seen as part of the plumbing here?

295. 44:00

     Are models gonna get good at routing to other models because they know they need to be collaborative, or are harnesses gonna know that they are working across multiple models?

296. 44:11

     I, I think we already see this. Like, uh, you know, w- at Cognition, we're training our, our models to be able to be good collaborators. Um, I think, uh, it's very clear that new frontier models, like dif- fable models and GPT 5.5, 5.6 models are, like, themselves, like, naturally collaborative and, and better at delegation.

297. 44:26

     So I think we're already there at that point.

298. 44:27

     Hmm. Interesting.

299. 44:28

     Yeah, I, I think that the systems are, are kind of becoming not muddied in some sense, but I think that ultimately we're understanding that as we step up the, the abstraction ladder and build more, more things to create this smarter, uh, blob, um, which obviously we should hopefully, and we do understand how we're building it and why

300. 44:47

     we're building it, um, that it's gonna, it's gonna become a system that you look at kind of both the, the different components of the system. But, um, it's not just gonna be just models.

301. 44:56

     There's not gonna, there's not gonna be a thing as, like, a really great harness that is in absence of a really great model and vice versa.

302. 45:03

     Yeah. Makes sense.

303. 45:05

     I think, uh, applications, especially, uh, built on, uh, non-deterministic systems like models ha- operate in a very low-trust environment. So yes, most of the, uh, improvements will likely be distributed across both models and the harnesses.

304. 45:23

     But I think, uh, overall, it's, it's, it's mostly, it's, it-- there, there will, there will have to be some form of controller, uh, trying to have some form of arbitration, because even from the model perspective, you aren't in a perfectly visible world.

305. 45:37

     You don't know the behavior of every model, so it's, it's gonna be at the orchestration level where you have these kind of things. And this has traditionally been shown by other industries.

306. 45:48

     Like when web, uh, when web launched, you know, uh, you had traffic-based routing. Sure, it's different. Uh, but all the sort of routing controls have been centralized over time.

307. 45:59

     Makes sense.

308. 46:01

     I think it's most likely gonna be good news in the future. Um, and, and I, I think, like, caching is a big reason for that. Even if you-- I think, like, a-- To take the flip side of, of this argument, um, the f- you know, it might be that in the future we have, like, one big model

309. 46:18

     that's like, "I know I am the, like, most efficient at everything, and I'm, like, way more efficient than Haiku. I'll solve every task better than Haiku can at, like, a lower price.

310. 46:29

     Um, why should I ever delegate to Haiku?" You know, something like that actually could, could be a model that we have in the future. Um, but you're always gonna have these, like, you know, for example, caching.

311. 46:41

     It could be that, like, you tell the model that this other model, like, does have the right context in cache and, uh, you know, the, the orchestrator model just always has more context, and the models have to be aligned. [chuckles]

312. 46:54

     So I, I think, like, it's-- I, I don't really see a world where, like, we wouldn't be able to get models to collaborate really well and, and I think they're gonna get better over time, um, in part because they're, you know, they just have limited memory. [chuckles]

313. 47:09

     So I, I think that's kind of one, one deciding factor, and another is that, um, there will be, like, uh-- there will continue to be... Like, if you just look at, like, the, the, the rankings on OpenRouter, if you look at our, our public data and you look at, like, the top model being used by dollar spent

314. 47:30

     on classification tasks, like, guess what it is. It's Opus. [laughs]

315. 47:35

     I think there are, there are, there are big opportunities for, like, using small models for in-distribution easy tasks and the-- and, like, as time goes on, that's gonna be a larger and larger percentage of tasks relative to, like, the most valuable tasks that, um, very smart models spend most of their time on.

316. 47:54

     Totally. Well, I wanna thank you guys so much. Can we all give everyone a round of applause? [clapping] Thank you. [outro jingle]