The Rise of Open Models in the Enterprise

Talk transcript

1. 0:00

   [on-hold electronic music] Yeah. Hi, everyone.

2. 0:15

   My, my name is Amir. I'm co-founder and CTO of Baseten, uh, the inference company. But I'm not here to talk about Baseten. I'm here to talk about the adoption of AI in the enterprise, why we should care about it, and how it's going, uh, based on what we've seen.

3. 0:33

   So, uh, first, why, why we should, uh, care about it. Um, uh, ul-ultimately, um, a... We've heard this before as like, "Hey, is there a hype in the market?"

4. 0:43

   You know, "Is, is AI hyped?" And like, it probably is, but, uh, the, the evidence that a lot of people point to that there's hype here is that the adoption in enterprise has been slow.

5. 0:55

   Uh, you know, you've- I've heard this so many times that like, you know, uh, enterprises are, are, are slow to adopt, uh, and, uh, and, and, and that, that has an implication.

6. 1:05

   Uh, if, if that is true, that has implication for really the impact of AI, uh, and h-how large it can be and whether it is truly a hype, uh, or, uh, or, or real.

7. 1:15

   Um, and the reason is that it's enterprises are massive, like their reach is massive, um, they have all the money, uh, and, uh, and, and if, if they're slow in adopting, uh, then, uh, the, that paradigm shift that we're talking about, that, uh, will be slow, uh, to, to materialize.

8. 1:31

   So but I'm here to tell you based on what I've learned, uh, uh, about adoption of AI in enterprise, um, why, why me? Why, um, uh, because we're, we're, hap-we happen to sit somewhere interesting.

9. 1:43

   We happen to sell to enterprises. Uh, and, and so over the past... The company is six years old, but over the past two years, uh, you know, in particular, talked to honestly 100-plus enterprises from software companies that are public to literally soft drink companies, uh, that, that are, you know, Fortune, Fortune 50.

10. 2:02

    Um, and, uh, and I've seen patterns that I want to share those patterns with you. Uh, one bias that I have, uh, is that I'm, uh, I don't sell a verticalized, uh, AI tooling.

11. 2:14

    I, I sell, I sell a very horizontal AI tooling and, and this is important. So, uh, enterprises are adopting vertical solutions, you know, um, AI for sales, AI for marketing, um, AI for customer service.

12. 2:24

    You just heard from Clay from Sierra. Um, that adoption is happening. But I think for the true value to get unlocked, uh, we need to see enterprises actually build with AI.

13. 2:34

    Um, the, the analogy that I use is that if in the 2000s, uh, uh, enterprises were, uh, not really building tech themselves and, and were just buying, uh, Salesforce, uh, or, uh, you know, products like that, verticalized product-products like that, then the, then the, the tech industry would just not be as big.

14. 2:53

    Uh, companies like Snowflake and Databricks and Datadog would not exist or would not exist to the shape that, that they do. And so I really think that the value is ultimately unlocked once the enterprises feel comfortable to actually build with AI themselves as opposed to just buy v- uh, verticalized tooling.

15. 3:09

    So let's talk about the j- the journey, uh, the, the, the journey that they go through. Um, uh, they, they, they all start with OpenAI and Anthropic. You know, enterprises, they're like us, uh, you know, for, for good reasons.

16. 3:22

    It's just so easy to get started. Um, it's just they do it differently from, from the rest of us in that they, they have their own, uh, dedicated deployments, uh, of, of these models on Azure or, or AWS, um, and, um, for, for reasons around security and privacy and, and all that.

17. 3:38

    Um, and then they, they get their engineers. A lot of times they're more, you know, predictive, uh, ML teams, uh, to become AI teams and, and, and build on top of these.

18. 3:47

    Um, and they're, they're happy with that. And if they can continue doing that, they will, um, because, uh, there's just ano- a lot of inertia in, in, in, in sticking to that if it actually works.

19. 3:59

    Uh, sticking to closed models, which is so easy to use, API-based, you know, build on top of it.

20. 4:05

    Um, but we're seeing cracks in that assumption. Uh, and, and so let me, let me tell you what, what I've seen going back in time and, and how that's changed over time.

21. 4:14

    So in 2023, I remember like going and, and trying to sell to enterprises and, and, and the, the terms toying around came up quite a bit. And I heard this actually literally from, um, the CIO of a massive insurance company, uh, back then.

22. 4:27

    It was like, "Yeah, we put a dedicated deployment of OpenAI, uh, uh, of, of GPT-4 or GPT-3, um, uh, so that our engineers can toy around with it." Kind of like almost dismissively talking about it.

23. 4:37

    It was like, "Yeah, go build something cute." Um, that started to change. In 2024, we saw actual production, uh, use cases, again, built on top of these closed models.

24. 4:48

    Um, I would say like 40, 50 out of the 100 like had something in production, uh, th- uh, in that year. Um, and then in 2025, this year, something changed.

25. 4:59

    Uh, and, and, and it's palpable from, at least from where I'm sitting. Uh, and, and, and the change is that there are cracks in that assumption. There are cracks in the assumption that we can actually build on top of these closed frontier models, uh, indefinitely.

26. 5:14

    So what are those cracks? Um, uh, I'll tell you what the cracks are not, and there's some misconceptions over here. Um, I'll tell you what they're not. Uh, so like often people say, "Oh, because people, like enterprises don't want to have vendor lock-in."

27. 5:26

    I don't hear that, honestly. Uh, like we go and talk to them. I, I think I know why. It's because, one, there's a few of them now, you know, you can go OpenAI, Anthropic.

28. 5:35

    Google has been coming up pretty well. They're somewhat interoperable, uh, uh, at a certain level. Like they all use OpenAI specs, so like building on top of them, yeah, you might have to like do your evals again and, and, um, do some prompt tuning.

29. 5:47

    But, but generally you can go from one to the other. So vendor lock-in is not something that I hear about. Um- Ballooning cost. Uh, I, I didn't hear that last year, um, and I, and I know why.

30. 5:58

    It is because when I ask them, they say, "Look, uh, the, the price per token is plummeting." And we, we just talked about this, like, right before this. Uh, and that's...

31. 6:07

    The-they were saying that problem will just take care of itself. Um, compliance, privacy, security, also not problems because th-these frontier, uh, model companies kinda take care of that with the help of the CSPs, with the help of the, the, the cloud providers so that these models are running in a dedicated way inside of their existing VPCs.

32. 6:27

    Um, if these aren't the, the, the, the cracks in that assumption of just use closed models, then, then, then what are the cracks? Uh, these, these are, uh, the, the, the, the reasons that I have seen.

33. 6:38

    Uh, and, and I'll go through them one by one, uh, and, and go through, like, examples of these as well. Uh, and, uh, and, and then at the end also talk about, um, you know, uh, if, if these are the cracks, then, uh, you know, how, how do you get around them and, and, you know, there be

34. 6:53

    the dragons. So one is around quality. Look, uh, none of these enterprises are in s- any sort of misconception that they can build the next GPT-4 better than OpenAI can.

35. 7:05

    That's just not the, the reality. Not, not as a gener-general model at least. But for specific use cases, for specific tasks, we are seeing this where the, the, the frontier models are not necessarily the right tool.

36. 7:16

    Uh, so the, the example, uh, uh, I, I've seen this in a couple of big health plans, is that they wanna do medical document extraction. So they have m-millions of medical documents, uh, prior auth and, uh, medical claims, and, and they're trying to get, you know, CPT, like, um, uh, procedure codes and diagnosis codes and, and prescriptions,

37. 7:36

    uh, and just giving that to, you know, Claude or, or GPT doesn't, uh, doesn't do it. But they have the data. Uh, over the years they've collected a lot of labeled data, and they're like, "Oh, we can do better."

38. 7:47

    Uh, and, and so that's... That... And, and, and they actually did. That's one example. Another example is, um, uh, uh, on the voice side, in p- in particular on the transcription side, like, uh, again, staying in the healthcare space, like understanding medical jargon, or like, you know, getting, uh, getting transcription models to understand medical jargon, um, that's...

39. 8:06

    That, that has been another reason to not just use an API-based generic model, um, but, but in-house it and, and do better than, than what they can, uh, what they can do with just, uh, API-based models.

40. 8:17

    Another one is, is around latency. Um, look, these, these models, um, uh, OpenAI, Anthropic, even, even, you know, the, the big players that serve open-source models behind shared APIs, inherently they're optimized for high throughput and high QPS at the expense of latency.

41. 8:37

    But a lot of times we're seeing more and more now where latency is becoming very critical, uh, especially when, you know, the, the AI, AI voices or, or AI phone calls, uh, latency starts really mattering.

42. 8:49

    Uh, time to first token, time to first s-sentence really starts mattering. Uh, and, uh, you have to just think about things differently. Uh, you can't just use the, uh, the, the frontier models as is because, again, they're optimized for something else.

43. 9:07

    Um, around the unit economics, uh, there's, uh, the... Again, like I said before, like pricing, they said this, this will take care of itself. Uh, then came this year and, and as you saw in the previous talk from Michael, that the agentic use cases ballooned.

44. 9:20

    Uh, and they... When, when, when they balloon, it, it's crazy. Like, I've seen this. Like, every single user action can result in literally 50 inference calls. Uh, and, and so suddenly the thing that you thought is gonna take care of itself is not taking care of itself.

45. 9:33

    Uh, that, that cost are, are really, uh, ballooning and, and, and the enterprises think that maybe they can do better on the cost and uni-unit economics. In order to show ROI, in order to show that the, the solutions that they're pushing are, are economically viable, they need to show, uh...

46. 9:49

    They need to reduce, uh, the, their costs somehow, and they're realizing that they can actually run these models and pay for the compute and have that be a lot cheaper than paying per, per token and, and covering all the margins of someone else, and, and really going from being a price taker to actually be the, the, the,

47. 10:05

    the maker of the price and, and be in control of that. And then lastly, destiny. This one is a bit vibey, uh, but, but I'm hearing it more recently, uh, that, um, you know, some CIOs, CTOs saying, "If we, the enterprise, use, uh, just the frontier models and so do our competitors, what is our advantage?

48. 10:25

    What is our alpha?" Uh, and, uh, and, and maybe we should bring in some of these things in-house and to be able to even differentiate not just at the workflow and application, uh, level, but also at the AI level.

49. 10:38

    So now what? If, if those are the reasons why they, they want to adopt open-source models and, and iterate on those and build those, uh, and fine-tune those and distill those, uh, then what changes?

50. 10:49

    Well, what changes is that they go from super simple world that just call an API and run with it, uh, to now you need to build inference. Uh, you need to build inference infra, and you need to make sure that it scales well, and you need to make sure that you can move fast, uh, that, that your

51. 11:04

    engineers can actually, uh, deliver instead of having to, you know, hire a bunch of new types of people, uh, and then wait for them for a long time to actually build this, uh, build this infrastructure, uh, in-house.

52. 11:17

    So one thing that I hear quite a bit at this point is, look, I've... I, I hear this from enterprises, I hear this from startups too actually, uh, which is that, look, you know, we've picked a model, an open-source model.

53. 11:27

    Um, we've heard of vLLM or SGLang or TRTLM. We have some GPUs, in the case of enterprises it's in their data center, in the case of startups it's in some cloud.

54. 11:37

    And you put these together and you get production inference. And I know for a fact that this is not true. Uh, and I, I wish it was true, but I know for a fact that, that this is not.

55. 11:46

    Uh, that, that there is a lot more that goes into, uh, making, uh, inference, especially mission-critical inference, work well, uh, in-inside of your company. So, so what are those?

56. 11:59

    These are the dragons. These are the dragons. So, so, uh, one, uh, at the, uh, at the performance layer, uh, you know, we talked about, you know, situations where things are very latency sensitive.

57. 12:11

    Um, the, the way that you optimize models, uh, for, for latency, uh, is, is, is actually quite, quite involved. Uh, both at the model level and at the infrastructure level.

58. 12:22

    You, you have to do it- you have to attack it at both levels. So as an example, um, you know, for, um, a- at the model level it's like, "Hey, do you use speculative decoding?

59. 12:31

    Uh, and if so, do you... Which, which, which route do you go? Do you go with a good draft model? Do you go with Medusa heads? With, with Eagle 3?

60. 12:37

    Uh, do you go with MTP?" Um, there's, there's a lot... And new techniques are coming out all the time. New... Like, the, the Eagle 3 paper came out, like, six months ago, and it's, like, running in production and actually being very meaningful.

61. 12:48

    Uh, and so as an enterprise, can, can you hire the right folks to be able to be on top of the research? Because you don't... This is- these are not just, you know, switches that you flip in, in SGLang or vLLM and, and, and get the, get the result.

62. 13:02

    Um, uh, some of these optimizations bleed out of the model level into the infrastructure level. Uh, so as an example, if you want to, uh, be able to do, uh, uh, prefix caching really well, if you wanna be able to desegregate a serving really well, uh, because that starts really mattering, uh, uh, especially in, in agentic use

63. 13:22

    cases where, like, the, the prompts are massive but the prompts are somewhat similar from one to the other, uh, it ends up mattering a lot in, in you hitting your, you know, time to first token and you hitting your P99 of that, uh, in, in a reliable way.

64. 13:37

    Another thing on the infrastructure is, especially if it's a mission cri- critical inference, which more and more I see that that's the case, h- how do you guarantee four nines?

65. 13:47

    And w- with, with, with this, with... Th- this formula does not guarantee you, you know, more than two nines. Uh, and I, and I, and I saw this firsthand.

66. 13:56

    Uh, so how do you make sure that when the hardware fails underneath that, uh, you, you actually recover? How do you make sure that, you know, when vLLM crashes, which it, which it happens often, I saw this firsthand with, like, when Triton crashes often, your tail latencies go through the roof while you wait for these things to

67. 14:12

    come back. Uh, and, uh, and, and, and during that time, like, your, your users are, are, are, are feeling that, um, how do you build, uh, uh, against those and, and make sure that, you know, you, you can still guarantee four nines and not be super over-provisioned and mess up all the unit economics that we talked about?

68. 14:30

    Um, when a big burst of traffic comes in, how do you make sure that you scale up fast? Uh, how do you make sure that, uh... You know, I was talking to this massive enterprise, like, uh, soft drink example, where they're like, "Yeah, it takes us eight minutes.

69. 14:43

    When we wanna bring up a new replica of the same model, it takes eight minutes." Um, and, and I believe that because that... if you add up all the different things that it goes into doing that, that is how long it takes.

70. 14:52

    But h- But, but that's not okay. Again, your tail latencies go through the roof as soon as there's a big spike of traffic. H- how do you account for that?

71. 14:59

    And then there are other things around, uh, again, like making sure that your engineers move fast. Uh, the tooling, life cy- life- lifecycle management. The observability massive, uh, uh, um, uh, iceberg, which is like...

72. 15:12

    It's like, "Oh yeah, just put some logs and metrics," and you realize there's a lot more to do underneath it. Uh, we just... Previous talk Michael talked about. Um, and then lots of stuff around controls and, um, o- audits and, uh, things that enterprises actually care about.

73. 15:27

    So, so these are the, these are the dragons. These are the dragons, and these are the times that then enterprises h- have a decision to make. Like, once they, once they get to this level, either, like, I tell them or...

74. 15:38

    And they have to believe me, or they don't and they go and build it [laughs]. And then, then, then they, they see some of these things that they have a decision, which is build or buy.

75. 15:47

    And that's my job to then try to convince them that I, I think they should buy this, uh, uh, uh, this layer o- of infrastructure and platform, uh, as opposed to, uh, build it.

76. 15:57

    Uh, and that's sometimes, uh, harder, harder than it seems.

77. 16:03

    So, so I'm happy to talk more about these things. So I, I'll be, I'll be at our booth. Like, two, two topics that I'd love to talk about if, uh, if you're interested.

78. 16:10

    One, self-servingly, if you're an enterprise and those problems resonate, I'd, I'd love to chat with you about. And two, less self-servingly, um, if you're a startup and you're trying to sell to enterprises, um, I'm happy to chat about all the right decisions that we made, the wrong decisions that we made along the way to, to build something

79. 16:27

    that then when it comes to selling to enterprises, when it comes to deploying it into their own clouds, uh, that, that is actually possible, uh, and, and, and not a massive other set of dragons over there.

80. 16:38

    And then last thing, we have a, a happy hour. Uh, we'd love to see you there. Thank you. [outro music]