What Lies Beneath the API — Benjamin Cowen, Modal

Talk transcript

1. 0:00

   [upbeat music] Yeah.

2. 0:15

   So good afternoon, and thanks for coming to this session. I know there's a lot to choose from. Uh, my name is Ben Cowen. I'm a Forward Deployed Machine Learning Engineer in Modal.

3. 0:25

   And, uh, I wanna talk about an interesting pattern that we've been seeing, uh, in AI application development. I'll give you the punchline now. Um, it's about, uh, as companies mature and their products mature and specialize, we're seeing more and more turn to fine-tuning to get impr- increased performance, better cost, and so forth.

4. 0:47

   So this brings up an interesting question of when does your application step over the line into a custom domain? When is fine-tuning worth it?

5. 1:00

   Um, so if you're not familiar with Modal, uh, we're a general purpose serverless compute platform. Uh, we provide sort of basic building blocks, like serverless functions and, uh, like hardened sandboxes for code execution.

6. 1:15

   And so as an FDE on a general purpose platform, I, I've had the opportunity to work with sort of a extremely wide range of AI applications. So from physics simulations to, uh, quantum chemistry and of course like voice processing, LLMs and agents.

7. 1:34

   A really interesting, um, one that's been really blowing up is large-scale reinforcement learning. Um, and so we've started to think about, um, some of these customers as where, where are they on the model spectrum?

8. 1:50

   So on one side of the spectrum is the frontier API. And the frontier APIs, uh, I think everyone here would agree has unlocked a completely new era of s-- uh, you know, accelerated growth.

9. 2:06

   Um, people can build basically anything exceptionally fast. Um, they're amazing. Um, but, uh, you can't customize it at all, uh, beyond prompt engineering. And so, um, you might y-- I mean, uh, I love the, uh, the whole caveman mode thing.

10. 2:23

    If you tell your LLM to speak like a caveman, you can reduce your tokens by like a lot. But that's not gonna scale if your startup, you know, hundred Xs or a thousand Xs, right?

11. 2:36

    Um, another, uh, kind of interesting thing we see is when, uh, startups win large enterprise contracts with very specific latency or throughput requirements. Um, there's very little ability to customize for those things, let alone if you have a custom metric that encapsulates your business logic.

12. 2:57

    Okay. So to get this model differentiation, a lot of companies turn to fine-tuning. And what that has meant traditionally is this huge jump to the other end of the spectrum, right?

13. 3:11

    We-- Training has a very different like scaling and compute characteristic to most production workloads. So if you wanna train and serve a production workload, in the past you have to get a big cluster.

14. 3:24

    Now you have to isolate that, those resources from your production resources. You're gonna need infrastructure engineers, or your AI engineers are gonna be working on infrastructure, maybe even your scientists.

15. 3:37

    So with this kind of extremely customized, uh, you know, powerful option, you also have this massive responsibility for the entire stack. Um, and so, uh, you might g-have a guess who I would recommend for this, but there's sort of a middle ground that's emerging.

16. 3:57

    There's a new type of cloud provider that makes this a lot easier. And, um, you know, we're, we're building this to address, uh, this problem that we're seeing, right?

17. 4:10

    So leader after leader in the space are announcing or publishing that they've fine-tuned and gotten incredible results. Okay. So Intercom is beating their frontier API at one-fifth the cost.

18. 4:27

    Um, Pinterest says orders of magnitude. I wish I knew the exact amount. But I think, uh, one of our customers, Decagon, has summed this up really well, which is that, um, basically the frontier labs probably don't have the exact same goal as you, right?

19. 4:44

    They want their models to win on everything possible. And, um, we want our models to win at our business logic, right? You wanna be the best at what you provide your customer.

20. 4:57

    Um, and so, yeah, I'm happy to announce that it's actually a lot easier than you might think to train a model. Um, with-- There's some incredible open source libraries out there now that make this extremely accessible.

21. 5:14

    They give you, uh, full control over the algorithm, right? So you get to kind of reach across the spectrum to doing it yourself at the algorithm level without having to also manage the cluster and so forth.

22. 5:29

    And the most important thing is that this retains the fast iteration cycles of the frontier end of the spectrum. Um, so that's basically our entire mission is to give you algorithm control and fast iteration.

23. 5:47

    Um, so this is my hot take that it's just a matter of time until your product steps into being domain specific, right? So in some sense, like if you have a differentiated product, it is custom, right?

24. 6:02

    So, uh- When exactly you cross that line, that's s- a decision you have to make, but it's something we'd love to talk to you about. Um, so I, I have here a few signals that might indicate that, you know, you're getting close to that time.

25. 6:20

    So if you've, uh, moved to caveman mode and you're still paying more for your API than your customers are paying you, that might be a signal that your economics aren't scaling, right?

26. 6:32

    And that you, you could probably benefit from a customized inference endpoint. Um,

27. 6:39

    same for latency and throughput, right? So if you, if you are plateauing on your evals, that is a-- that's a signal that you might get something out of fine-tuning a model.

28. 6:50

    Um, there's a decades-old adage in training that if you have garbage data, it's garbage in, garbage out. So if you haven't been collecting data and you don't have mature evals, it's probably not time to train.

29. 7:05

    You need to collect the data. Um, that said, uh, this is one of the main takeaways that I'd love for everyone here to walk out with, is that if you have built a product, you probably have at least touched all of the things you need to train if you haven't already done it.

30. 7:24

    Okay? If you've built an agent harness, then you have what you need to have a new model, you know, learn through reinforcement learning how to provide your service, right?

31. 7:36

    So if you're evaluating your products, um, and collecting that data on what's working and what's not, then you have training data to train your model. Um, and with, you know, the advent of, uh, serverless compute platforms and, uh, these open source libraries, I don't know, like, uh, a lot of us when we started training models, we were

32. 8:00

    taping the gradient by hand and like implementing the linear algebra. You don't, you don't have to do that anymore unless you have a freaky model, um, which if you do, I'd love to talk to you about it.

33. 8:12

    Um, but yeah, you don't need the infrastructure experts, um, and so forth. So this is an exciting time.

34. 8:21

    Um, I knew the vi-video wouldn't play. Anyway, so, uh, the next couple slides are just some snippets of code. I don't expect-- Yeah, you can't even really read it, but the-- I just wanna, uh, k-kind of illustrate what it looks like to set up a training algorithm today.

35. 8:39

    It's not, uh, a gigantic monorepo with thousands of lines of code. You can do supervised fine-tuning in three hundred lines of Python.

36. 8:49

    Okay, so once you have your data curated, uh, once you have an account on Modal or some other serverless platform, you can get started really fast, and this code is on our examples repository.

37. 9:02

    Um, the, the thing in this video, it's just showing how, um, we can scale containers really fast. And so just to bridge, bridge these concepts a little bit, people usually associate serverless with, uh, inference.

38. 9:18

    But with training it can be really, really handy too, uh, for doing something like hyperparameter tuning, right? You, you can-- You don't have to, uh, you know, every minute on your cluster isn't sacred anymore.

39. 9:33

    You can fan out to a bunch of containers, get them on demand. As soon as it's not promising, kill it. And y- It's kind of, uh, almost like an, a meta evolutionary algorithm at that point.

40. 9:47

    Um, so it's, it's a exciting time to be doing that. And the same goes for reinforcement learning. Um, a lot of us who got our, you know, graduate degrees in machine learning in the last ten years, uh, didn't do reinforcement learning, right?

41. 10:01

    This is relatively, I mean, it's actually really old, but the stuff we're using today is pr- kind of new. But they have these libraries too. Um, you can do it in three hundred lines of code.

42. 10:13

    Um, and something interesting about Modal part- in particular is we have sort of unified APIs for sandboxes and GPU containers or clusters. Um, so what this means in a nutshell, when you're training a model with RL, it needs to sort of practice a lot and so this is massively embarrassingly parallel kind of evaluation thing called a rollout.

43. 10:38

    And so we have one of the most amazing things in the last quarter has been customers scaling up to fifty thousand, a hundred thousand sandboxes just to do RL.

44. 10:48

    Um, and you can do it too. Uh, the code is open source.

45. 10:55

    And then I'd be remiss not to mention what comes after the training. You have to serve the model, right? And this is what the a-- the frontier API is doing, you know, under the hood.

46. 11:07

    Um, well, they pro-- I don't know if they use vLLM, but my point is that you can do it too, and it's actually not that much code. vLLM, SGLang, Trident Inference Server, or a custom inference workflow with just Python, um, on Modal or other, you know, serverless platforms, you can autoscale all of this stuff, uh, to match

47. 11:30

    your traffic as it's coming in. So yeah. So, uh, just to kind of sum everything up what I'm saying here, I'm not saying go train your model right now.

48. 11:42

    I'm saying it's not something that is like, "Oh, I'll do that in ten years." You might, you might wanna train your model in one year, right? You might wanna do it in six months.

49. 11:53

    So start thinking about what is... when am I gonna know, okay, it's time to train my model, and how can I prepare for that moment by collecting data, developing your evals.

50. 12:06

    And yeah, I'd love to, you know, come by our booth. We're like at kind of the end over on that side. Uh, love to talk to you more about this, or you can reach out, uh, at my email here.

51. 12:18

    That's it. [clapping] [outro music]