AI Engineering 201: Inference

Talk transcript

1. 0:00

   [upbeat music] Uh, in the workshop today was everything that comes after once you've, you know, written your thin wrapper

2. 0:25

   around the OpenAI API to make your ChatGPT-powered app. You've acquired $100 million in venture funding at a four billion valuation, and now you're like, "Uh, g- what, what am I supposed to do next?"

3. 0:40

   Um, so we're gonna-- I'm split this into two parts so we can take kind of a break. It's like a three-hour long workshop. It's a long time. Uh, so in the first half, we're gonna talk about, uh, inference.

4. 0:53

   So about the... what exactly this workload is that we have given over to the, uh, the inference-as-a-service provider. What-- you know, what's the shape of it? Um, why do we need these expensive accelerators?

5. 1:06

   What are the other options available? Um, and we're gonna spend half of our time on that because that's a place where we can actually talk, you know, in engineering terms about constraints, uh, about, uh, service-level objectives and service-level agreements, uh, the kinds of things that, that lead to robust systems.

6. 1:26

   Um, and then we'll spend ninety minutes on the rest of the owl, uh, the rest of what it takes to make a successful AI-powered app, just because there's so, uh, very little to say just yet about how to engineer these robustly.

7. 1:37

   But we'll talk about what the, what the emerging consensus on what to do is, uh, so far and what tools are out there and available to start accelerating that process.

8. 1:49

   Um, so yeah. So for inference, what, uh, what are we talking about when we're doing inference workloads? How do we decide between using open and proprietary models, uh, to do that inference?

9. 2:00

   Um, where do those models live? Do they live on a device? Do they live in a cloud server? Um, uh, and then we'll, uh, spend some time talking about what it takes to self-serve inference.

10. 2:14

    Um, for the rest of the owl, we'll talk about architectures and patterns. So what are the emerging kind of, uh, patterns for usage of, of large language models in AI applications?

11. 2:26

    And we'll talk about monitoring, evaluation, and observability, which is how we try and actually improve applications that fit those patterns over time.

12. 2:37

    Um, yeah. Any high-level questions? Um, my mic's off. The Zoom is not hooked up. No questions?

13. 2:47

    Probably not. Still at a very high level. Um,

14. 2:51

    no? Just stretching. Great. Yeah, maybe we should all stretch. Um, yeah. So who, uh, who am I? And before we dive in, like, who am I and why should you listen to me tell you about any of these things?

15. 3:02

    So, uh, my name's Charles. I like to teach people about AI. I've been doing it for a while now. Uh, I went to Berkeley, studied neural networks back in the twenty tens.

16. 3:11

    Um, taught people about, uh, how to use them and Bayesian networks, RIP, um, for data science. Um, and then I worked, uh, in developer relations and education for Weights & Biases, um, a previous generation MLOps tool.

17. 3:26

    Um, generation times being shorter than more doubling times, I guess, at this point. Um, and then for the last two years, I've been working with Full Stack Deep Learning, uh, teaching not just things like the math of machine learning or how to do monitoring of an ML application, but how to build an application that uses ML from

18. 3:46

    soup to nuts, uh, from the GPUs up to the user experience.

19. 3:52

    All right. So now let's dive into the first half here on inference and specifically what is actually going on. Why does it cost, um, so much to, uh, to ping OpenAI?

20. 4:05

    Um, and so generative model inference, the kind of inference that's done, uh, by these-- the generative AIs is, uh, you see when you use it as a-- via an API, you see a kind of data-to-data function where both sides of that data are, are human interpretable data.

21. 4:24

    So things like text, images, sounds and out come back new generated text, image, and sounds. So for example, this is from the PaLM-E paper from Google. You might show a picture of a restaurant and the question, "If a ro- robot wanted to be useful here, what steps should it take?"

22. 4:39

    And the output of this, uh, uh, language modeling generative API would be, you know, clean the table, pick up trash, pick up chairs, wipe chairs, put chairs down. Um, which could then be sent to a cleaning robot and now you've got something, you know, that's a, that's a pretty useful system there.

23. 4:56

    So this is what you see from the outside. Um, but as you start, like, digging in a little bit, you'll start hearing about things like tokens, um, and, and log probs and temperatures and, um, to understand what's going on and, and weights and networks.

24. 5:13

    And to understand what's going on, you have to realize that this has been broken down into kind of three pieces. One part that goes from what humans understand, like text and images and sound, to what neural networks understand, uh, and then one part that goes back, uh, and in between the operations of a neural network that operates

25. 5:31

    on arrays and returns arrays. So tokenizers take in text that a human can read and turn it into an array of numbers, a tensor. Um, just due to the kind of physics-y background of people in the ML community, they ca-- what people would call array is, I guess, in other software engineering communities, they get called tensors.

26. 5:53

    They also-- because they have, um, uh, derivatives attached to them a lot of the time. It's like there's a connection there, but really are an array of numbers.

27. 6:02

    Um, those tensors get turned back to things that humans care about, um, and in between neural networks map tensors to tensors.

28. 6:13

    This step is the bottleneck. This step is the hard part. There's interesting stuff going on in the sampling process. There's interesting stuff going on in tokenization. There's cursed stuff going on in tokenization.

29. 6:23

    Um, but this step is the bottleneck. This is where the me- vast majority of the engineering time is spent. This is where the vast majority of the compute time, the memory are spent, and this is the interesting part, and this is the part where the engineering focus, uh, needs to be.

30. 6:38

    Um, or this is the part that you farm out to somebody else. Um, so diving in, double-clicking on that, uh, tensor-to-tensor arrow at the bottom, that, uh, a neural network is kind of a, a fancy term for a composition of a bunch of tensor-to-tensor functions.

31. 6:57

    If you have a function that takes in an A and returns an A, you can just stack those one after another. Um, and that's what gives neural networks their kind of LEGO-y flavor.

32. 7:06

    You can grab, uh, bricks from one set and bricks from another set and, uh, attach them to each other.

33. 7:12

    Um, so this neural network, this is an ancient neural network, uh, the Inception v3 model from Google that was state-of-the-art in computer vision for a few months in the twenty tens.

34. 7:21

    Um, and each of those little blocks there takes in a tensor and returns a tensor. Uh, and so it starts with a tensor that looks like an image, so it's got a red, green, and blue channel.

35. 7:33

    It comes out something, uh, that eight-by-eight-by-twenty-forty-eight example there. That's probably somewhere in the middle of the network. It's not the final classification output. Anyway, big block of numbers, and they get passed into each other.

36. 7:46

    Uh, and then each one of those is itself parameterized by a tensor. So it's not just like, uh, a map that you could kind of like write down by hand that's like add one to every entry or something like that.

37. 7:57

    Um, it's, uh, it's defined by like another big pile of, uh, of numbers, the weights and biases of the neural network. So this is, uh, the weights from the first layer of a computer vision network, the AlexNet that kicked off this, uh, whole deep learning revolution, um, and it's these, these are, these are little-- this is a

38. 8:19

    visualization of a little block of red, green, blue, uh, n-- uh, like tensor with three color channels in it. So humans can actually look at it and interpret it unlike the rest of them and see what's going on, that it's got little things for detecting edges and textures and color differences and things.

39. 8:39

    And so, uh, we want to run a big tensor-to-tensor map, and we are going to parameterize that with a big pile of tensors. And before it's time to actually serve users, those tensors need to be generated by the training process.

40. 8:56

    And back a, a couple of years ago, um, or even as recently as eighteen months ago, this would be the part where we would stop, talk about gradient descent, optimization, statistical learning theory, and, uh, you know, GPU acceleration and all the things that are needed to turn something into-- to, like, get those numbers, to get things like

41. 9:17

    that involving like grabbing, you know, Hoovering up a bunch of information from the internet, uh, without consent, um, and then crystallizing it into those piles of numbers. But nowadays, you don't have to do that anymore.

42. 9:31

    Specialized foundation modeling teams, uh, generate these weights, and then they either put them behind proprietary service or they share them with everybody else to use, um, and so we can skip past all of that stuff and jump into the actual application.

43. 9:45

    Um, uh, yeah. So any questions before we start talking about, you know, where those, you know, where those weights come from, uh, or, or rather what the various ways to get ahold of such a set of weights or to be able to use such a set of weights are?

44. 10:00

    Um, any questions at the level of what we're trying to do with inference?

45. 10:06

    Probably pretty clear. That's, uh, maybe a reminder from the one-oh-one stuff. All right, so now let's start diving a little bit deeper.

46. 10:16

    So you want to run an AI. Uh, your fir-- one of the first choices that you need to make, as is, you know, pretty common with, um, with software, is a build versus buy question.

47. 10:28

    Um, are you going to like m-- Are you gonna make this, uh, you know, out of existing open components, or are you going to, uh, are you gonna kick it off to a service?

48. 10:39

    So, um, there's in the proprietary corner, there are a couple of players. In the open corner, there are a couple of players. Let's walk through what those are and, um, what the sort of dividing lines are and why to choose one or the other.

49. 10:54

    So there's, uh, a number of proprietary modeling services, uh, like Anthropic's, though, uh, from whom we just heard. Uh, and the good thing about these proprietary models is that they are the most capable models out there.

50. 11:09

    Uh, so this is from the LMSYS, uh, leaderboard, maybe the, uh, the Hugging Face hosted version of that leaderboard. If you look at the, the top five models on that are all proprietary models, and they're all from OpenAI or Anthropic.

51. 11:26

    Um, so there are a couple of other players out there in, in the future that, you know, they could release really high-quality models. Um,

52. 11:35

    but for now, this is the, the, the state of things. So if you need the absolute highest level of intelligence in your application, uh, then you wanna roll with one of these.

53. 11:47

    Um, it's also often common to start with some of the most highly capable models and then kind of, uh, prove out your application there and then move to doing it, uh, like with a less capable model that's cheaper or easier to run, uh, the sort of like rewriting it in Rust or something.

54. 12:06

    Um, so the usual concern with using a proprietary service, there's a number of them, including things like vendor lock-in, um, but one of the, one of the ones that comes up immediately is like how much is it gonna cost me to use a proprietary service and can't I save money by doing it myself?

55. 12:23

    So the fact that the capabilities are higher with the proprietary models is one reason to say, well, you're not going to get exactly the same thing right now, um, using an open model.

56. 12:33

    Um, but then the other kind of kicker here is that the proprietary models are priced very affordably. Um, so this is something that's, that has been the case. Uh, this right quote here is from a blog post I wrote back in like January, um, when the only open large model was GLM 130B from Tsinghua.

57. 12:53

    Um, and at that time just like trying to get the thing running, uh, in a day I got to within an order of magnitude of the cost of OpenAI, but you know, on the up, on the upper end.

58. 13:04

    Um, and then more, for a more recent, uh, and more serious example, the folks at Honeycomb, uh, made a natural language SQL, uh, kind of transformation, maybe not SQL, but query language, uh, transformit- transformation, uh, AI product and like their opinion was that OpenAI was very inexpensive to run for their task.

59. 13:24

    Uh, and so you'll find it seems that like nobody's attempting to, uh, like extract rents from monopoly pricing the way that you can get with some proprietary services, uh, where they know that you have no choice but to pay them $10 million for a MATLAB license every year or whatever.

60. 13:42

    Um-

61. 13:43

    Not to pick on anything

62. 13:45

    ... not to pick on it. Sorry. For a license for an array programming language. [laughs]

63. 13:50

    Um. [laughs] Uh, yeah so and the costs here, you know, a, a dollar per a million tokens for Claude Instant, $10 for a million tokens for Claude 2, uh, relative to how much we're used to paying like humans for text.

64. 14:04

    You know, that's, that's like a pretty decent deal. Anthropic has been slightly more expensive than OpenAI without like a clear win on, um, capabilities, uh, but that's the current state of things.

65. 14:17

    They do have longer context windows which is kind of nice but, um, uh, but yeah most people for the pricing reason and the capability reason, uh, choose OpenAI at this time.

66. 14:29

    Um, so the bad thing about proprietary models besides like, you know, uh, hearing Richard Stallman screaming in the back of your mind all the time, uh, is that proprietary mode- models cannot offer you full control by dint of their like very nature as proprietary models.

67. 14:47

    Um, so Alex Gravely created, uh, Copilot, part of the team that created Copilot at GitHub, um, was celebrating that GPT 3.5 Turbo Instruct, a recent release from OpenAI, had brought back log probabilities.

68. 15:00

    So you can see what... not just like what text did the model generate, but what probabilities the model gave each token along the way. And back when the, back when it was just the GPT 3 API and the playground you could see that information and that's where a lot of the like early work on sort of like

69. 15:18

    intuition about prompt engineering came from being able to see those numbers and there's all kinds of cool techniques that you can get up to if you have those log probs, um, and if you can manipulate those log probs.

70. 15:28

    Uh, so yeah you can read out confidence information, you can use it like during your development process to sort of like more gather more rich information about the system.

71. 15:37

    Um, and there's really, you know, you are, you're interacting with a probabilistic model and you can't see the model's probabilities you're like fundamentally hamstrung. Um, and so that was mid-September and then like a couple of days ago they turned that off.

72. 15:52

    Um, [laughs] and the reason why is because if you give somebody that amount of information they can start to reverse engineer your model pretty quickly. Um, and then you are stuck in the situation of IBM creating the personal computer and then a bunch of people with soldering ion irons and oscilloscopes turn around and make clones of your machine

73. 16:09

    like within a year. Uh, so proprietary models are like fundamentally disincentivized from giving you that level of control despite the fact that it's very critical for, um, for like actually effectively operating the system.

74. 16:24

    So there needs to be that capabilities edge, um, that like raw capabilities edge in order to, uh, make up for this fact.

75. 16:33

    Um, then lastly maybe some people work in an enterprise, uh, in this room. Don't, you don't have to out yourself, but maybe some. [laughs] Um, and if you're operating in this sort of situation you can't just ship a ping to an external API out there, uh, into your business.

76. 16:52

    You know, people want to know about governance. People want to know about GDPR compliance. Um, and the, one of the nice things about OpenAI's offering probably true about Anthropic by this point and definitely true about Google AI soon is there's a nice white glove enterprise tier of, uh, around this that gives you like right underneath like, you

77. 17:15

    know, launch an artificial intelligence application and achieve your childhood sci-fi dreams is built-in security and compliance.

78. 17:22

    Um, we will spend $20 billion on cybersecurity so that you don't have to. Um, so this like if you are in a situation where you need to like, you know, assuage concerns about data privacy, um, this sort of enterprise tier, um, yeah SOC 2 compliance et cetera is, uh, can be really critical for making your life easier.

79. 17:45

    Uh, any questions about proprietary models and such?

80. 17:48

    How much more expensive is it to run consuming the API versus the cloud rights?

81. 17:53

    Hmm. With, uh, with Azure?

82. 17:55

    Yeah.

83. 17:56

    I want to say that Azure was cheaper at the start. Um, but maybe it m- uh, I haven't ha- had any reason to use it. Um, so I'm, I'm not sure.

84. 18:05

    And uh, just a follow-up question. Like are there like any, um, trade-offs in general with regards to performance? I know-

85. 18:15

    Ah, so for... You mean for the enterprise tier versus... Yeah. So the o- the enterprise tier also offers, like, an actual SLA, which the OpenAI API, like, doesn't, um, and is, like, geared...

86. 18:28

    That's, that's, you know, that's maybe another very critical feature besides security and compliance. Like, they will promise that you will get a response and not, like, a 500. Um, and they have much more generous, uh, rate limits and things like that.

87. 18:42

    And I think they also offer, like, a little bit more control over stuff, so you might be able to do some fine-tuning that you can't do via the generic OpenAI API.

88. 18:51

    Oh, the fine-tuning for, for Azure, uh, limited to just, like, three models? Okay. Yeah. But there's also limits on the public API for fine-tuning, right? Yeah. Um, I personally did not find it particularly useful, the, like, fine-tuning API, both, like, hard to use and not clear benefits.

89. 19:09

    Um, I think the example from Gradient did show that, like, if you want to achieve a style and you don't want to spend money on, like, context to set up that style, uh, then maybe you can win with fine tunes.

90. 19:21

    But it's, like, it's not really a successful way to inject new information, so you aren't saving on the tokens that you would retrieve, um, and you have to pay more to inference a fine-tuned model, and that just goes down to the fundamental, like, you're asking them to do more work for you, um, and they can amortize that

91. 19:38

    cost over fewer users. And so it's just always going to be more expensive. Um, and so, uh, yeah, so that limits the utility of those fine-tuning APIs. Yeah. Please definitely, like, ask questions, um, uh, customize to what people are interested in.

92. 19:55

    And also, if I don't know about something, like, please do, uh, interrupt.

93. 20:02

    Great.

94. 20:03

    Google has Vertex.

95. 20:04

    Yeah. Yeah. So-

96. 20:05

    Do you know about it?

97. 20:06

    Uh, well, so Vertex is a little bit different from this, I think. Um, I think of Vertex, which I was gonna talk about later, as sort of like a...

98. 20:16

    something that I can launch my own services into as opposed to, like, uh, oh, here's a private version of the PaLM Bison API. But maybe, maybe... Is that part of Vertex?

99. 20:27

    Yeah, it's after the-

100. 20:27

     Okay. Great. Yeah. Um, so yeah, that's, um... So they're already available for Google AI. Does anybody know if Anthropic's AWS, like, uh, pri- like, enterprise offering is up yet?

101. 20:41

     Petra?

102. 20:42

     Yes. Yes.

103. 20:43

     It is. Yes.

104. 20:45

     Great. Yeah. I, like, refuse to make slides about this stuff more than, like, forty-eight hours in advance, and I still find myself getting cut. Like, yeah.

105. 20:56

     Yeah. So proprietary models and open source models, like, which ones you, are you betting on? Which one's good to invest? [laughs]

106. 21:05

     So the question was, uh, between proprietary and open models, which ones are you betting on? Um, gambling is illegal in the state of California, so yeah. [laughs] Um, uh, we'll get, we'll get there.

107. 21:15

     So, um, let's talk about the open models, and then we can answer or, uh, open up that discussion. Um, so open models are less capable but catching up, um, and their hackability is very powerful.

108. 21:28

     Um, so going back to that leaderboard that I showed, if you look at the next five out of ten, um, four of them are Llama Two model. Uh, actually, so this column here, to be clear, is the license.

109. 21:42

     Uh, so the top five are proprietary, like there's no, uh, there is no license for those weights. Um, for the bottom five, they have, uh, special licenses. Um, so these are fine tunes of Meta's Llama model series, and this model series has, like, kind of captured mind share in the free and open source software world.

110. 22:05

     So m- a lot of the people who are, like, hacking independently, um, and, uh, you know, making public Git commits, um, and, you know, funded by the Linux Foundation and things like that, these people are working in the main on, uh, adjustments to or improvements to the Llama model series.

111. 22:24

     Um, and this is really critical because the secret to, like, the success of open source software in general is the ability to do this kind of, like, highly parallelized development where lots and lots of people are adding tiny little features and, like, you know, going out into the last mile and, and adding those tiny little things that

112. 22:43

     they need, um, and sort of like making use of all that, uh, work by others. So insofar as you're able to do that, you are able to provide useful open source software that can compete with software that's made by, you know, highly remunerated teams, um, you know, in, in Northern California.

113. 23:03

     Um, so the important question, uh, is this actually, uh, open source? So you'll notice these licenses here do not have friendly beloved names like LGPL or MIT or Apache.

114. 23:16

     They, they have, uh, a special unique name. Um, and that's because Meta's license for the Llama Two weights, the Llama One weights were released under a research-only license and were only sent to certain people and then were immediately, like, torrented and the license was violated.

115. 23:31

     So they gave up on that, uh, on, like, fully controlling it. But they did say you cannot use the, uh, data or output to improve other large language models.

116. 23:41

     You can only use it to improve Llama models, um, and also release under the same license, which is pretty typical with open source, um, which, uh, is partly an attempt to sort of like capture this, like, as people are doing parallel development, they should only be contributing to the development of this, um, this branch.

117. 24:00

     Um, and then also, if your product's monthly active users in June of two thousand twenty-three was seven hundred million users or above, um, you're not allowed to use it.

118. 24:09

     Or sorry, you have to pay for a special license. Um, and that, uh... So apologies to anybody, you know, who's, um, you know, if you're running an app with more than seven hundred million users, um, uh, you'll have to go elsewhere, I guess.

119. 24:23

     But the Um, the key thing is that these are violations of the, like, sort of agreed terms of what makes something an open source license according to the Open Source Initiative, who, you know, has some, uh, claim to controlling how that term is used.

120. 24:41

     I didn't... They... I don't think they ever ended up getting a trademark. Um, but they, uh, you know, they are... got the community aligned around a small number of licenses that...

121. 24:51

     and around a key set of principles that include, like, you can't tell... You can't say who's allowed to use this software, for example, which is included in the Llama license.

122. 24:59

     So there's, like, uh, uh, they're... They have opened up a multi-stakeholder process to define open source AI. This is occurring at a time in which o- like, the meaning of open source is also being contested in sorta like software as a service, so things are a little, things are a little tense there.

123. 25:16

     Um, but hopefully we'll come to, uh, an agreement as a community on what that means. Um,

124. 25:23

     so this is a fast-moving space still, so just because you get mind share early on, if, like, things change rapidly, that doesn't mean, like, Llama's locked in forever. Um, so Mistral, for example, dropped a model like two weeks ago, um, that y- at only seven billion parameters was outperforming, um, like larger models in the 13 to 30

125. 25:42

     billion parameter range, and those models were outperforming the previous models, um, at their size. So there's, like, except insofar as those things, like, continue to get updated, um, you know, they, uh, yeah, they can be outcompeted.

126. 26:00

     Um, the other thing to watch out for is that there are a lot of people who are very excited about, like, taking on the Death Star of OpenAI or whatever and, um, get very excited about these open models.

127. 26:10

     There's also some political things about the politics of how ChatGPT likes to respond to questions versus the politics of how people who meet other people in Discords like to respond to questions.

128. 26:21

     Um, and that can, that sort of, like, enthusiasm can lead to, like, pretty big errors. So for example, there was a lot of excitement about these models that were take, take an Open Llama model, uh, the, the weights from that, and then grab like 10,000 requests from the OpenAI API or scrape like r/ChatGPT or whatever, and then

129. 26:45

     just fine-tune. Like now there's a data set, use it to fine-tune the model, and those are the ones that were, like, up there on the arena, uh, on that, uh, like ELO ranking, um, from the leaderboard that I showed before.

130. 26:59

     Um, and the, there was a claim that these, uh, had like 90% of ChatGPT's quality. They're only seven billion parameters. Like, you know, people were, like, very enthusiastic about this back in April when I was talking about this.

131. 27:13

     A lot of people were like, "Why are you even talking about OpenAI anymore? We've, like, you know, Vicuna's done it." And like, yeah. Uh, so, um, this is a fake screenshot, uh, that's, uh, from a paper or that I modified from, from a paper about, uh, about this, this topic of how well these models work.

132. 27:31

     Um, so this is one output from a language model anonymized, um, on answer, uh, how does actor critic improve over reinforce? Um, so this is an algorithm from reinforcement learning.

133. 27:43

     So one language model says, um, actor critic algorithms are an extension of reinforce that combine both policy-based and value-based methods. Um, it's got a critic network, it's got advantage estimation, it's got function approximation.

134. 27:57

     So that's one answer. That's A. Answer B, uh, actor critic algorithms are a type of reinforcement lear- learning algorithm that improves the reinforcement algorithm by combining actor, policy, and critic value components.

135. 28:09

     Actor critic algorithms use a single critic. Reinforce has a separate critic for each action. Actor critic algorithms learn a policy and actor simultaneously, but s- reinforce learns them separately.

136. 28:19

     Um, so, you know, those might seem fairly similar. So does anybody have a strong preference for, uh, answer A here?

137. 28:31

     Anybody have a weak preference for answer A over answer B here?

138. 28:37

     Some people raising like a, like a, a soft hand maybe. Yeah. Um, anybody have a strong preference for answer B?

139. 28:48

     Maybe one, maybe two. And a weak preference for answer B?

140. 28:52

     This one's got like a... They, they, they both got these nice numbered lists, you know, which looks very authoritative. Reminds me of a medium article, which is likely to be true, of course.

141. 29:01

     Um, uh, so the, uh, so, uh, answer B comes from, uh, I wanna say this was GPT-4. Yeah. Um, answer B comes from GPT-4 and... Oh, wait, wait.

142. 29:18

     Sorry. Answer A comes from GPT-4 and has the advantage of being correct. Um, answer B comes from one of the, uh, fine-tuned models, um, and is like gibberish, basically.

143. 29:31

     Um, and so this just, you know, like just having humans rate the outputs of, of language models in the way that a lot of those, uh, leaderboards were constructed, um, did not, like, it didn't have any grounding in the actual utility of the answers.

144. 29:46

     It was just a lot of people going like, "Looks good to me." Like, "Nice. Yes. Merge." Um, and [laughing]

145. 29:53

     um, without, like, knowing whether it was actually right or not. Um, and so, uh, there's a nice paper from some folks at, uh, Berkeley about, um, sort of walking through, like, what's going on.

146. 30:04

     Basically, the models are picking up style from a fine tune, which is things like that delightful little split into bullet points and like, you know, like a very authoritative and friendly educational style, um, but not like actual knowledge, not like reasoning capabilities.

147. 30:21

     And like a lot of people in the open modeling communities, like, sort of missed this or, like, willfully ignored it. Um, some of the sharpest people were definitely up on this.

148. 30:30

     Like, the Guanaco paper, for example, mentions that there's, uh, some issues with evaluation. Came out before this paper, um, but definitely a lot of people missed it.

149. 30:40

     Um, so the, the immediate question that comes up is like between these open models and these, uh, proprietary models, who's going to win long-term? Like, who should I bet on?

150. 30:51

     Um, and in some ways, I think that's a bit of a misguided question. Um, so consider operating systems, uh, like the first operating system roughly was System 360 from IBM on mainframes, extremely closed.

151. 31:07

     Um, in the '80s, there was a rash of operating systems, most of them closed. The original Xerox pilot on the Xerox Star, uh, was extremely closed. DOS and Win-- DOS was closed.

152. 31:19

     Uh, Mac OS at the time was like completely closed. Um, and there were Unix operating systems that were kind of like mixed. Um, then over time, the like closed versions of Unix lost out to more like friendly licensed ones, and in particular to GNU/Linux.

153. 31:35

     Um, and there's been a bit of a trend towards open, uh, open operating systems kind of taking more mind and market share over time. Like data centers have more Linux in them now than they did in two thousand and five and than, than they did in nineteen ninety-five.

154. 31:49

     But, uh, from what I can tell, it's still like seventy percent plus Windows,

155. 31:54

     um, for, uh, for operating web servers. In mobile phones, we also have an open operating system and a closed operating system, and these things have been able to coexist and serve different needs for different organizations throughout like the history of operating systems.

156. 32:09

     Um, and the same is true of databases. Uh, so back in the '70s and '80s, it was Oracle and IBM's DB2, um, which is still around, I found out.

157. 32:22

     Um, [chuckles] like you live too long in, in the, in San Francisco, and you forget that there are people who use IBM DB2. Um, in the '90s, there was some like consolidation around more open implementations of, of SQL, uh, and in the 2000s to 2010s, there was the NoSQL movement, but that was still like mostly open source databases.

158. 32:45

     Uh, so there's been a lot of like movement in the direction of open databases. Um, with streaming databases, we have both pro-proprietary and open options. If you look at the top ten, uh, databases as ranked by db-engines.com, which is...

159. 32:59

     Uh, you could quibble with the ranking thing, but the, the key point is that there are like... It's like half and half split between, um, between proprietary databases and, um, and open source databases.

160. 33:12

     Uh, and that has been like relatively stable over time with like a soft, maybe a soft trend in the direction of open, uh, databases. Uh, so with these like very like la-language models, foundation models are this very like low-level s- uh, component of a software stack, more like an o- like an operating system or a database, I

161. 33:32

     think, than like, um, you know, than a SaaS app or a, or a UI. Uh, and because of that, they're likely to be subject to some of these same forces that say there's some people who want to work s- one way, there's some people who want it to work in another, and for some of them, that openness,

162. 33:48

     that hackability is gonna be critical. For others, the like reliability, the existence of a white glove enterprise version is gonna be really critical, um, and those will allow these two things to coexist.

163. 33:58

     Um, and, uh, yeah, and the CEO of Hugging Face liked my tweet when I said that, so it's probably true. [laughing]

164. 34:09

     Um, but I think a lot of people are like, "Well, no, but like who's, like who's gonna win? Like, um, who should I bet on?" Um, and I think the closest thing to an answer that I have is that if capabilities requirements saturate, if people no longer want the absolute smartest model out there, they just want a

165. 34:26

     model smart enough for XYZ, then open models will probably catch up and then like start to dominate. Um, the thing that keeps open models behind proprietary models is the extreme expense of maintaining a large resource team and like, you know, continually constructing new data centers at an increased scale, um, uh, to the tune of like five hundred

166. 34:46

     million dollars in order to hit that next capability level before everybody else. But, you know, uh, uh, at a certain point, processors got fast enough that people were not like clamoring for the next upgrade as soon as possible.

167. 34:59

     Um, and at that point, we're starting to see like a little bit more opening up in the sort of like in the chip space with like RISC-V. Um, and so like a- like with, in any number of other technological domains, you've seen that when requirements start to saturate, um, then like open, uh, like open versions can catch

168. 35:16

     up. Um, if they are unbounded and it's like, you know, uh, what's a good example? Like RAM, like nobody has enough RAM, everybody wants more RAM. I don't think there are any like open like attempts to make like an open RAM architecture or something.

169. 35:32

     Um, and that's becau- and one reason why I think is capabilities requirements there are re-remain unbounded. Um, and so, um, if that's the case for, uh, cognition and AI models, then proprietary models should be able to maintain that edge in, in capabilities, which would sort of tilt the balance in favor.

170. 35:51

     More people would say, "Oh, no, I need this, this proprietary thing." Um, so it's the closest to an answer that I have. Um, yeah. Any questions on that front before we dive into, uh, um, where we actually run these models?

171. 36:05

     No.

172. 36:05

     I was curious, uh, what's the language support for these language, for these models? Like-

173. 36:11

     Yeah

174. 36:12

     ... can you use, um, you know, like a language other than English with these models?

175. 36:17

     Yeah. Um, so the question was what kind of language support do these models have? Um, and because it's only an API call away, you can of course use Python or Node or whatever you want.

176. 36:29

     Uh, no. So the question was about like these, these are language modeling like machines, what languages do they model? Um, and the basic answer is that the more text in that language that is available on the open internet, the better the language models will be on, on that language.

177. 36:46

     So they are like I wanna say GPT-4 is smarter in

178. 36:53

     m- maybe smarter in Malayalam than it is in Mandarin. Um, I forget. There's like some interesting inversions of like number of people who speak the language versus how, uh, how intelligent the language models are.

179. 37:06

     Um, so I think a lot of them release benchmarks that say like how multilingual is this language model and for which languages. Um,

180. 37:15

     there is... You run into the fundamental token constraint of like you need, uh, you need

181. 37:22

     existing la-- like you need examples of that language that you can get ahold of in order to train the model in them. Um, and there just are more English tokens.

182. 37:32

     Um, but for a given capacity, you can probably achieve like higher quality in a specific model by looking for, um, by looking for a model trained in that language.

183. 37:44

     So there's definitely some like good [REDACTED:age] [REDACTED:political_affiliation] European endeavors to make like a French language model that insults you if you ask it for stuff in English. Um, which it of course picks up just from reading French.

184. 37:56

     Um [audience laughing]. Um, but yeah. But the, but the core models, like they support English really well. The instruction fine-tuning and the RLHF is actually mostly applied to them in English since the annotators who, uh, enforce that policy, uh, through their examples are mostly writing in English.

185. 38:15

     Um, so fun fact, you can get ChatGPT to te-- and Claude probably to tell you how to build a bomb if you ask in the right low resource language.

186. 38:22

     Um, uh, just, uh, fun facts about language models. Um, yeah. So that, that does, that, that is a problem and it does sort of like inge-- uh, it has a multiplying effect on Engl-- the English language's kind of like cultural hegemony, um, which is a bit unfortunate.

187. 38:39

     Yeah.

188. 38:40

     So for the, for languages that are less represented-

189. 38:44

     Mm-hmm

190. 38:44

     ... is, um, is reasoning capability lower or is understanding lower? And also are there arbitrage opportunities in translating first, uh, English prompts for example translating first, um, and then doing the reasoning at that time?

191. 39:01

     Yeah. I'm unaware of any aca-- like, you know, any benchmarking work on this. My gut tells me that translating to English first, doing chain of thought, and then translating back to the original language would work better.

192. 39:14

     Um, you'd, you're kind of like wondering whether the lost in translation effect is bigger than the like boost of chain of thought in English. A lot of the like circuits in language models are very token specific.

193. 39:27

     Um, and then, yeah. So this is like the, like just one example. If you ask it who Tom Cruise's mother is, then it answers better than if you ask it that [REDACTED:gender]'s name's son.

194. 39:39

     Uh, I don't know her, her name at all, um, so I can't really do this example effectively. Um, ChatGPT wins again. Um, but the, uh, there-- So that's like an example of a very, of a token specific circuit.

195. 39:50

     It's like related to Tom Cruise. Uh, and so you can see like it's not reasoning the way that a person would or that, or that you would guess from like, you know, how, how you would think about a knowledge graph or something like that.

196. 40:03

     Um, and so that's where you get these unintuitive things. Um, but yeah.

197. 40:09

     So I saw you, um, used to or maybe still do a deep learning bootcamp-

198. 40:15

     Oh yeah

199. 40:15

     ... that you offer. And I was wondering, like one of the pieces of the whole event is, okay, there's these reasoning-as-a-service APIs now, right, where you can do a lot more things without having your own ML team.

200. 40:29

     Um, in your opinion, for the typical AI engineer who wants to build things, does it still make sense to learn some amount of deep learning, some amount of ML, you know?

201. 40:40

     And is it like just watch some videos, read some blog posts? Is it like actually do like a bootcamp? Is it like actually read some papers? Like what would you recommend?

202. 40:51

     Yeah. That's a great question. Um, for individuals, I think it's a matter of your personal interest in understanding the modeling. Like,

203. 41:06

     the, I guess the analogy I would immediately jump to is as an individual developer, you can get away with not knowing anything about databases. Like, I have done that.

204. 41:15

     I couldn't write a B-tree right now. I don't wanna ever learn how to do that, like, and to think about page sizes and yeah, it makes me ill to think about that.

205. 41:24

     And whereas I get excited if I wake up in the morning and I can think about Bayesian inference in language models. And so as an individual, I think you can like kinda be guided by your, like what you find most exciting.

206. 41:36

     As a team and as an organization though, if you have nobody who understands databases in your organization, you're probably gonna be in trouble. Um, just like it ends up like most applications require like pretty decent knowledge of databases and when they go down or when they need to be configured.

207. 41:53

     Even if you are using Redshift or, you know, you're using some managed service, being able to like understand some stuff about them is actually critical for debugging and being able to know when you need to switch managed services or like, yeah, or how to reconfigure them.

208. 42:08

     So I think like the direction that we're gonna go is to evolve there. It's a question of whether you wanna be a, a, a

209. 42:17

     site reliability engineer focused on LLM reliability or, um, you know, a, um, um, a modeling engineer or whether you want to be like more at the like application layer.

210. 42:27

     Um, yeah.

211. 42:30

     So just an interesting one. What's your personal opinion on this? There's a handful of companies that have got recent funding to build like vertical oriented commercial models for finance, healthcare, et cetera.

212. 42:41

     Yeah.

213. 42:41

     What's your opinion on those?

214. 42:43

     Yeah. So the question was what about these models that are foundational but s- like less broad? So it's like a foundational model for law, a foundational model for, um-

215. 42:55

     Healthcare

216. 42:55

     ... for healthcare. Yeah. I- Um, my experience has been that if you bet that some capability is not gonna be available in a language model, um, or in a foundation model, like you will get-- you will lose that bet.

217. 43:12

     Um, uh, so just as an example, in the deep learning boot camp, uh, we spent a long time trying to make an optical character recognition system. Uh, and it's like, you know, it's the like pinnacle of the class when you can finally like deploy a web service that does optical character recognition.

218. 43:29

     And that's like an accidental side feature of GPT 4V, and it's like better at it than the thing that we built. Um, and a lot of ML teams have experienced something like that.

219. 43:39

     Um, so I worry-- I would worry if, if somebody were like offering me that as a job opportunity, for example, I would worry that it's gonna get like scooped on either side by a hyper-specific model that's like ten times more efficient and isn't like a generic healthcare model, but is like a, um, uh, ultrasound for the heart

220. 43:59

     model. This is one I worked on before. Um,

221. 44:03

     yeah. Uh, or just send it to the ChatGPT API or to the GPT API.

222. 44:11

     Great questions. Um, so, uh, maybe another reason to think that there might be like a little more alpha in, in actually learning more about the models is, um, inference doesn't have to be executed over a network.

223. 44:26

     It doesn't have to be executed like in some central server. There are lots of reasons why you might want to execute your, uh, uh, your inference on an end user device.

224. 44:39

     Um, so we'll talk about the different types of end user devices and the different constraints they put on inference and the implications,

225. 44:46

     um, like engineering and strategic, and then also, um, talk about what the options are for doing things over a network.

226. 44:54

     Um, so running stuff for end users is li-- uh, like where the-- Sorry, the end user actually executes it themselves is not quite there yet, but it's like, uh, it's getting there and maybe a little bit faster than I personally expected.

227. 45:07

     Um, like I've run Llama two, uh, thirteen B on this very laptop, um, without it catching on fire. Um, so there's, uh, there's some hope, uh, that there will-- that that will continue to get better.

228. 45:23

     Um, and so this, uh, this is critical for latency sensitive applications. So like being able to actually execute the inference at the same place that the user-- where-- at the same place where the user is.

229. 45:37

     Um, and the reason why it goes back to this, like this famous set of numbers every engineer should know from, uh, Peter Norvig and Jeff Dean at Google, um, which is that the time it takes to send a packet, just one packet.

230. 45:51

     So probably-- This probably isn't even a whole HTTP request. I'd have to check again. But let's just say you send information back and forth from like here in California to Europe and back.

231. 46:02

     It's a hundred and fifty milliseconds. Um, and there's like a number of kind of made-up numbers in the UX world about like how fast you need to be for something to feel interactive.

232. 46:15

     Um, so one of them going back to like the seventies or eighties is the Doherty threshold, which says, "If the user and the computer can interact with each other in under four hundred milliseconds," then the like human won't feel like they're waiting on the computer.

233. 46:28

     And as you're programming things, you won't like end up blocked on human input. Um, but you'll-- and you'll still have plenty of time for doing stuff in, in side threads and things like that.

234. 46:38

     Uh, so if you were s-- like if you have this like do a network call every single time, you're using up like a third of your budget just on like waiting for information to come back.

235. 46:47

     And now you're gonna spend a ton of engineering effort on like trying to find ways, things that you can do asynchronously during that like pa-- like that network call and like you can, you can work around it, but it is punishing.

236. 46:59

     Um, and that's like there are even tighter like reaction time things like if you have a self-driving car, um, you can't wait a hundred and fifty milliseconds to, uh, to find out that you need to brake.

237. 47:11

     Um, so, uh, yeah. So and the nice thing about this, uh, the other benefit to it, beside it being necessary in some places, is that if end users run the computation, then you don't need to pay for it.

238. 47:27

     Um, so your inferencing costs can be zero dollars, which would be, which would be great. Um, so the cost that you pay, um, is control. So you have less control over the execution environment.

239. 47:41

     Um, your ability to do telemetry and see what is going on is limited. Uh, people don't like it when you like carefully observe their activity using software on their machine.

240. 47:52

     But if they-- you put the same software at a URL, you can spy on them as much as you want, and they don't get mad. Um, so you lose out on telemetry.

241. 48:01

     You, uh, have to worry about compatibility with different execution environments, and you have to actually support past versions, unlike, uh, if you're running it as a service. Um, or rather if you like, you know, con-control the execution environment.

242. 48:15

     Um, so the things that are unlocked by this are some of the best applications here, uh, like some of the most exciting ones, especially to me. So, uh, used on smart-- used in smartphones, used in robots, used in wearables.

243. 48:28

     Um, so Google just in the past couple of days announced that the Pixel Pro 8, um, is gonna have, uh, large language models directly on device. Um, they mostly showed off stuff that looked like kind of like summarization and some like light image editing.

244. 48:42

     So not like full on like, you know, like, "Hey, Siri, um, why did the Ottoman Empire fall?" Like, I don't know what you talk about with ChatGPT, um, but, uh, like it's not quite that level, but it's a move in that direction and a trend we can expect to kinda continue getting that in-inference onto the device.

245. 49:01

     Um, so And, uh, there was also a recent hardware hack, um, on, like, using, you know, getting this inference on mobile robotics platforms. Um, and so there's- there was a ton of cool applications there like, um, yeah, some stuff with like three, like, point cloud rendering for, for your, for inside your house, like a Roomba you can

246. 49:22

     control with your voice. Very cool stuff. Um, the, the constraints that appear here, uh, that you'll have to engineer around are, like, very tight hardware constraints. So, um, there, there's memory limits, both disk and, like, VRAM and RAM are, like, are extremely tight and, like, current language models you can always trade more

247. 49:47

     up to points where you're spending like ten... hundred thousand dollars on a machine, you can trade more money for smarter models. Um, and phones are down at like gigabytes, uh, low gigabytes of RAM.

248. 50:00

     Uh, like, yeah. Um, was running a language model on a single board computer, and that had like two gigabytes of shared RAM between the CPU and GPU. Not a lot of space.

249. 50:10

     Um, and the like real, uh, deep limit is power. Um, it's, uh-

250. 50:19

     Uh, or the-- Sorry, there's a limit on power, which is like an A100, uh, which you might use for inference, draws three hundred watts of power, and something like the single board computer was using the Jetson Nano, that's ten watts of power.

251. 50:30

     So a factor of thirty. Not gonna make that up anytime soon. Um, and underneath both of these is the problem of heat dissipation. Um, there's-- that's like a really, like, tough thing to deal with when you are in these like small environments, um, and like prevents them from just being like, "Oh, I'll just like make a chip

252. 50:49

     where you can actually move like nine petabytes a second," um, or like across an inch, and it's like, uh, like you just do some like back of the envelope math, and it's like that's gonna like egress so much heat, the thing's gonna catch on fire.

253. 51:04

     Um, so, um, yeah. This is-- We're talking about some hardcore engineering stuff here. Um...

254. 51:13

     All right. So the like mobile environments, uh, may be like further out in the future to get, uh, large capabilities onto them, but, um, not impossible. What about, um, what about other consumer hardware?

255. 51:25

     Desktops, um, which are a place where you could have video games with actual artificial intelligence in them, uh, operating system level assistance, na-native apps with these like kinds of features that we're starting to see in, um, in browser apps.

256. 51:40

     Uh, so you still ru- Like you run into even more heterogeneous hardware, and that's gonna give you different constraints depending on the system that you're on, and that is gonna require like really heterogeneous software to meet those constraints.

257. 51:53

     Like, you probably can't assume that everybody has an NVIDIA thirty series or later GPU, even though it would make your life a lot easier, and you probably can't assume that you can use up all the RAM on that, uh, uh, uh, you know, on that chip, even if it would make your life easier.

258. 52:09

     Um, I think the long term, we might be able to expect ecosystems to adjust around the requirements of these workloads a bit. So like kind of, uh, like make it, uh, like make cleaner interfaces for using these things, so you don't have to write fifteen different versions, um, or write a make file that looks like Llama.cpps.

259. 52:27

     Don't look at it. Um, uh, very scary. Uh, there's kind of a sweet spot actually in what look like next generation of video game consoles, 'cause you have, you've total authority to just use up as much of the system as you want.

260. 52:40

     People pay lots of money for them. They often build custom silicon based on what developers want. So that could be, if you're thinking about what you wanna be doing in like five years, seven years in this field, consider that as a possibility.

261. 52:54

     Lots of people would love to have, um, a real like human-like intelligence in the, um, uh, in the things they're shooting in their first-person shooter. You know? Like that, that I think that would make a lot of money.

262. 53:08

     Yeah.

263. 53:08

     Uh, when it comes to building, especially for mobile hardware with those constraints you're talking about-

264. 53:12

     Yeah

265. 53:13

     ... can you say a little bit about quantization and what that looks like?

266. 53:17

     Yeah. So the question was for mobile hardware, like what are solutions and specifically quantization. So, um, when one of the key constraints is memory, like just trying to make the size of the model smaller and the size of the computation smaller is helpful.

267. 53:34

     So the, like, people are pushing to try and take the parameters of language models down from being two bytes to one byte to half a byte to like a single bit.

268. 53:46

     Um, and I think people are kinda stalling out at the like half byte level. Um, and often to actually recognize those gains, you need to like write assembler and stuff.

269. 54:00

     It's like it can get pretty gnarly. Um, so that's often only like highly resourced teams working for a long time that they can actually see those benefits. Um, the other thing that people talk about a lot, uh, for like making models work on smaller devices is sparsity.

270. 54:15

     Um, and so sparsity means like, oh, there's this giant weight matrix, maybe most of them are close to zero, and maybe we can just like get rid of those.

271. 54:24

     Like, if we were gonna go to one bit, there's zero or one, like why not? There's zeros, and then zero is like a very easy number to work with.

272. 54:31

     Like you-- the number that comes out is multiply, it's zero. Add, you just keep the number. So it's-- you don't need like a full logic circuit to handle it.

273. 54:39

     Um, so there are some things that make use of sparsity. The problem is that the type of sparsity that neural networks need is called unstructured sparsity. You have just have zeros kind of like scattered around your matrix multiply, and the all the like existing easy to use has like, you know, Python API, uh, stuff is, um, uh,

274. 55:01

     is in structured sparsity. And so you get gains there, and it's like you might need, yeah, a lot like, yeah, hand-tuned CUDA kernels or yeah, to like actually take use- make use of unstructured sparsity.

275. 55:13

     So that's something y- if there's a ton of pressure, we could see those developments in five years or so, but, um, we haven't seen-- People have been thinking about that for almost, for like seven years, and it's like not made a ton of progress.

276. 55:25

     But yeah, helps definitely, has made it easier. And like Google has been able to fit decent amount of m- language modeling capabilities on a, uh, on a mobile device, um, using distillation and quantization and probably more secrets they won't share.

277. 55:42

     I had a dumb question. What's the size of these models in terms of memory? Like if you have-

278. 55:48

     Yeah

279. 55:49

     ... suppose you have to keep the whole model in memory-

280. 55:51

     Mm-hmm

281. 55:52

     ... for instance.

282. 55:52

     Yeah. Um, so the-- if somebody tells you a number like fifty B, you know, like Llama s- Llama-- Anthropic fifty-two B, Llama seventy B, that's billions of parameters.

283. 56:07

     And then the question is like how-- what, what-- how big is a parameter? Like, how many bytes? And the like-- They're trained or where they, the way, the way they come out of the factory is two bytes per parameter.

284. 56:18

     So take the number that somebody gives you, multiply it by two, and then the B is giga. So like a small Llama model is like fourteen gigabytes, seven B times two, fourteen gigabytes.

285. 56:32

     So not gonna fit that in phone RAM. Uh, and that does make, yeah, doing this, uh, a lot harder. Um, you can do thing-- You can like try and do stuff with paging, like put stuff on the disk, bring it, bring it into RAM, then like execute with it.

286. 56:46

     Um, but that like slows things down a ton. Um, so in general... Yeah?

287. 56:52

     Four bytes per parameter?

288. 56:53

     Hmm?

289. 56:54

     Four bytes per parameter?

290. 56:55

     So y- people used to train in float thirty-two and release models in float thirty-two. Maybe-

291. 57:00

     Yeah

292. 57:00

     ... I thought the Llama models were released in float sixteen. No?

293. 57:03

     It's only float sixteen usually.

294. 57:05

     Yeah, a lot of the, like a lot of people would train in this new like Google Brain float, uh, thing, and then they, they're doing that because they want to be able to use float sixteen and so have two bytes per parameter.

295. 57:16

     But like definitely the... So the like the default before that was four bytes per pa- per parameter. And e- before that, when people were doing scientific computing with graphics, with graphics cards, like, um, people at the national labs, the default was like four bytes per parameter.

296. 57:32

     Like, um, or sorry, eight bytes, sixty-four bits, um, because they really needed that like high precision and high range. Um, and yeah, but now the trend has been to push them lower, and many model releases are now like already two bytes, sixteen bits.

297. 57:49

     Yeah. Um, and now Georgi Gerganov is like immediately converting them down to four bits, uh, and three bits, um, which is like wild. Like, what does that even mean?

298. 58:01

     Um, like a non-power of two. It's scary. Unsettling. Um, okay. So, uh, with-- There's another execution target that gets you a lot of the benefits of desktops, which is like you have a beefy machine to run on, and it's not yours, so you don't have to pay for it.

299. 58:23

     Um, but then you get a more homogeneous execution environment, which is the browser. Um, so this is not like a web app where they like talk to a model running a service, but like there is a model inside of the browser that runs inside the browser's like runtime.

300. 58:37

     Uh, and that like the homogen- homo- homogenization environments would be very huge. Um, right now, this is kind of like awaiting some technical improvements in the world of browsers.

301. 58:47

     So there is a target in WebAssembly that you could compile your programs down to, um, and in principle run them. Uh, the support for, uh, GPUs is very gross.

302. 59:00

     Um, there is, uh, a working draft from the WWW Consortium for WebGPU, which would make it cleaner and easier to use. Um, so that would help the like ecosystem, the like stack and ecosystem around this for other kinds of web applications is developing, will ma- like maybe lead developments in using this for, uh, delivering inference.

303. 59:21

     Um, you have a new constraint distinct from the other ones, which is you now, at least as it stands right now, you would need to deliver weights over the network.

304. 59:31

     And so now it's like you're, you're, you have kind of the model size constraints that you might associate with mobile hardware, um, but only during the like first load.

305. 59:41

     Um, so, um, there are probably clever ways to get around that, like progressively delivering them, um, or, uh, like browser, uh, companies sort of agreeing to incorporate some foundation models into the actual browser runtime thems- itself.

306. 59:57

     Um, so like inside of, uh, like a, uh, like V9, uh, an update to V8 with a foundation model already built into the runtime, that would make your life a lot easier.

307. 1:00:10

     Um, so this like would, uh, yeah, browser assistance, maybe sort of like general, uh, like executing apps inside of a browser that feel more like native apps, um, that's the potential applications here.

308. 1:00:23

     But, um, still a little, um, at the edge.

309. 1:00:30

     That was a pun, I guess, at the edge.

310. 1:00:33

     Um, okay. So because... Oh, yeah. Question.

311. 1:00:38

     Uh, how many gigabytes is a small and a large model right now?

312. 1:00:42

     A small and a large model. So when I hear

313. 1:00:46

     small, large language model, first I cringe internally, and then I accept GPS system, ATM machine, whatever. Um, so a small large language model, in my mind, is something that has like kinda limited Ability to like speak and interact with.

314. 1:01:06

     And that's what you see at like the like thirteen billion to thirty billion parameter range. The like the medium size is like the seventy billion parameter range, which is like the largest open models.

315. 1:01:18

     And then like a true large language model, the ones that like make people scared about losing their jobs, are generally like mixtures of seventy to a hundred billion parameter models or maybe they are themselves two hundred billion, two eighty billion parameter models.

316. 1:01:33

     Yeah. So then for all of those, take that and multiply it by, we'll call it two, um, to get the number of gigabytes. So like half a terabyte for the, um, for like a, you know, PaLM.

317. 1:01:45

     Well, a whole terabyte for PaLM five forty B, um, which is one of the larger ones ever trained.

318. 1:01:50

     I guess the context was like if they are preloaded in the browser-

319. 1:01:54

     Mm-hmm

320. 1:01:54

     ... it would have to be a smaller one, right? It's still in the tens of gigabytes. Am I missing something?

321. 1:01:59

     Um, yeah. I... It's been a long while since I downloaded a browser to my computer, but I wanna say the, the package that you download to install a browser is in the like couple of gigabytes range, right?

322. 1:02:09

     No, I think it's like hundreds of megs.

323. 1:02:11

     Hundreds of megs?

324. 1:02:12

     Yeah.

325. 1:02:12

     Really?

326. 1:02:13

     Yeah. They actually download the whole thing in the installer.

327. 1:02:16

     Uh, yeah. Oh. Uh, no. Wait, so you download an installer and then you have to download... Anyway. Yeah, so if, if people want in- to install stuff that's only a few hundred megabytes, then that's a non-starter.

328. 1:02:29

     Um, I guess I expect a Linux distro image to be in the like couple of gigabytes, like if I'm playing around with containers. So that's, um, uh, that's like another anchor point.

329. 1:02:41

     Um, and also if for those things we're probably talking like two, four, eight years before that kind of like standardization effort agreement like happens. And we can hope that internet speeds will increase in that time to match the increasing needs, uh, uh, of the internet.

330. 1:02:59

     Um, but yeah. The, that's... It's a pretty tight constraint and like probably looks a lot more like mobile stuff for a very long time.

331. 1:03:07

     Um, yeah.

332. 1:03:08

     Uh, programming languages like Gojo or Modular, do they change any of this math?

333. 1:03:15

     No. I think like right now w- I've been kind of assuming that you're doing stuff relatively efficiently. And to be honest, like PyTorch is like pretty good at this already.

334. 1:03:24

     Like, um, the fact that the application layer is written in Python isn't the problem. Um, but yeah. Good question.

335. 1:03:33

     I have a question. I was curious, what's the cost of inference like?

336. 1:03:36

     Yeah.

337. 1:03:37

     Inference on CPU versus inference on GPU. Uh, is it like just the latency or?

338. 1:03:43

     Yeah. Um, I think we'll come to that in, uh... Once, like wanted to talk about, um, after we talk about running AI over network, talk about the actual inference workloads.

339. 1:03:53

     Um, so we'll definitely get to talking about that. I don't think... I'm not gonna have a price number to give to you. Um, but you have to take whatever tokens per second you can get and then like however much you're spending on GPUs, um, and then convert that into a dollars per token.

340. 1:04:10

     Um, and that's gonna give you something you can compare to the like model providers. Um, and until you put some decent optimization into it, you aren't gonna match them.

341. 1:04:20

     Um, you did ask about CPU inference. That is rapidly evolving. I think there are cases where you can kind of like compete in price there, but um, yeah. We'll cover that more later.

342. 1:04:32

     Uh, it seems like... Well, let's put a pin in those two things, and we'll come back to them after we talk about the, like... Really the thing that almost everybody's gonna do like immediately after they leave is gonna be run AI somewhere in a data center.

343. 1:04:45

     Um, but those are important questions long term. Okay. So, uh, like, uh, the running stuff on end user devices has a lot of reasons why it's not so great right now.

344. 1:04:57

     So what, what, what do you get when you run AI in a, in a, AI workloads in a data center? Um, the biggest win is simplicity. Uh, the biggest pain point is latency, as we've already discussed.

345. 1:05:08

     Um, so simplicity, like you just, you control the whole environment. It makes your life a lot easier. Yeah.

346. 1:05:13

     You said that latency is the biggest pain point. Is that really a thing for LLMs compared to like vision models? Because like anyway the tokens that you can infer per second are quite a bit slower than any of the network latencies that we talk about.

347. 1:05:28

     Yeah. So I would say that you can... Let's see. So the question is whether latency is actually a problem. So, um, if you need to do like back and forth, like you need to get something back from the OpenAI API, then possibly like call it again with some added context or like run some if statements and then

348. 1:05:48

     send it back. Now you're looking at like multiple network calls, and you could avoid all of that overhead if you were running things locally. So that's an example of the case where people would run into a latency problem.

349. 1:05:59

     But locally, you're gonna get way lower tokens per second anyway. So you're completely dominated by the time it takes to generate tokens.

350. 1:06:09

     So tokens per second is a throughput number, not a latency number, right? So it doesn't matter if your tokens per second is half-

351. 1:06:16

     So it doesn't-

352. 1:06:16

     If your tokens per second is half that of what OpenAI is getting, but you only need to generate thirty tokens, right? That then it, like the, the latency number is going to be the larger one even though your like throughput is lower.

353. 1:06:32

     Like this is definitely something that people have run, like run into when you have like highly interactive things. Um, like there definitely... So there-- To be clear, there are tons of applications in which you don't feel this pain.

354. 1:06:44

     Um, and like ChatGPT, for example, is at this point, like the latency of the response from the machine is not really the problem. Um, so yeah. It's not guaranteed to be a pain point, uh, I would say as well.

355. 1:06:57

     Um, yeah. Um, so Yeah, and I guess I'm also kind of maybe imagining situations that are closer to the computer vision case in which you need cognition.

356. 1:07:13

     Well, in the, in the computer vision case, you need rapid responses because it's like in the motor loop of a system, for example. And if we want to use language models as the cognitive component of a m-- of a moving system, then they would need latencies like that, um, like in the tens of milliseconds or something.

357. 1:07:28

     Yeah. Um, and, uh, you are never going to be able to achieve that over a network.

358. 1:07:36

     But yeah, great question. Um, all right, so inference-as-a-service providers, this, it makes it re-- super easy to get started. It's what, you know, when you're using OpenAI, they are inference-as-a-service provider.

359. 1:07:46

     Um, also all the proprietary models basically live here. There's not some, like, way that they would ship you the model and you could run it in its proprietary license, um, that doesn't exist yet.

360. 1:07:56

     Um, open models are also available. So like, if you want, if you want to bet on the, the like open ecosystem, uh, you can use a service like Replicate, um, that will, like they'll run, uh, open models for you.

361. 1:08:10

     It's, uh, generally like easy to get started. It's not that much more expensive than running it yourself in a lot of cases. Um, but you have limited control of the model.

362. 1:08:20

     Um, for proprietary models, we already talked about how you would have less control kind of inherently. Um, for even for people who are providing open models since they don't have IP they wanna protect, in order for them to like serve it cheaply to you, they need to have like, and to have like an economic win that they

363. 1:08:37

     can like pass on to you, um, and like keep a little bit for themselves. They need to do something like amortize costs across many users of models, many more than, than you have.

364. 1:08:47

     And that requires some amount of homogeneity of usage. Um, and it's like right now it's proven to be like pretty hard to give people control while also giving them homogeneity of usage.

365. 1:08:56

     Um, for something like AWS, they came up with really smart ways to cache pieces of containers so that the fact that everybody's using kind of the same software allows them to amortize while also giving customization.

366. 1:09:09

     Um, but people have not figured out a similar trick for, um, for language models or image generation models yet. Um, so y- you don't have as much control as you would have yourself.

367. 1:09:23

     Um, so new constraints arise. So there are things like API rate limits. Um, and now this is sort of like a cost management game. You look at this as like, um, rather than like paying up front, uh, for compute that you have, and then you think about maximizing the use of that compute, you think the other direction.

368. 1:09:43

     You try to minimize your use of compute while fitting the rest of your constraints. So it's a very different feeling if, you know, if you ever switch between having your own compute and switching to cloud.

369. 1:09:52

     It's the same idea. Um, yeah. Uh, so,

370. 1:10:01

     right, I-- so you could do that inference yourself, so rather than having somebody else do it for you. Um, and this works pretty well and is getting easier every day.

371. 1:10:11

     So cloud, like this is like running stuff on a public cloud, um, is like one of the most popular choices for how to run ML workloads. Um, and for like SaaS in general, um, there's some specialist cloud providers in this space, um, like Lambda Labs, that can be like very competitive.

372. 1:10:30

     They're like often cheaper than the, um, than the like big three. Um, and it's a nice balance of control with like complexity and ha-- like which things you actually care about having to deal with versus not.

373. 1:10:43

     Um, it can get expensive over time. It's definitely like, you know, more expensive than, um, uh, like over a long period of time than if you bought the stuff yourself.

374. 1:10:52

     Uh, GPUs sometimes feel like second-class citizens in especially a lot of the like big public clouds. Um,

375. 1:10:59

     uh, Google Cloud's a bit of a distinction there in that you can just a- add GPUs to any instance, which is kind of nice, but in other, other, uh, public clouds that's not really the case.

376. 1:11:10

     Uh, and again, this is like, uh, a cost management problem. And, um, the-- one of the popular ways to solve the cloud costs is to just agree to a large deal up front, and now you're starting to get, uh, some of the illiquidity associated with actually build-- buying hardware.

377. 1:11:27

     Um, and you start to get some of the like vendor lock-in, um, that you would also associate with that. Um, so the, you have the opportunity to kind of like trade those things off, but, um, they are your constraints to work with.

378. 1:11:40

     Um, did wanna call out that there are some serverless approaches which gives you some of the like usage ba-- like really tightly usage-based pricing associated with inference-as-a-service providers, um, but also the like control associated with like, you know, have, uh, renting servers in the cloud.

379. 1:12:00

     And by this, by serverless, I mean anything with like scale to zero semantics and pricing, um, uh, that doesn't involve you having to like literally manage servers. So like thinking about the operating system, uh, for example.

380. 1:12:13

     Um, and that offers high availability. Um, this is an, like a relatively new category in software in general, and especially in machine learning. There's a couple of players here.

381. 1:12:22

     Um, Modal, uh, Labs is one that I like quite a bit 'cause it doesn't just do the ML stuff, though it is very good at it. Um, Replicate, which also does inference as a service, will do this.

382. 1:12:33

     Uh, Hugging Face Spaces recently changed their endpoints to scale to zero. Um, and there's also, yeah, Banana.dev and, and others. Um, the good thing is that it's like easier to get started, especially if you're not like a cloud ops person, um, and very inexpensive at low traffic.

383. 1:12:50

     Like you only have to pay when you have traffic. And if you're like running a small-- If you're running a demo that only needs to be up when you're showing it to, um, investors or if you are working on a tiny feature at a large organization, then you might have very low traffic patterns. [plane flying]

384. 1:13:08

     Oh my. It's, uh, it's Fleet Week, I think, so that might be the Blue Angels. Um, yeah. Um-

385. 1:13:17

     Scale to zero means that you, um, when there are no requests, you are not... When your requests go to zero, the amount of resources that you are using and being charged for also goes to zero.

386. 1:13:32

     Yes. Um, so yeah, for a while, Hugging Face, uh, end- spaces, endpoints, they changed the... There's inference endpoints, and yeah. They, uh, for a while it was like it could scale down to one, and it would auto-scale.

387. 1:13:48

     So there's like having a cloud server with auto-scaling built in, and then there's that thing, but then it also scales to zero, and you don't have to think about server management.

388. 1:13:57

     And that, that like is the combination that it's the original like AWS definition of serverless that has kind of fallen. Not everyone goes by the [REDACTED:age] ways. Um,

389. 1:14:08

     yeah. So inexpensive at low traffic. When nobody's calling your API, um, you don't pay for anything. If you like come up with a feature, it doesn't work, then doesn't matter.

390. 1:14:17

     Um, the bad news is that you kind of generally lose like tight control over auto-scaling behavior that you could have if you were like, you know, if you have a, you know, Kubernetes team to work with, they can very tightly set it up so that the auto-scaling is- delivers exactly the throughput and latencies P99s that you promised.

391. 1:14:40

     Um, and you kind of give all over some of that control to these serverless providers who are themselves probably running Kubernetes, but for a lot of people. Um, and then the re- the thing that has kind of prevented this from being as successful as maybe serverless architectures in many other places is latency.

392. 1:15:00

     Um, so when your... It shows up as, uh, kind of P99 latency, so the 99th percentile of requests that hit a point when you need to do auto-scaling. Um, the...

393. 1:15:14

     You need to get the weights of the model you're using into, uh, not just off of disk and into RAM, but then from there into the RAM of the accelerator, and that takes, uh, like that can take a very long amount of time.

394. 1:15:31

     And so you're looking at like thirty-second, one-minute, three-minute cold boots in some cases, 'cause you are moving, uh, half a terabyte of data around. Um, and so that's a place where people could maybe come up with these like clever ways to, to cache and share.

395. 1:15:47

     Um, but yeah, it's the memory constraint that you hit in other domains showing up like in disguise as latency. Um, and, uh, yeah. So the... Like you still are probably gonna be thinking of this in terms of like cost management and, and cost reduction as opposed to like resource maximization.

396. 1:16:06

     Um, yeah.

397. 1:16:07

     Uh, yeah. So like that's not necessarily always true, like with, with large models-

398. 1:16:16

     Yeah

399. 1:16:16

     ... things like Cloudflare Workers, um,

400. 1:16:24

     Yeah. So, uh, the point was about Cloudflare Workers. So I did see that Cloudfl- I didn't include them on the slide, but Cloudflare actually recently released these like GPU workers, which is their entry into this, and I haven't had time to play with it, so I don't know that much about it.

401. 1:16:37

     Um, I think if I need to go from not consuming any of your resources to having a terabyte of my own personal bytes like in the VRAM of a GPU, I find it hard to believe that they don't have a latency problem there.

402. 1:16:57

     Like, so that was-- But so I'd be cur- I'm curious what the s- what you know about the solution. Yeah.

403. 1:17:01

     I work there. [laughs]

404. 1:17:01

     Yeah, yeah, yeah.

405. 1:17:03

     Um, with, with these new-

406. 1:17:04

     Yeah. That's interesting. I'd, yeah, I'd love to hear about that. That's been my experience with the other serverless GPU providers. So I'd love to hear more about the Cloudflare Workers and, um, yeah, if that goes away, then serverless becomes a much more competitive way of delivering inference.

407. 1:17:20

     So yeah. Um, so, uh, I maintain a page for Full Stack Deep Learning that has information about, uh, like cloud GPUs and serverless providers' pricing and, um, uh, and what compute they provide.

408. 1:17:34

     Um, so you can check that out from the slides later if you're interested. Um, all right. And then last, uh, let's talk, um, like actually what if you actually physically owned the computers that the inference ran on?

409. 1:17:51

     Like, uh, you can do that. Uh, and rather than having to like actually, you know, construct a building which onl- y- which maybe the largest, uh, enterprises could go, uh, go about, uh, using a co-location facility isn't so bad.

410. 1:18:07

     Um, and there's more reason to do this than for other kinds of workloads. And in particular, there's actually room to beat a lot of the major public clouds, which is why there's competitive clouds like alternatives in this space.

411. 1:18:21

     Uh, a lot of data centers that have been like around or which-- w- that were designed before twenty twenty-one or so are configured for like disk and network-heavy workloads rather than power-heavy workloads.

412. 1:18:33

     So even if you can, uh, get a hold of like, uh, thirty thousand A100s, you can't just necessarily put them in the same US East data center that used to run, uh, u- that was designed for like, uh, running databases.

413. 1:18:48

     Um, it's-- so it's capital intensive but ends up being cheaper in the long run. You have total control if you, if you need it, uh, which is awesome. Um, but it's very hard, very rare skill set, um, because it like kinda crosses this like the ML stuff and the, the, the hardware stuff.

414. 1:19:06

     Um, and all of these people can go and work for OpenAI for like a million and a half a year. So, uh, uh, good luck holding on to them.

415. 1:19:13

     Um, yeah. Uh, and the biggest constraint that shows up is illiquidity. So you're gonna make a big bet on what this looks like. For example, that inference is not gonna move onto CPU or not gonna move onto custom silicon that behaves very differently from graphics cards.

416. 1:19:28

     Um, there's a great talk from Nitesh Agrawal of Lambda Labs about this that goes into kinda detail. I think it's, it's only like a year [REDACTED:age] if I, if I remember this talk right.

417. 1:19:38

     Um, and of course, he makes it sound very hard because he wants you to use their cloud, um, or to pay them to, like, help you build, uh, your co-location...

418. 1:19:49

     uh, um, uh, yeah, help you actually build it. Um, but it is a detailed explanation of everything involved and, uh, you know, there's not very many of those out there.

419. 1:20:01

     Um, all right. We're at half time. Uh, so I plan to take a break when I finished part one, which goes the rest of self-serve inference, which will... I'll say is another fifteen minutes.

420. 1:20:12

     So let's do that, and we'll leave, uh, an hour for part two after a little break.

421. 1:20:19

     Um, all right. So let's... uh, we actually haven't talked in great detail about, uh, like, you know, why, why are we using GPUs in the first place? Like, what, what is actually going on here?

422. 1:20:33

     When we run this, uh, this, like, tensor-to-tensor map with neural networks, um, like, what, what, what actually does that workload turn into? We have two tasks. We need to load numbers from memory, and then we need to do math on those numbers.

423. 1:20:49

     Those are our, are like two basic tasks. Um, and that is the reason why we ha- why we end up using graphics processing units, um, because memory is slow and math is fast.

424. 1:21:03

     And in most... in the, in the Transformer architecture in particular, but in many, like, sort of most neural network architectures you might write down, um, you only need a given number from the weights on- like one time per input.

425. 1:21:21

     So that means you need to do a memory read, like, uh, of this... of like a couple of bytes for a particular parameter to use it in a single floating point operation, and the memory read is gonna be very slow, and the floating point operation is gonna be basically instant.

426. 1:21:37

     Um, so in order to do this economically, you need to do a lot of math for each read from memory. Uh, you need to, like, load, uh, you know, load the weight, um, out.

427. 1:21:49

     And in particular, we're talking here about, like, getting out of the VRAM and into the place where the, you know, into the, like, um, the... What is it called?

428. 1:21:59

     Well, yeah, it's basically like an L1 cache, like closer to the actual comput- uh, uh, computation. Um, and so you wanna do that and then, like, use it multiple times, you know, and, uh, that means you want to run on multiple inputs at once, and that is memory intensive, single, uh, instruction, multiple data, parallel linear algebra, the

429. 1:22:22

     same thing you need for graphics workloads. So the, uh, like, graphics cards have turned out to be, like, pretty good at solving this problem.

430. 1:22:31

     Um, and the, the numbers there in the corner are, um, uh, show-- like demonstrate this, uh, like general fact of like memory is slow, logic or math is fast.

431. 1:22:46

     The... You can do three hundred and twelve teraflops per second, uh, in... for two byte numbers, uh, two byte floating point numbers in a tensor core in an A100.

432. 1:22:57

     Um, and you only get one and a half terabytes a second of memory bandwidth. Um, and when you're using like, you know, optimized existing CUDA kernels, these two things are like multiplexed.

433. 1:23:11

     So like you load a weight, well, you start doing math on it, and then the next weight gets loaded like, you know, uh, concurrently. Um, but you do still have this like mismatch in the bandwidths that means that you want to be able to, like, amortize a memory load across as many computations as possible.

434. 1:23:29

     And like, in principle, this could be flipped around and we would have like very, you know, things would look very different. Um, uh, so you can get like very, very large throughput gains by amortizing memory reads, where basically if you are operating on a very small number of tokens, uh, then you'll see that as you add...

435. 1:23:49

     Like if you're running th-this workload yourself, as you add more tokens, you would think like you should expect like a s- a slight increase in the amount of time that it takes, and you'll see like basically a flat curve for a very long time.

436. 1:24:01

     Um, and then once you hit the point where the, um... Yeah, so if you look batch size at which you'll see that flip. So for an A100, it's about two hundred, uh, el- uh, elements in the batch.

437. 1:24:21

     Um, and you'll see for an H100, you'll see that that ratio's... These numbers both go up, but the ratio becomes more extreme. Um, so you, uh, there's a great blog post from Carol Chen on like inference arithmetic that both goes through in greater detail and then like matches that onto some like actual experimental results and is able

438. 1:24:43

     to like track where did each, you know, microsecond basically of, uh, of inference time come from. Um, so the like key takeaway from this is that if you want to get large throughput in like an inference system that you're running yourself, you're gonna need batching, you're gonna need to like collect up multiple inputs, uh, from multiple users

439. 1:25:04

     and operate on them at the same time. Uh, so this is, uh, it's challenging to achieve the same thing in an end user device. Um, you are... Like, if you're only working for one person, then they might not make ten requests, uh, you know, quickly enough for you to fill up a batch.

440. 1:25:22

     Um, and that actually might kind of tilt things in the direction of, uh, of CPUs or of, uh, you know, uh, different, uh, different architecture with different, um, uh- You know, memory bandwidth versus tensor core, uh, versus, uh, math bandwidth, um, uh, trade-off.

441. 1:25:43

     Um, the... One of the useful things that came out of this, I guess, as a restatement of kind of what I just said, when you have a, like, low load API, um, you will end up with smaller batch sizes because maybe you have, you have to deliver with a certain latency, so you can't just wait forever.

442. 1:26:00

     Um, and so you will make different decisions about compute memory trade-offs. For example, like caching past computations of, uh, keys and values is very common when you're doing batch inference, but it's actually not necessarily the right choice when you are already memory bound.

443. 1:26:16

     Trading off memory for compute, um, isn't a good idea. Um, if you're like at... For larger batch sizes, um, you- if you're doing something that's an API where you're, like, serving requests directly, then you would want to, like, roughly balance being, uh, flop bound and memory bound so that you can get quick latency of responses even though

444. 1:26:38

     your throughput is less than it would be otherwise. Um, whereas if you're doing like a batch job, like an overnight type job, um, or that is like typical of data science or, or big data workloads, then you would just go for the largest batch that you can, and that's what people do during training.

445. 1:26:54

     They go for the absolute largest batch that they can because there's no latency requirement. Uh, there's only throughput.

446. 1:27:00

     I'm assuming all this still applies to like other accelerators like TPUs and like CuCNAs or these kind of chips?

447. 1:27:07

     Uh, yeah, yeah. So TPUs end up... I, I, like, looked around and they... I, I've never personally used them for anything serious, so I wasn't able to, like, directly map on, like pull out the equivalent of a tensor core, um, like flop bandwidth and the, and V- and VRAM to L1 latency.

448. 1:27:25

     Um, but the results and benchmarks that I've seen is that they're like thirty percent better. Um, like they, they have a di- they do have a slightly different choice of trade-off, um, and they...

449. 1:27:36

     And, and it's better for neural network workloads than graphics workloads, and that gets what, from what I have seen, like thirty percent improvement, but not like a ten to fifty X improvement, which is what you would really like to see if you're making as drastic of a, a decision as, like, going over to a completely different accelerator

450. 1:27:55

     with a completely different software stack. Um, yeah, I guess... Yeah, I, I got into a discussion with, uh, one of the people on the TPU team about, like, the hardware lottery, and he, his, he was like, "The GPU is already, like, is just an excellent machine.

451. 1:28:12

     Like, it does this workload really, really well." Um, s- and I think that is bo-borne out in the numbers. Yeah.

452. 1:28:18

     From what I heard, the main advantage of TPUs is, uh, operational efficiency and power efficiency, not like-

453. 1:28:26

     Mm

454. 1:28:26

     ... throughput or performance.

455. 1:28:27

     Yeah. So, um, for the folks listening online, repeating the point, uh, the, one of the benefits of TPUs is power. I totally buy that. Um, yeah. Uh, I think, yeah, the numbers that tend to get reported in things like Google's Pathways paper and Palm papers are open and, like, what is known about OpenAI, it's like, it's all

456. 1:28:46

     about, like, flop utilization and, and total flops and stuff like that and not things like power that do matter. Um, I'd say, like, people have not gone rushing to try and get a hold of them, which feels, uh, uh, like a strong signal, um, similar with, like, other types of custom silicon like Cerebras' um, uh, chip.

457. 1:29:08

     Uh, but, you know, who knows? I think this is getting back to a question that got asked earlier. Um, I believe the, like, having custom chips works really well when workloads stay very fixed in, like, kind of precise detail.

458. 1:29:22

     Not just this like, "Oh, we need to load from memory, and then we need to do it," but like, "No, we need to do this shaped thing, like with this many..."

459. 1:29:28

     Like, that works super well, and you can see that in blockchain mining where there is, like, for a, for many chains for a long time, ASICs were the, like, only profitable way to mine, um, or like the most profitable way, way to mine, and that's the workload is, like, unchangeable except by, like, distributed consensus.

460. 1:29:48

     Um, and that w- allows you to, like, very tightly target a specific workload. Whereas, like, neural network architectures actually kind of change reasonably often and the like, you know, even the difference, the difference between a seven billion parameter model, a thirteen billion parameter model, a hundred and seventy billion parameter model, that's like a bigger difference than you

461. 1:30:07

     would see between workloads for, um, like the same blockchain over time. Um, and so that's, that, like, difference in tech- like, that technical difference is I think a big reason why we haven't seen, uh, much uptake of custom, custom-

462. 1:30:24

     Yeah, I think th- like they are one of the, um, they're one of the providers for, um, like they're one of the startups working on this. P- They come up most frequently.

463. 1:30:38

     Um, uh, I don't... It's still like, uh, fairly experimental, I think. Like, it's not, it's not the, um... It's not like generally available in a public cloud or something.

464. 1:30:51

     So, um, yeah, I don't have much to say about it, I guess. Yeah.

465. 1:30:56

     I think the main thing you can see there is their, the, the memory that they have-

466. 1:31:02

     Yeah

467. 1:31:02

     ... is forty gigabytes. Compare that with an A100 which has eighty-

468. 1:31:06

     Yeah

469. 1:31:07

     ... and for most of LLMs, memory is your biggest problem. So-

470. 1:31:10

     Only LLM they trained was like JS, that Arabic thing, right? But-

471. 1:31:13

     Mm-hmm

472. 1:31:13

     ... they, they trained this Arabic LLM on this one, but you haven't worked with GPUs or-

473. 1:31:19

     Yeah, yeah. So Cerebras has done some large model training, so it's definitely, like, possible. Um, they, um, the, the... They have this very fast memory bandwidth, I think is like maybe the, the big one, twenty petabytes memory bandwidth.

474. 1:31:32

     Now, like the, the point that was raised about the bottleneck being the memory capacity is very, is, uh, is well taken. Um, you can network multiple s- multiple chips together, and then you have more storage.

475. 1:31:45

     But, um- Yeah. And so, like, bal- better balancing of, of memory bandwidth and math bandwidth would maybe get you more effec... Like, more efficacy at lower batch sizes. Um, but you can't exactly put that chip in a phone, um, so that-- Yeah.

476. 1:32:03

     Um, but definitely, uh, uh, like, people are really converging on transformer architectures, so that could mean that this, um, custom silicon works better, um, in five years. Um, yeah.

477. 1:32:16

     I think I... Yeah. The way that you, like... The way that you solve this problem is if you have multiple GPUs, these, the, like, memory per second and FLOPs per second s- like, scale at the same amount, right?

478. 1:32:27

     You have... Now you have two GPUs loading. You have two GPUs doing operations. They both scale linearly. If you're really clever, you can, like, actually get that linear scaling in practice.

479. 1:32:36

     Um, the GPU memory goes up as well. Um, and so you can serve larger batches and eventually hit this ratio as staying the same, so you can eventually hit that point.

480. 1:32:46

     Um, and that's what... And then, like, once you hit the point where you're actually, uh, you know, FLOPs bound, um, you can just, like, crank it, you know, just crank it up.

481. 1:32:56

     Uh, just get really big, uh, uh, really big batches, speed up, uh, speed up the computation, and that is what people do during... Are, are, like, used to do, used to doing during training.

482. 1:33:07

     Um, and so you'll very commonly see people talking about this way of making inference efficient. Um, so if you are running a, um, like a web service, this is gonna give you your sort of like, uh, pod size, roughly.

483. 1:33:20

     Like, um, like a pod in Kubernetes is, like, a bunch of services that end up on a single physical machine. So this is your, like, unit, uh, um, uh, is determined by, like, how many GPUs do you need to hit the, like, batch size, um, for efficiency implied by the, like, uh, this FLOPs memory per second.

484. 1:33:40

     And you're checking to make sure this is also true for your architecture. Um, yeah.

485. 1:33:46

     Um, all that's, like, kind of theoretical stuff or, or sort of like trying to be relatively first principles. Uh, when you... When it comes time to actually check whether you're doing things correctly, make sure to use, like, a profiler or a tracer and actually, like, look at these things.

486. 1:34:00

     Um, even, like, people who are pretty good at this will, like, miss stuff that shows up on a, on a tracer. Um, like the vLLM, uh, implementation was launching a CUDA kernel for, like, every individual element of a batch.

487. 1:34:15

     And, like, a s- a tiny change ch- made it switch to, like, one CUDA kernel launch, and that saved them, like, uh, uh, increased their throughput by thirty, thirty-three percent.

488. 1:34:27

     Um, and it's something that shows up as, like, a t- like way more CUDA kernel launches, which would show up as, like, I guess, yeah, if I had the interactive version of this, would show up as this, like, huge thing of red lines.

489. 1:34:40

     Um, and also just the, like, general utilization shows up as these big, uh, like, patches of gray here, which shows you where you're, like, not actually using your GPU at all.

490. 1:34:49

     Um, and, uh, I found, like, looking directly at these, um, at these traces, not just at statistical profiles, helps, like, catch those, uh, those kinds of, um, easy to fix eighty/twenty kind of bugs.

491. 1:35:04

     Um, you can also profile and trace memory, uh, and that's gonna be, like, given that memory is this key constraint that's likely to be at least as useful. Um, and these two are examples from PyTorch.

492. 1:35:16

     Um, these, uh, this, the c- compute trace and the, um, the memory trace both from PyTorch. You might not be using PyTorch, um, so you might have to fall back on generic, like, GPU profiling or CPU profiling or memory profiling and tracing tools, um, if you're using, like, a custom inference solution.

493. 1:35:39

     Um, yeah. Speaking of which, there's a bunch of specialized LLM inference libraries out there. There's a nice blog post from Hamel Hussein about using these, and this is specifically for, like, batch size one.

494. 1:35:50

     Um, and so the, the vLLM is people are, like, pretty... That seems to be getting, like, the most community excitement and community contribution. Um, but Hamel suggests some reasons why you might be into NLC and CTranslate too instead, um, for doing your, uh, Llama inference.

495. 1:36:11

     Um, so actually, so all that was about, like, just thinking in terms of, like, an individual workload.

496. 1:36:19

     Um, setting up an entire inference service, um, which is now something that maybe s- like, somebody in the company sets up an inference service or a inference platform, and then, uh, people can, you know, submit workloads to it.

497. 1:36:32

     That can be fairly challenging. Um, containerization for GPU accelerated workloads is less painful now than it used to be. Um, like NVIDIA Docker actually works in a way that it did not, um, like, early on.

498. 1:36:47

     Um, but, uh, containerization is, like, fundamentally, uh, you know, more dubious for these, uh, these workloads than a lot of other ones. So one is that the, like, application layer of the container is probably where the weights are gonna live.

499. 1:37:04

     Like, it doesn't live in the operating system yet. Um, it's not built into the Docker engine. So it's like that's up there, and that's, like, maybe half a terabyte in, in, in bad situations.

500. 1:37:15

     Um, so now your container images are really large, and that's, um, that can be pretty unpleasant. Um, you could try and do the things that people do with databases and, like, move it into remote storage.

501. 1:37:26

     But, um, yeah, then the container is not as, like, u-unitary. Um, and then the, like, the worst problem maybe is that the CUDA driver- Changes the choices that you need to make kind of at the, like, application level of, like, we were just talking about actually going one level down, but, like, which GPU it's gonna run on

502. 1:37:48

     changes the point at which you switch from being memory bound to FLOPs bound because that has a different, uh, like, memory bandwidth to math bandwidth ratio. Um, so things at the level of the NVIDIA-- of the GPU are entangled with things at the application layer, um, like choices of, like, batching strategy.

503. 1:38:07

     Um, so that's... And a similar thing can be said for the CUDA driver. I was, like, naive about containers, and I was like, "Why is the, like, CUDA driver showing up as 12.0, which is what I have installed in my host operating system, when I specifically downloaded the CUDA 11.2 container?"

504. 1:38:25

     And it's like containers can only virt- only virtualize so much. Um, and so that also changes, like, which kernels are available by default. Um, uh, I guess, well, kernels would be a layer above.

505. 1:38:39

     So there are features of CUDA, of CUDA drivers that change the, like, you know, how Hugging Face Transformers or PyTorch will work, um, and so that will, and that will also change decisions that are made at the application layer.

506. 1:38:51

     So when you're in these, like, super per-- like, this is clearly a performance limited regime that we're talking about here. Like, the inference is, like, extremely expensive, um, and we're, like, trying to maximize performance really aggressively, and that, like, s-starts to reveal the limitations of virtualization and containerization.

507. 1:39:09

     Um, so that doesn't mean it's impossible, it just means it's hard. Um, and yeah, I don't know, if you've ever worked with, like, you know, trying to set up a heterogeneous compute Kubernetes cluster where there's, like, some have GPUs, and some have Arm, and some have, uh, like, x８６, you, like, this-- dealing with this requires a, a

508. 1:39:31

     better, um, uh, ops engineer than if you can ignore that. Um, speaking of which, this, like, application serving can be and is in fact done with the, like, industry standard for container orchestration in Kubernetes.

509. 1:39:44

     Um, I would note that now the, like, NVIDIA stuff is showing up here again and, like, all the problem, like the entangling th- And now Kubernetes is in between them.

510. 1:39:54

     So now there's, like, a lot of opportunity for crosstalk, a lot of opportunity for tears. Um, and, uh, so this is a hard mode version of that problem, which is not notorious for being easy.

511. 1:40:08

     Um, so you might choose and need to do it, uh, by hand. There was a comment on r/MLOps that was like, "Oh yeah, we loo-- uh, you know, I was looking around to see what the opinions are on these tools," and, uh, a opinion I saw in a couple places was like, if you care about, like you've

512. 1:40:27

     chosen to do serving of inference yourself and not do, like, just API calls, then, um, maybe you should make MLOps a core competency of your company, and you're gonna want to do something more like building it out of the, like, Kubernetes aff- uh, affiliated ecosystem of open source tools or other open source things, um, like Cube Ray

513. 1:40:50

     to do, um, to run Ray on Kubernetes for serving or, uh, Seldon Core to run that on, uh, again on Kubernetes as your container orchestration layer. Um,

514. 1:41:03

     but given how, like, painful that is and the fact that there are so many people who are trying to run a relatively similar workload, um, you might consider, um, like either of two kind of tiers of managed services, either the, like, white glove end-to-end kinda cloud provider approach, the oldest one, Amazon SageMaker, um, more recently Vertex AI

515. 1:41:24

     from Google. I'm sure there's an Azure version that I'm forgetting. Um, there are also from s-some startups a sort of more toolboxy approach, a less end-to-end approach, in part 'cause it's not, like, integrated all the way at the layer of the, like, actual hardware like the cloud providers are.

516. 1:41:40

     Um, so, uh, BentoCloud, uh, Seldon, and Anyscale. Anyscale, um, being a managed version of Ray. Seldon offers a managed version of SeldonCore. BentoCloud, managed version of BentoML. Um, so, uh, depending on where you want to...

517. 1:41:58

     Maybe you have a really great suite deal with GCP, and so Vertex is the right choice. Um, but yeah, I think from... I have had limited experience with these things, so I've tried to keep my life, uh, simple and happy.

518. 1:42:11

     Um, but if you're, uh, if you end up in this space, I've, I've seen some nice things and played around with Ray Serve, um, and Anyscale. Um, Ray's the tool that a lot of people use for cluster management for training, so a lot of the teams that train the foundation models use Ray.

519. 1:42:29

     Um, and so there's kind of like a natural competency for them, uh, both in the open source library and Anyscale as a, as a, um, layer around that, so maybe it'd be a good choice for, um, for inference as well.

520. 1:42:45

     All right, um, so rather than take questions, I think I'm gonna do a five-minute break in which you can ask me questions up here while we all get a little stretch, maybe a little air, um, and we'll come back for the rest of the hour in five minutes. [outro music]