A Practical Guide to Efficient AI

Talk transcript

1. 0:00

   [upbeat music] I go to a lot of AI conferences.

2. 0:15

   I go to AI research conferences. I go to, you know, more general tech conferences. And what I absolutely love about this conference is that it's really about the builders, and it's really about the techniques that we need to get AI into the hands of our customers.

3. 0:29

   And so we're all here in the AI space. We're all AI practitioners here, and we know that AI is developing at an unprecedented pace. It's pretty hard to keep up with it, right?

4. 0:40

   Every week there's a new model, a new capability, a new feature. So much to keep up with. And when we see these new models, capabilities, and features, they're often shown to us as a demo or a prototype.

5. 0:55

   And as builders and engineers here today, we all know there is a big difference between a demo and a prototype, and scaling and productionizing AI. So one of the biggest ways that we can bridge that gap, we can go from having cool, awesome demos to actually bringing that to customers, is with efficiency.

6. 1:15

   So if we have techniques for making our AI efficient, we can get closer to productization. And so that's what I wanna tell you about today. I'm gonna tell you about some practical ways you can take away today to start making your AI models more efficient.

7. 1:31

   So let me introduce myself. So I'm Shelby, and I lead an AI research team at Salesforce. My team ships AI today. So we deliver, for example, LLM solutions to our data platform at Salesforce.

8. 1:45

   The data platform is the foundation of all of Salesforce, that scale. Now, while we're delivering AI today, we're also envisioning what we'll need for tomorrow. And to do that, we've released over fifteen cutting-edge research papers in agents, LLMs, on-device AI, and more.

9. 2:05

   And we've also released over six open source repos. So my team has released these repos. I'm gonna talk about one of them, uh, today, so you'll get to see.

10. 2:13

    And, um, this is all in vein of pushing AI forward and getting the AI that we'll need for tomorrow. Now, a little bit about my personal background. I have a PhD in machine learning, so I focus on developing learning algorithms, uh, that are sample and communication efficient, and I have a bachelor's and master's in math as well.

11. 2:34

    So if you're interested in learning more about my team, my background, wanna connect on LinkedIn, feel free to scan the QR code. I'm always happy to chat with you all.

12. 2:44

    Now, what about Salesforce? This is the-- This is the AI in the Fortune 500 section. Let's talk about Salesforce and what we're doing. Salesforce has been deploying AI for ten years, everyone.

13. 2:54

    Ten years. So the AI research team was founded in twenty fourteen, and since then, Salesforce has accumulated over three hundred AI patents, over two hundred and twenty-seven AI research papers, all in the p-- all in the last ten years.

14. 3:10

    And you can see here the map of all the deployments that have taken place. Okay. And at Salesforce, trust is our number one value, so we don't just deliver-- we don't just build and deliver AI in isolation.

15. 3:22

    We build and deliver trusted AI. That is a key. So to do that, we're a part of six ethical AI councils, and we're also involved in the White House commitment for trusted AI.

16. 3:34

    So I wanna zoom in here. Past two years. That's where all the AI action has been happening, right? The past two years. Let's look at twenty twenty-two and twenty twenty-three.

17. 3:43

    What's Salesforce been up to? Well, we've been deploying a lot of LLM products, right? If you look here, you'll see, um, you'll see code gen-based products. You'll see Service GPT, Einstein GPT, Tableau GPT.

18. 3:56

    That's very similar to the rest of the tech industry, right? Like, if we zoom out, the rest of the tech industry, we're de-- we're deploying LLM products. And now for us to do it at Salesforce, efficiency is key.

19. 4:06

    Think about Salesforce scale. Think about Fortune 500 scale that we're talking here. Efficiency is key.

20. 4:13

    And we're all in the same boat here. We're all working on the same deployment environment. So let's review that a little bit. When we've got an AI model, we're mostly deploying...

21. 4:21

    You know, a lot of times we're deploying on a cloud, right? Private or public cloud. We're paying for resource consumption. We're paying for, uh, we're, you know, we're paying for GPU, CPU, disk space.

22. 4:31

    We're paying for all of that, so we've gotta keep that in mind, right? When we deploy, we're paying for that cost to serve. Or now we're seeing even on-prem solutions.

23. 4:38

    Maybe we have an on-pre-- Maybe you have an on-prem cluster. So not only, only are you paying for that, you've got restricted GPUs to work within. And more recently, and this is pretty exciting, small devices.

24. 4:50

    We're seeing LLMs being feasible on small devices. So if you guys were paying attention to the news in the past couple of weeks, you'll see-- you'll remember that Apple has announced their LLM on their newer devices.

25. 5:02

    So this is so exciting. Um, and so if we're seeing it on, uh, on iPhones and small devices like that, we can think maybe LLMs and LMMs, multimodal models, on tablets, on laptops, on edge devices.

26. 5:16

    Now, that's an even more challenging situation, right? Small devices have even worse hardware, have even more resource constraints. The point here is that when we're deploy-- when you're deploying AI models, you're deploying in these constrained environments.

27. 5:28

    It's never-- We're never in a situation where we have infinite resources. So efficiency is going to be key. So how do we make AI more efficient?

28. 5:38

    So that's what I wanna talk to you today about. I've summarized it into five dimensions, five orthogonal directions that I would love for you to consider as you're de-- as you're thinking about building your AI for customers and deploying.

29. 5:50

    The first-- And this is just scratching the surface. This is just scratching the surface, but I'm hoping these five dimensions will be easy for you to remember. The first is picking efficient architectures from the very beginning.

30. 6:02

    From the very beginning. So this includes picking small models. I'm gonna talk about that today. Um, this includes using sophisticated architectures such as mixture of experts, for example.

31. 6:13

    And if you're building your, um, architecture from scratch, includes, uh, choosing efficient attention mechanisms and so on. So there's a lot we can say there. Today, I'm just gonna touch on a little bit.

32. 6:23

    Moving on to the second one, efficient pre-training. Now, not a lot of us are doing pre-training. It's a really expensive thing to do. But if you're doing it, you know the GPU costs.

33. 6:32

    You wanna use mixed precision training, scaling methods, among other methods here. So definitely make your pre-training efficient. Now, efficient fine-tuning. This is the, this is the case mo-- a lot of us are in today, efficient fine-tuning.

34. 6:44

    It's-- You wanna pick methods that are not optimizing all of the weights, every single weight, full fine-tuning. You wanna pick, um, methods that are, that are only optimizing, you know, a subset of those weights.

35. 6:55

    So think about LoRA, QLoRA, and so on.

36. 6:59

    And fourth, the fourth dimension, efficient inference. So you've got your model, it's pre-trained, it's fine-tuned, you're ready to al-- you're almost ready to serve it. How can we do that efficiently?

37. 7:08

    We're paying for cost to serve, right? So with that, you wanna consider post-training quantization, which I'll get into today, and speculative decoding, and there's many others to cover as well.

38. 7:17

    And finally, prompting. Prompting, we gotta think about that. Prompts, uh, you know, consume memory. They also, uh, uh, directly affect latency as well. So you want your prompts to be as concise as possible.

39. 7:28

    Concise as possible. So think about template formatting and prompt compression. Now, with our limited time here today together,

40. 7:37

    I'm gonna dive into two crucial directions that you can take away with you and apply right away. The first direction is around efficient architecture selection. I wanna tell you about the power of small models.

41. 7:48

    Small models are coming back, guys. We went, we went big models. Small models are, are super efficient. We'll talk about it. Second, I wanna go into efficient inference. I wanna tell you about post-training quantization.

42. 7:59

    This is something that you could, you could actually apply at the end of the day on your model. So efficient and so quick.

43. 8:05

    So let's get started with small LLMs, the power of these small LLMs. So let's, uh, think about the past two years. As I mentioned, every week, new model, new model, new feature.

44. 8:19

    When we look at these, at these LLMs that have been released, they're mostly pretty big. They're mostly pretty big. So here are just a few. These are older models, but I just wanted to prove a point here.

45. 8:28

    If we look at the PaLM model, for example, five hundred and forty billion parameters, right? So parameters, again, every one is the number of weights in that deep neural network.

46. 8:36

    Five hundred and forty billion parameters. These other models, Bloom, Yalm, one seventy-six billion parameters, one hundred billion parameters. So those parameters have got to be stored in memory. They're all gonna be used in computation, GPU computation, CPU computation.

47. 8:51

    They're gonna take up space. Long story short, these huge models are resource hungry. They're gonna take a lot of resources to train, certainly to pre-train, to fine-tune, and to serve.

48. 9:02

    Now, in parallel, let's think over the past several months. We're seeing these smaller models emerge. And, and when I think about small LLMs, I'm thinking models that are thirteen billion parameters or less.

49. 9:16

    We're seeing these emerge, and for very good reason. They're emerging for good reason. There's so many benefits to these smaller models. So as you can imagine, with less parameters, with less weights, they consume less RAM, they consume less GPU, less CPU, less dis-disk space, and they're just faster to fine-tune.

50. 9:35

    They're super e-- they're super resource efficient. This is exactly what we're looking for today. They're also low latency. Fewer weights mean the forward pass is faster. There's just fewer weights to go through, right?

51. 9:46

    And both of those together, the resource efficiency, the low latency, makes them perfect for additional deployment options. So not only can you take these small LLMs and deploy them on the cloud, on on-prem, they can also be deployed on mobile if they're small enough.

52. 10:01

    They can be deployed on laptops for personal models. They can be deployed on edge devices. They're super nimble and super useful. So let me tell you about how... So what I wanna do today is tell you about a few small state-of-the-art LLMs to keep in mind as you're building your solutions for your customers.

53. 10:18

    So the first one I'm gonna tell you about is, uh, Phi-3. You guys may have been, um, may have heard of this one. This is a three point eight billion model.

54. 10:25

    Super, super small. And as I'm talking about small models, I showed you these five hundred and forty billion parameter model. Now we're talking about a three point eight billion parameter model.

55. 10:34

    Your first question might be, "What is the performance? Is the performance good?" So interesting. So Phi-3 is actually a pretty-- it's a very strong performing model. So as you can see here, I took this clip right from their technical report.

56. 10:47

    Feel free to check it out. As you can see here, Phi-3 is outperforming a very, very well-known seven B model, a model that's almost twice its size. So this three, three B model is pretty powerful for being so small.

57. 11:00

    And now, with that model being powerful, we're seeing even smaller models emerge, even smaller models, because even smaller models will fit on edge devices, on mobile, and so on.

58. 11:11

    So what I wanna point out to you is MobileLLM. It has less than one B. So we're-- This has three hundred and fifty million parameters. Three hundred and fifty million, not even a billion parameters, so super, super tiny.

59. 11:24

    And here's the key. After fine-tuning, after fine-tuning, it's on par with the seven B model on tasks. So this is one of the takeaways I wanna share with you, is that the power of these small models, you-- the way you use them is important.

60. 11:38

    They're, they're great. After fine-tuning, they are very competitive. That is what this is showing. And finally, I wanna bring up a model that's really interesting for function calling. So, um, this, this Octopus model is a fine-tuned model.

61. 11:51

    It's fine-tuned Gemma two B. It's fine-tuned on Android tasks. And again, a two B model, they are showing after fine-tuning, it's outperforming GPT-4, Llama 7B on these Android tasks.

62. 12:04

    So super, super promising. So definitely check out these small LLMs. They have a ton of potential.

63. 12:11

    And finally, I will go to our next topic, which is quantization. This is about inference.

64. 12:17

    So what is quantization? Quantization is actually not a new topic. Not a new topic. What's new is applying it to LLMs and LMMs. So the idea of quantization is to take a big number and to map it to a smaller number.

65. 12:31

    So what we wanna do for quantization for LLMs is we want to reduce the precision of the weights. So typically, weights in LLMs, depending on the model, is, um, can be 32-bit or 16-bit floats.

66. 12:44

    What we wanna do, what, what quantization does is reduces that 32 or 16 down to 8, down to 8, 4 bits, down to... Actually, you can specify just a smaller number of bits, reducing the precision of all those weights.

67. 12:56

    So as you can imagine, that's hugely, hugely beneficial. Massive efficiency gains. So as you can see here, if... As you can imagine, if each weight was originally 32-bit, taking up 32-bit space, now we reduce it to 4-bit, it's gonna take up a lot less space, it's gonna consume a lot less memory, and it's gonna be...

68. 13:12

    consume a lot less CPU and GPU. So as you can see here really quickly, just some models, looking at these Llama models, 7B, 13B, 70B. The original, you can see the disk space it was taking up.

69. 13:22

    After 4-bit quantization, it's taking up a fraction of the dis- the space. And now, l- what about latency? So as the resource consumption comes down, the latency improves. So as we can see here in this, in this study on large multimodal models.

70. 13:37

    So these are large LMMs. 16-bit was their original, um, encoding. Originally 16-bit. Now, if you look at the 4-bit, 4-bit quantization, you can see that the latency, measured here as time to first token, has decreased.

71. 13:51

    So lots of benefits. So reduced, uh, so reduced resource consumption, faster. Again, though, the most important question is, is the performance still there? Are we making this hu- are we making a huge trade-off by, with this?

72. 14:03

    And the good news is, no. This is, this is pretty amazing. Quantization generally has negligible effects on performance. So I wanna show you that. So look here at this chart, at this graph.

73. 14:16

    And again, we're looking at LMMs. And you can see here, we've, uh, on this particular task, this well-known vision language task, we took the LMMs and we measured performance on 16-bit, and then 18-bit quantization and 4-bit quantization.

74. 14:31

    And as you can see, essentially no movement. 4-bit quantization was essentially free. Like, we could just quantize it with 4 bits and just enjoy reduced latency, enjoy inc- increased, uh, improved lat- uh, sorry.

75. 14:44

    Enjoy reduced resource consumption, enjoy improved latency, and no effect to performance. Retained performance. However, you can take this too far. There is, there is a way to take this too far.

76. 14:54

    So as you can see, when we quantize down to 3 bits, performance did drop. So evaluating your quantized model is super important. So don't just assume 4-bit is the answer.

77. 15:04

    Definitely measure. Uh, you definitely wanna measure your quantized performance. So really quickly, so you can get started on this today, you can add... You can quantize any of your models, whether it's ML models, LLMs, LMMs, and so on.

78. 15:16

    I wanna just highlight a couple of, of frameworks that are really, um, awesome for that. So Llama CPP is one of the most well-known frameworks right now. It's gaining a lot of traction.

79. 15:25

    It has quantization from 16-bit all the way down to 1.5 bits, so pretty crazy. Um, wide adoption. So actually, you may not even need to quantize the models that you're using.

80. 15:34

    Just check Hugging Face. A lot of people are, as they're releasing their models, they're go- they're releasing the, the Llama CPP compatible quantization models too. So pretty awesome. And there's, uh, Python and Java wrappers.

81. 15:46

    Second thing I just wanna quickly mention, ONNX Runtime. This is, uh, ONNX has been around for some time. If, if you've been around since the ML days, ONNX was around in the ML days.

82. 15:55

    And so, um, they have some 8-bit quantization. And y- you know, the beauty of ONNX is that it's compatible across so many programming languages. So definitely take, uh, take a look at these, and there's a bunch of others ones you can consider too.

83. 16:09

    Now, final point here. As we mentioned with quantization, now, um, you have your quantized model. I mentioned before, it is still important to evaluate your quantized model before you deploy it.

84. 16:18

    So I want to introduce to you one of the open source repos that my team just developed. We just released this maybe, like, a w- a week ago. It's called Mobile AI Bench.

85. 16:27

    And the point of this is an open source framework for you to evaluate your quantized models. Okay? So this is gonna give you some rigor before you actually deploy that quantized model, just to make sure that it is performing as expected.

86. 16:41

    So it's gonna streamline evaluation, uh, your evaluation across text tasks, trust and safety. That's really important. Make sure trust and safety doesn't degrade with quantization. Vision language. Now, if you're interested in deploying your quantized models to device, we even have an iOS app right now, an iOS app that you can use that will measure the latency of

87. 16:59

    the quantized model, and even measure the hardware usage. So you can even check, like, bat- battery drainage for, um, for deploying these models. So feel free to check out our open source repo.

88. 17:11

    And with that, that wraps up the content for today. It was absolutely great being here. So again, remember these five dimensions of AI efficiency as you're building and deploying your models.

89. 17:22

    Thank you so much. And if you're interested, feel free to check out these QR codes. Thank you. [upbeat music]