AI Engineer World's Fair 2024
A Practical Guide to Efficient AI
Read the talk
A Practical Guide to Efficient AI
Getting an AI demo into customers’ hands means fitting real resource budgets. Small models and quantization offer two practical starting points, with evaluation deciding what is safe to deploy.
From a talk by Shelby Heinecke
Before you start: Basic familiarity with language models, model weights, fine-tuning and inference will help; no prior quantization experience is required.
From a working demo to a service customers can use
A new model demonstrates an impressive capability. What does it take to turn that prototype into something customers can use at scale? The demonstration establishes that a task is possible; production also demands that the system fit its operating budget. Efficiency helps close the gap between capability and deployment. That is the practical problem behind Shelby Heinecke’s guide.
Heinecke leads an AI research team delivering LLM solutions to Salesforce’s data platform, the foundation beneath its products. Alongside that production work, the team investigates agents, language models and on-device AI. At the time of the talk, she reports more than 15 research papers and more than six open-source repositories from her team. Her own research background includes learning algorithms designed to use samples and communication efficiently.
That work sits inside a longer enterprise deployment history. Heinecke reports ten years of Salesforce AI deployment, with its research team founded in 2014, more than 300 AI patents and more than 227 research papers. Trust is part of the deployment requirement: she describes participation in six ethical AI councils and the White House commitment to trusted AI. The timeline then moves into the 2022–2023 wave of code-generation products, Service GPT, Einstein GPT and Tableau GPT. At Fortune 500 scale, the resources needed to serve these products become a central engineering concern.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Every deployment has a resource budget
The constraint changes with the deployment environment, but it never disappears.
| Environment | Main constraint |
|---|---|
| Public or private cloud | Ongoing GPU, CPU and disk costs |
| On-premises cluster | Operating costs and a fixed supply of GPUs |
| Phones and other small devices | Tighter hardware and resource limits |
In the cloud, resource consumption becomes cost to serve. On premises, paying for infrastructure does not remove the capacity ceiling: the model must work with the GPUs available.
Apple’s then-recent announcement of an LLM for newer devices gives Heinecke a concrete reason to look beyond servers. Phones suggest a wider set of possibilities for language and multimodal models: tablets, personal laptops and edge devices. Those options also impose tighter budgets. A model that is affordable in the cloud may still be too demanding for a small device, so the deployment target needs to influence model design and selection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Five places to improve efficiency
Efficiency is not a single optimization applied just before release. Heinecke separates it into five dimensions, each offering a different place to intervene:
- Architecture. Start with a small model where possible. Other choices include mixture-of-experts architectures and, when building a model, efficient attention mechanisms.
- Pre-training. If training from scratch is necessary, mixed precision and scaling methods help address the substantial GPU expense.
- Fine-tuning. Parameter-efficient methods such as LoRA and QLoRA avoid updating every base-model weight. They concentrate trainable changes in a much smaller set of parameters.
- Inference. Once the model is trained and adapted, post-training quantization and speculative decoding offer ways to reduce serving costs.
- Prompting. Prompt tokens consume memory and affect latency. Concise templates and prompt compression reduce the amount of input the model must process.
These are separate opportunities, so choosing an efficient architecture does not eliminate the need to examine training, serving or prompts.
The two practical directions developed here are small-model selection and post-training quantization. The first changes the model you begin with; the second changes how an already-trained model’s weights are represented for inference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fewer weights expand the deployment options
The resource implications of model size become clear when looking at the large models that helped define the preceding wave of releases. Heinecke’s examples are PaLM at 540 billion parameters, BLOOM at 176 billion and YaLM at 100 billion. Parameters are the network’s weights: they occupy storage and participate in computation. Large weight sets therefore affect pre-training, fine-tuning and serving, not just the size of a downloaded checkpoint.
For this discussion, Heinecke defines a small LLM as one with 13 billion parameters or fewer. The sizing slide puts that category beside the much larger models, making the difference in scale visible.
Fewer weights generally mean lower RAM and disk requirements, less GPU or CPU work, and faster fine-tuning. A forward pass also has less work to do, which can lower latency. Together, these benefits widen the set of places a model can run: cloud and on-premises systems remain options, while sufficiently small models can also fit phones, personal laptops and edge devices. The next question is whether that smaller model can perform the task well enough.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Small models become compelling on specific tasks
The first example is Phi-3-mini, the 3.8-billion-parameter member of the Phi-3 family. Heinecke presents a technical-report comparison in which it outperforms a 7B model on the displayed evaluation. The useful implication is that parameter count alone does not determine capability: a much smaller model can be a serious candidate. The comparison is specific to that evaluation, rather than a claim that it exceeds every larger model.
The next example goes below a billion parameters. MobileLLM is presented in its 350-million-parameter configuration, small enough to make mobile and edge deployment especially interesting. Heinecke reports that, after fine-tuning, MobileLLM-350M performs on par with a 7B model on the referenced tasks. Here, the relevant setting is specialized API calling, not general language-model parity. Fine-tuning is part of the result: the question is what the small model can do after adaptation to its intended job.
Function calling provides an even more explicit specialization example. Octopus v2 adapts Gemma 2B for Android tasks. Heinecke reports that the fine-tuned Octopus model outperforms GPT-4 and a Llama 7B model on the referenced Android function-calling tasks. The scope matters: this is a model trained for a particular interface and workload. It shows why a deployment decision should compare candidates on the job they will actually perform, including the benefit of task-specific fine-tuning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reduce the bits used to represent each weight
After model selection comes an inference optimization: post-training quantization. Quantization is an established technique; the application here is to language and multimodal models. Instead of retaining every weight as a 32- or 16-bit floating-point value, use a lower-precision representation such as 8 or 4 bits. The operation reduces representational precision, rather than reducing the number of model parameters.
The storage arithmetic explains the appeal. Moving from 32 to 4 bits reduces the nominal bits per weight eightfold. The following Python calculation lets you inspect that weight-storage budget for a model with seven billion parameters:
python
parameter_count = 7_000_000_000
for bits_per_weight in (32, 16, 8, 4):
weight_bytes = parameter_count * bits_per_weight // 8
weight_gb = weight_bytes / 1_000_000_000
print(f"{bits_per_weight:2d}-bit: {weight_gb:.1f} GB of weight data")
This calculates weight data only. A deployed model also needs runtime memory, and a quantized representation can carry additional metadata, so the bit ratio is not an identical reduction in total system memory.
Heinecke’s displayed Llama examples—7B, 13B and 70B—show reduced disk usage after 4-bit quantization. She then turns to a separate multimodal-model study: the displayed comparison reports lower time to first token at 4 bits than at the original 16 bits. Time to first token measures how long a request waits before generation begins, rather than the rate of generating the entire response. Lower precision can help latency, but the gain depends on the model, hardware and runtime. Once storage and latency improve, the remaining question is whether the model still performs its task well.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Accuracy determines how far quantization can go
The vision-language evaluation compares multimodal models at their original 16-bit precision with 8- and 4-bit quantization. Heinecke reports essentially unchanged task performance through 4 bits in the displayed vision-language evaluation. In that example, the storage and latency benefits come without a meaningful movement in the task score.
The next part of the graph establishes the limit: performance drops at 3 bits in the displayed comparison. This is why a favorable 4-bit result is a reason to test quantization, not a reason to skip testing. Evaluate the quantized model itself. The appropriate precision is the one that preserves the behavior needed for the deployment, and quantization can be considered for conventional ML models as well as language and multimodal models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with existing quantization tools and releases
You may not need to produce the quantized weights yourself. Heinecke points first to llama.cpp, describing a range from 16-bit representations down to approximately 1.5 bits. She recommends checking Hugging Face for an existing compatible quantized release before doing the conversion, and mentions Python and Java wrappers for integration.
The second route is ONNX Runtime, a runtime with a history beyond LLMs and support across many programming languages. Heinecke highlights its 8-bit quantization. These are the capabilities discussed in the recording; the linked project documentation describes current tooling, whose formats and APIs have continued to evolve. The choice of route should fit both the model and the environment where it will run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test quality, safety and behavior on the device
Obtaining a quantized checkpoint is not the final deployment step. Heinecke introduces Mobile AI Bench, a recently released open-source framework from her team, to make evaluation more systematic before a model reaches customers. Its purpose is to confirm that the quantized model still behaves as expected.
The evaluation covers text tasks, trust and safety, and vision-language tasks. Trust and safety deserve explicit attention: preserving a task score is insufficient if quantization degrades safety behavior. For on-device deployment, the described iOS app adds latency and hardware-usage measurements, including battery drainage. Quality evaluation and device profiling answer different questions—whether the model remains suitable for its job, and whether it can do that job within the device’s operating budget.
That brings the deployment problem back to all five dimensions: architecture, pre-training, fine-tuning, inference and prompting. Small models and quantization are practical entry points, while the final choice depends on measured quality and resource use in the environment customers will actually use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original report describes the 3.8B-parameter Phi-3-mini, its training approach and benchmark comparisons.
Sub-billion-parameter architectures and fine-tuned API-calling evaluations, including comparisons with LLaMA-v2 7B.
A specialized small model for function calling, evaluated on Android APIs against larger model baselines.
Evaluation tools and mobile app setup instructions for testing model quality and resource consumption.
C/C++ model inference project with tools and documentation for running quantized models.
Further reading
Evaluation methodology and results for quantized language and multimodal models, including quality, safety, latency and device resource usage.
Updates since the talk
Current guidance for quantizing ONNX models and assessing the resulting accuracy and performance.
Read the complete timestamped transcript
- 0:00
[upbeat music] I go to a lot of AI conferences.
- 0:15
I go to AI research conferences. I go to, you know, more general tech conferences. And what I absolutely love about this conference is that it's really about the builders, and it's really about the techniques that we need to get AI into the hands of our customers.
- 0:29
And so we're all here in the AI space. We're all AI practitioners here, and we know that AI is developing at an unprecedented pace. It's pretty hard to keep up with it, right?
- 0:40
Every week there's a new model, a new capability, a new feature. So much to keep up with. And when we see these new models, capabilities, and features, they're often shown to us as a demo or a prototype.
- 0:55
And as builders and engineers here today, we all know there is a big difference between a demo and a prototype, and scaling and productionizing AI. So one of the biggest ways that we can bridge that gap, we can go from having cool, awesome demos to actually bringing that to customers, is with efficiency.
- 1:15
So if we have techniques for making our AI efficient, we can get closer to productization. And so that's what I wanna tell you about today. I'm gonna tell you about some practical ways you can take away today to start making your AI models more efficient.
- 1:31
So let me introduce myself. So I'm Shelby, and I lead an AI research team at Salesforce. My team ships AI today. So we deliver, for example, LLM solutions to our data platform at Salesforce.
- 1:45
The data platform is the foundation of all of Salesforce, that scale. Now, while we're delivering AI today, we're also envisioning what we'll need for tomorrow. And to do that, we've released over fifteen cutting-edge research papers in agents, LLMs, on-device AI, and more.
- 2:05
And we've also released over six open source repos. So my team has released these repos. I'm gonna talk about one of them, uh, today, so you'll get to see.
- 2:13
And, um, this is all in vein of pushing AI forward and getting the AI that we'll need for tomorrow. Now, a little bit about my personal background. I have a PhD in machine learning, so I focus on developing learning algorithms, uh, that are sample and communication efficient, and I have a bachelor's and master's in math as well.
- 2:34
So if you're interested in learning more about my team, my background, wanna connect on LinkedIn, feel free to scan the QR code. I'm always happy to chat with you all.
- 2:44
Now, what about Salesforce? This is the-- This is the AI in the Fortune 500 section. Let's talk about Salesforce and what we're doing. Salesforce has been deploying AI for ten years, everyone.
- 2:54
Ten years. So the AI research team was founded in twenty fourteen, and since then, Salesforce has accumulated over three hundred AI patents, over two hundred and twenty-seven AI research papers, all in the p-- all in the last ten years.
- 3:10
And you can see here the map of all the deployments that have taken place. Okay. And at Salesforce, trust is our number one value, so we don't just deliver-- we don't just build and deliver AI in isolation.
- 3:22
We build and deliver trusted AI. That is a key. So to do that, we're a part of six ethical AI councils, and we're also involved in the White House commitment for trusted AI.
- 3:34
So I wanna zoom in here. Past two years. That's where all the AI action has been happening, right? The past two years. Let's look at twenty twenty-two and twenty twenty-three.
- 3:43
What's Salesforce been up to? Well, we've been deploying a lot of LLM products, right? If you look here, you'll see, um, you'll see code gen-based products. You'll see Service GPT, Einstein GPT, Tableau GPT.
- 3:56
That's very similar to the rest of the tech industry, right? Like, if we zoom out, the rest of the tech industry, we're de-- we're deploying LLM products. And now for us to do it at Salesforce, efficiency is key.
- 4:06
Think about Salesforce scale. Think about Fortune 500 scale that we're talking here. Efficiency is key.
- 4:13
And we're all in the same boat here. We're all working on the same deployment environment. So let's review that a little bit. When we've got an AI model, we're mostly deploying...
- 4:21
You know, a lot of times we're deploying on a cloud, right? Private or public cloud. We're paying for resource consumption. We're paying for, uh, we're, you know, we're paying for GPU, CPU, disk space.
- 4:31
We're paying for all of that, so we've gotta keep that in mind, right? When we deploy, we're paying for that cost to serve. Or now we're seeing even on-prem solutions.
- 4:38
Maybe we have an on-pre-- Maybe you have an on-prem cluster. So not only, only are you paying for that, you've got restricted GPUs to work within. And more recently, and this is pretty exciting, small devices.
- 4:50
We're seeing LLMs being feasible on small devices. So if you guys were paying attention to the news in the past couple of weeks, you'll see-- you'll remember that Apple has announced their LLM on their newer devices.
- 5:02
So this is so exciting. Um, and so if we're seeing it on, uh, on iPhones and small devices like that, we can think maybe LLMs and LMMs, multimodal models, on tablets, on laptops, on edge devices.
- 5:16
Now, that's an even more challenging situation, right? Small devices have even worse hardware, have even more resource constraints. The point here is that when we're deploy-- when you're deploying AI models, you're deploying in these constrained environments.
- 5:28
It's never-- We're never in a situation where we have infinite resources. So efficiency is going to be key. So how do we make AI more efficient?
- 5:38
So that's what I wanna talk to you today about. I've summarized it into five dimensions, five orthogonal directions that I would love for you to consider as you're de-- as you're thinking about building your AI for customers and deploying.
- 5:50
The first-- And this is just scratching the surface. This is just scratching the surface, but I'm hoping these five dimensions will be easy for you to remember. The first is picking efficient architectures from the very beginning.
- 6:02
From the very beginning. So this includes picking small models. I'm gonna talk about that today. Um, this includes using sophisticated architectures such as mixture of experts, for example.
- 6:13
And if you're building your, um, architecture from scratch, includes, uh, choosing efficient attention mechanisms and so on. So there's a lot we can say there. Today, I'm just gonna touch on a little bit.
- 6:23
Moving on to the second one, efficient pre-training. Now, not a lot of us are doing pre-training. It's a really expensive thing to do. But if you're doing it, you know the GPU costs.
- 6:32
You wanna use mixed precision training, scaling methods, among other methods here. So definitely make your pre-training efficient. Now, efficient fine-tuning. This is the, this is the case mo-- a lot of us are in today, efficient fine-tuning.
- 6:44
It's-- You wanna pick methods that are not optimizing all of the weights, every single weight, full fine-tuning. You wanna pick, um, methods that are, that are only optimizing, you know, a subset of those weights.
- 6:55
So think about LoRA, QLoRA, and so on.
- 6:59
And fourth, the fourth dimension, efficient inference. So you've got your model, it's pre-trained, it's fine-tuned, you're ready to al-- you're almost ready to serve it. How can we do that efficiently?
- 7:08
We're paying for cost to serve, right? So with that, you wanna consider post-training quantization, which I'll get into today, and speculative decoding, and there's many others to cover as well.
- 7:17
And finally, prompting. Prompting, we gotta think about that. Prompts, uh, you know, consume memory. They also, uh, uh, directly affect latency as well. So you want your prompts to be as concise as possible.
- 7:28
Concise as possible. So think about template formatting and prompt compression. Now, with our limited time here today together,
- 7:37
I'm gonna dive into two crucial directions that you can take away with you and apply right away. The first direction is around efficient architecture selection. I wanna tell you about the power of small models.
- 7:48
Small models are coming back, guys. We went, we went big models. Small models are, are super efficient. We'll talk about it. Second, I wanna go into efficient inference. I wanna tell you about post-training quantization.
- 7:59
This is something that you could, you could actually apply at the end of the day on your model. So efficient and so quick.
- 8:05
So let's get started with small LLMs, the power of these small LLMs. So let's, uh, think about the past two years. As I mentioned, every week, new model, new model, new feature.
- 8:19
When we look at these, at these LLMs that have been released, they're mostly pretty big. They're mostly pretty big. So here are just a few. These are older models, but I just wanted to prove a point here.
- 8:28
If we look at the PaLM model, for example, five hundred and forty billion parameters, right? So parameters, again, every one is the number of weights in that deep neural network.
- 8:36
Five hundred and forty billion parameters. These other models, Bloom, Yalm, one seventy-six billion parameters, one hundred billion parameters. So those parameters have got to be stored in memory. They're all gonna be used in computation, GPU computation, CPU computation.
- 8:51
They're gonna take up space. Long story short, these huge models are resource hungry. They're gonna take a lot of resources to train, certainly to pre-train, to fine-tune, and to serve.
- 9:02
Now, in parallel, let's think over the past several months. We're seeing these smaller models emerge. And, and when I think about small LLMs, I'm thinking models that are thirteen billion parameters or less.
- 9:16
We're seeing these emerge, and for very good reason. They're emerging for good reason. There's so many benefits to these smaller models. So as you can imagine, with less parameters, with less weights, they consume less RAM, they consume less GPU, less CPU, less dis-disk space, and they're just faster to fine-tune.
- 9:35
They're super e-- they're super resource efficient. This is exactly what we're looking for today. They're also low latency. Fewer weights mean the forward pass is faster. There's just fewer weights to go through, right?
- 9:46
And both of those together, the resource efficiency, the low latency, makes them perfect for additional deployment options. So not only can you take these small LLMs and deploy them on the cloud, on on-prem, they can also be deployed on mobile if they're small enough.
- 10:01
They can be deployed on laptops for personal models. They can be deployed on edge devices. They're super nimble and super useful. So let me tell you about how... So what I wanna do today is tell you about a few small state-of-the-art LLMs to keep in mind as you're building your solutions for your customers.
- 10:18
So the first one I'm gonna tell you about is, uh, Phi-3. You guys may have been, um, may have heard of this one. This is a three point eight billion model.
- 10:25
Super, super small. And as I'm talking about small models, I showed you these five hundred and forty billion parameter model. Now we're talking about a three point eight billion parameter model.
- 10:34
Your first question might be, "What is the performance? Is the performance good?" So interesting. So Phi-3 is actually a pretty-- it's a very strong performing model. So as you can see here, I took this clip right from their technical report.
- 10:47
Feel free to check it out. As you can see here, Phi-3 is outperforming a very, very well-known seven B model, a model that's almost twice its size. So this three, three B model is pretty powerful for being so small.
- 11:00
And now, with that model being powerful, we're seeing even smaller models emerge, even smaller models, because even smaller models will fit on edge devices, on mobile, and so on.
- 11:11
So what I wanna point out to you is MobileLLM. It has less than one B. So we're-- This has three hundred and fifty million parameters. Three hundred and fifty million, not even a billion parameters, so super, super tiny.
- 11:24
And here's the key. After fine-tuning, after fine-tuning, it's on par with the seven B model on tasks. So this is one of the takeaways I wanna share with you, is that the power of these small models, you-- the way you use them is important.
- 11:38
They're, they're great. After fine-tuning, they are very competitive. That is what this is showing. And finally, I wanna bring up a model that's really interesting for function calling. So, um, this, this Octopus model is a fine-tuned model.
- 11:51
It's fine-tuned Gemma two B. It's fine-tuned on Android tasks. And again, a two B model, they are showing after fine-tuning, it's outperforming GPT-4, Llama 7B on these Android tasks.
- 12:04
So super, super promising. So definitely check out these small LLMs. They have a ton of potential.
- 12:11
And finally, I will go to our next topic, which is quantization. This is about inference.
- 12:17
So what is quantization? Quantization is actually not a new topic. Not a new topic. What's new is applying it to LLMs and LMMs. So the idea of quantization is to take a big number and to map it to a smaller number.
- 12:31
So what we wanna do for quantization for LLMs is we want to reduce the precision of the weights. So typically, weights in LLMs, depending on the model, is, um, can be 32-bit or 16-bit floats.
- 12:44
What we wanna do, what, what quantization does is reduces that 32 or 16 down to 8, down to 8, 4 bits, down to... Actually, you can specify just a smaller number of bits, reducing the precision of all those weights.
- 12:56
So as you can imagine, that's hugely, hugely beneficial. Massive efficiency gains. So as you can see here, if... As you can imagine, if each weight was originally 32-bit, taking up 32-bit space, now we reduce it to 4-bit, it's gonna take up a lot less space, it's gonna consume a lot less memory, and it's gonna be...
- 13:12
consume a lot less CPU and GPU. So as you can see here really quickly, just some models, looking at these Llama models, 7B, 13B, 70B. The original, you can see the disk space it was taking up.
- 13:22
After 4-bit quantization, it's taking up a fraction of the dis- the space. And now, l- what about latency? So as the resource consumption comes down, the latency improves. So as we can see here in this, in this study on large multimodal models.
- 13:37
So these are large LMMs. 16-bit was their original, um, encoding. Originally 16-bit. Now, if you look at the 4-bit, 4-bit quantization, you can see that the latency, measured here as time to first token, has decreased.
- 13:51
So lots of benefits. So reduced, uh, so reduced resource consumption, faster. Again, though, the most important question is, is the performance still there? Are we making this hu- are we making a huge trade-off by, with this?
- 14:03
And the good news is, no. This is, this is pretty amazing. Quantization generally has negligible effects on performance. So I wanna show you that. So look here at this chart, at this graph.
- 14:16
And again, we're looking at LMMs. And you can see here, we've, uh, on this particular task, this well-known vision language task, we took the LMMs and we measured performance on 16-bit, and then 18-bit quantization and 4-bit quantization.
- 14:31
And as you can see, essentially no movement. 4-bit quantization was essentially free. Like, we could just quantize it with 4 bits and just enjoy reduced latency, enjoy inc- increased, uh, improved lat- uh, sorry.
- 14:44
Enjoy reduced resource consumption, enjoy improved latency, and no effect to performance. Retained performance. However, you can take this too far. There is, there is a way to take this too far.
- 14:54
So as you can see, when we quantize down to 3 bits, performance did drop. So evaluating your quantized model is super important. So don't just assume 4-bit is the answer.
- 15:04
Definitely measure. Uh, you definitely wanna measure your quantized performance. So really quickly, so you can get started on this today, you can add... You can quantize any of your models, whether it's ML models, LLMs, LMMs, and so on.
- 15:16
I wanna just highlight a couple of, of frameworks that are really, um, awesome for that. So Llama CPP is one of the most well-known frameworks right now. It's gaining a lot of traction.
- 15:25
It has quantization from 16-bit all the way down to 1.5 bits, so pretty crazy. Um, wide adoption. So actually, you may not even need to quantize the models that you're using.
- 15:34
Just check Hugging Face. A lot of people are, as they're releasing their models, they're go- they're releasing the, the Llama CPP compatible quantization models too. So pretty awesome. And there's, uh, Python and Java wrappers.
- 15:46
Second thing I just wanna quickly mention, ONNX Runtime. This is, uh, ONNX has been around for some time. If, if you've been around since the ML days, ONNX was around in the ML days.
- 15:55
And so, um, they have some 8-bit quantization. And y- you know, the beauty of ONNX is that it's compatible across so many programming languages. So definitely take, uh, take a look at these, and there's a bunch of others ones you can consider too.
- 16:09
Now, final point here. As we mentioned with quantization, now, um, you have your quantized model. I mentioned before, it is still important to evaluate your quantized model before you deploy it.
- 16:18
So I want to introduce to you one of the open source repos that my team just developed. We just released this maybe, like, a w- a week ago. It's called Mobile AI Bench.
- 16:27
And the point of this is an open source framework for you to evaluate your quantized models. Okay? So this is gonna give you some rigor before you actually deploy that quantized model, just to make sure that it is performing as expected.
- 16:41
So it's gonna streamline evaluation, uh, your evaluation across text tasks, trust and safety. That's really important. Make sure trust and safety doesn't degrade with quantization. Vision language. Now, if you're interested in deploying your quantized models to device, we even have an iOS app right now, an iOS app that you can use that will measure the latency of
- 16:59
the quantized model, and even measure the hardware usage. So you can even check, like, bat- battery drainage for, um, for deploying these models. So feel free to check out our open source repo.
- 17:11
And with that, that wraps up the content for today. It was absolutely great being here. So again, remember these five dimensions of AI efficiency as you're building and deploying your models.
- 17:22
Thank you so much. And if you're interested, feel free to check out these QR codes. Thank you. [upbeat music]