Llamafile: bringing AI to the masses with fast CPU inference

[upbeat music] So we are here to talk to you about Llama File today and what we've been doing on this project.

So I'll get us started. I'm gonna tell you what Llama File is, how it works. I'm gonna spend a little time talking about why we're building it, why Mozilla specifically is involved, and then I'm gonna hand it over for the fun part to Justine.

Justine's gonna talk about the actual work that she and the open source community have been doing on this project. Lots of insights and tricks and hacks that have made CPU inference go faster than ever before, so that'll be fun.

When we're done, we want you to share the feeling that we have, which is kind of a sense of excitement and empowerment from the knowledge that there are lots of really interesting, juicy, impactful problems still left to be solved in AI, a lot of them.

And the key thing is, it's not just the big folks who can solve these problems. It's individuals and small groups working together in open source. So anyone in this room or anyone listening to this talk can potentially make a big impact in this space.

So what's Llama File? Llama File is an open source project from Mozilla that has the goal of democratizing access to AI. So we do that in a few different ways.

The first is probably how, if you've heard of Llama File, the reason you heard of it is the original magic trick of the project that Justine figured out, which is how to turn weights into programs.

So Llama File is a single file executable that runs without any installation on pretty much every operating system, every CPU architecture, and every GPU architecture.

And that's all. Thank you very much. [laughing] [laughs] That was easy.

Yeah, so by the way, this isn't just one file, like for Windows, right, and a different one for Linux and Mac. It's actually a single file. You can download a Llama File, run it on any computer in the world, and it'll just work, and it'll use the hardware you have, whether that be fancy GPUs or your CPU.

So she'll talk a little more later about how, uh, Justine made that work.

But we're here to talk about another topic too. Most of the talk's actually about this, which is CPU inference speed. Now you might ask, why do we need to worry about CPU inference speed?

We've got these fancy GPUs, right? Well, no disrespect,

almighty Jensen, first of his name, master of market cap. [laughing] Uh, don't strike me down. But I would posit that it is not a universally good thing that we are so dependent in this room on GPUs.

Uh, they are expensive. They're difficult to source. Let's face it, they consume a lot of electricity, which we might wanna think about. But, uh, bigger picture, we have an entire planet of CPUs out there, literally all over the world.

Great hardware, often affordable hardware, and we are at risk of just kind of throwing that all away with this new era of, of AI, and we don't need to do that.

So who here knows llama.cpp? This is an easy question. [laughs] Yeah, right. So ev-- we all know and love this project. We build on top of that project with Llama File, and we contribute our, uh, performance enhances back to it.

Many have been merged in. That project proved that CPUs could do inference perfectly well, and so we have been basically trying to take that performance to the next level.

And as a result of Justine and the community's work, depending on what CPU you're using, what model you're running, what weights, you will see between thirty and five hundred percent speed increases with Llama File, which- Whoo!

... kinda still blows my mind. And I, by the way, I don't think we're anywhere near done. [clapping]

So these things also run locally, by the way. This runs totally on your machine. There's no network access. You could take a pair of scissors and cut the ethernet cord, and it'll work, which is what I asked DALL-E 3 to draw. [laughing]

Okay. I don't think it understood the assignment, but that's all right.

Uh, but seriously, like we're not calling cloud LLMs. There's no monitoring or analytics. No bits leave your machine. It's totally private and local. And everything you need comes in the box.

So whether you wanna just play with a model that you just found on Hugging Face or you wanna start building local, locally running LLM applications on your machine, you got everything you need in the box.

And they're readily available. So, uh, Hugging Face now supports Llama File as a file type, so you can search and filter by Llama File. You could also just search Mozilla on Hugging Face.

You'll find we have a bunch of Llama Files that we've already published. And with a single command, you can create your own. So really, this project is collapsing all the complexity of the open source AI stack down into a single action and a single file.

So why are we involved? Why is Mozilla involved in this? You might be saying, "Don't you folks make browsers?" In fact, we do. We make a damn fine browser, and you should try it out if you haven't lately. [laughing]

But we exist also for a bigger purpose, which is to fight for the web. So I'm gonna ask you a question here. Who here remembers using the original Netscape Navigator?

Don't be shy. No one can see how old you are. [laughing] They can only see how old I am.

A lot of hands, right? So you are my people. You remember the nineties. My CTV,

terrible haircuts. [laughing] Nilly Vanilli, I don't know, whatever. My point is, you remember the early days of the web, and you remember how close we came to one company and one product kinda tr- controlling the whole thing.

And we kinda see that maybe happening again today with AI. No matter what we may think of these companies, the reality is there are some very influential big tech companies that are in a position to maybe control the future of machine intelligence.

And that's itself not a great thing. It's not great for equity. It's not great especially for users' sense of privacy and, uh, safety and agency and control.

And we've had an answer to this for many years. It's called open source. And the answer is right in the name, right? Open source. Transparency is the solution here, and it's important for us to have viable open source alternatives in AI.

And that's why Mozilla is getting involved. That's why we made llamafile and more projects to follow. And, uh, I know many of you in this room are already working on open source AI.

We wanna help support what you're doing. So with that, I'm gonna hand it over to Justine, who's gonna tell you actually the cool part, which is all the things that she and the community have been doing on this project.

Justine. [audience applauding]

Thank you, Steven. Um, so I'm Justine Tunney. I'm the lead developer on llamafile, and as Steven mentioned, I'm gonna talk about some of the cool work we've been doing in the community to help you run the fastest local LLM experience possible.

And in order to do this, we started by first getting it to run on the systems at all. And with Cosmopolitan, what it enables us to do is take your weights in a single file and run it on six OSs.

And there's a really cool hack that makes that possible, which is we basically take a Unix Sixth Edition shell script, put it in the MS-DOS stub of a portable executable, and that enables it to run on Mac, Windows, and BSDs and Linux, et cetera.

Really cool stuff. And once we conquered the portability issue with CPUs, um, I had the opportunity to work with Mozilla on bringing this to AI. And with AI, GPUs are indispensable.

As much as we focus on CPUs, we care very much about GPUs too. But GPUs have always had the problem of distributability. Many people have needed to ship CuBLAS binaries with their project, five hundred megs in size.

Can we really call our software open source if it spends the majority of its time in a proprietary blob? So I never felt comfortable with that, and one of the ways we're solving that is by distributing a library called tinyBLAS that enables you to ship your LLMs to platforms like Windows without depending on SDKs.

It'll run with only the driver installed. But more importantly, performance. Now, LLMs spend the majority of their time doing matrix multiplication. Probably the most important algorithm in the world.

Has a really simple definition. We've been making it go faster for prompt processing, and the way we did it is with a very simple trick we figured out, and this is something all programmers can adopt in their code, and it entails unrolling the outer loop.

So let's talk about what not to do first, and that would be unrolling the inner one. Um, we've all seen fun roll loops, Gen two. It's a bad idea.

Computers can generally do that on their own. If you unroll the outer loops, then your algorithm with matrix multiplication can sort of unfold like a flower and focus on pure flops like a BLAS kernel.

And that's really all there is to it to getting the majority of the benefits of BLAS to make prompt processing go really fast. So what's the impact to this really simple solution?

Um, this generalizes to a wide variety of hardware. We've seen everything from a scrappy hobbyist Raspberry Pi, um, to much bigger computers going significantly faster. You need algorithms like this to exploit the latest capabilities of hardware.

Token generation rates I wanna believe. If you use a gaming computer like Intel, you're going to see better performance with llamafile on those too. Really exciting stuff, like particularly with Alder Lake, we are able to get a four X improvement.

But Threadripper most of all, for the first time, AVX512 is available to consumers, and we've been able to help you prepare for that future. So if you have the Threadripper, you're gonna see better performance than ever, almost like a GPU.

Now, prompt eval speed, what makes it important is

it's really cool to be able to generate text and use a chatbot, but the way I want you to think about llamafile is it's more of a word-crunching machine that can help you understand our world.

And I love to use it personally for tasks like summarization. I love that it can help me read a blog post. And we've used other performance tricks too.

When... With NVIDIA, part of what makes them so successful, it's not just great hardware, but they built a great framework too. And their framework helps developers think about programming in a different way that helps them be successful.

I mean, who here thinks that software with CPUs just gets slower each year? Can I see a raise some hands? [laughs]

Well, part of the, um, one of the things that's great about NVIDIA is they showed us a better alternative to getting performance. And when I learned how to program in CUDA, I found one of the most important functions was syncThreads.

This is how you can implement it for CPU in, like, 10 lines of code. And if you use the lockstep programming model, use your CPU as though it were a GPU, you can get really good performance.

Now, this is gonna be a demo.

Showing the impact of this work before and after for summarization. And here we're gonna be processing an essay by Dijkstra, really cool, worth reading. But I want you to watch, like, as it processes it in terms of speed.

Here we see it going, and on the right we have the new version.

And it's like bam, bam, bam, bam. Huge night and day difference. It's already summarizing it, and the old version is, like, nowhere close.

So that is the kind of new performance you can expect, and it's the kind of performance that's actually possible, which I wouldn't have imagined beforehand. [cheering] It's really great. [clapping]

Thank you. CPUs can do so much. And people in the community have, like, loved this work. We've managed to attract some, like, really amazing contributors like Aiwyn, the inventor of KQuant.

It's very popular. I'm sure many of you have used them. He got them going 2X, 4X faster too on both x86 and ARM. So if you use quantized formats, those are gonna be better than ever with Llama File now too.

And it's worth mentioning that we've, um, seen really interesting things about this. Like, people, once we put this out into the world, people have come back and given us feedback and reported, like, their own experiences.

We found out that someone was running Mixtral 8x22B on a $350 CPU. And to me, that's just wonderful, um, because performance matters, but it's not really the thing we care about.

What we care about is intelligence. And to have the intelligence, you need to run bigger models, and RAM is cheap with CPUs. For the price of a graphics card, I put 512 gigs in my workstation, and that means I can run all the frontier models coming out, and I just have to wait a little longer, but I

get a much more intelligent answer. And the fact that that went from impossible to possible for most consumers is, um, you know, a story I want you all to tell.

Individuals are making a big difference, and you can be a part of that too. And I'm gonna hand it back to Steven, who can explain what Mozilla can do to support you getting involved in that effort. [clapping]

Thanks, Justine. Thanks a lot for all your efforts.

So yeah, that, that's a key message of this talk is, uh, anyone in this audience, you don't have to work for these big, giant, largest-in-the-history-of-humanity companies necessarily to have a big impact.

There's lots of headroom here. There's lots of unsolved, interesting problems in this space,

and we wanna get involved in helping. So we recently launched a program called Mozilla Builders, and this is a program by which we either sponsor or, in some cases, co-develop impactful open source AI projects.

Llama File was actually the first in this program. I'm happy to announce today the second, which is SQLite Vec. This is from a developer named Alex Garcia. Alex is adding vector search capability to SQLite.

And so for some folks in this audience, that'll have some obvious implications that are kinda cool. [clapping]

But just imagine, remember that little modest Raspberry Pi 5? So, like, imagine now a local LLM, open LLM, running privately on that machine with no network connection, connected to your personal private data, which you can use with confidence that it's safe to do RAG and other interesting applications.

That's the kind of stuff we're talking about here.

We also just launched our own accelerator. It's called the Mozilla Builders Accelerator. So we are offering 100,000 US in non-dilutive funding for open source projects that advance the promise and potential of local AI.

So that's AI applications running at the edge on user devices. These are some of the bullet points of areas we're particularly interested in, but it's not an exclusive list.

And you don't have to necessarily be building a company to apply for this accelerator.

So if you wanna learn more about the accelerator, this QR code will take you there. Take a picture of that, or just go to future.mozilla.org/builders. And, you know, uh, Justine and I, and a lot of Mozillians are here this week.

If you have something you're working on or something you think we should know about or you wanna collaborate with us, please find us, reach out, or reach out to me via email.

So thanks again. Thanks to Justine and the community on all their work on Llama File.

Thank you, Steven.

Thank you. [outro music]