Harnessing the Power of LLMs Locally

[upbeat music] Right. Good day, everyone.

Good to see you all. Today, I'm here to tell you to har- how to harness the power of local LLMs using our Rust library. Quick intro. I'm Mithun, as you just heard, but I go by [REDACTED:username] online.

I hail from Australia, hence the accent, but I live in Sweden. I do a lot of things with computers, but my day job is at Ambient, where I build a game engine of the future.

Today though, I'm here to talk to you about llm.rs, a Rust library that I maintain.

So llm.rs or LLM between friends. I realized that I have to disambiguate when I started, when I signed this Simon's newsletter. [laughs] It's an all-in-one solution for local inference of LLMs, but what does that actually mean?

Well, most of the models we've discussed at this conference have been cloud models. You have ChatGPTs, your Claudes, your Bards. Local models offer another way, where you own the model and it runs on your computer.

So let's quickly go over what that actually means.

First up, size. Model size can be used as a rough proxy for the intelligence of the model. Most hosting models are really, really big. You can see that it's dominating the right-hand side of the chart there.

You have your GPT-3, your GPT-4, we'll get back to that. Your Gopher, your PaLM 2. These are all insanely big in comparison to the open source models we have.

We're gonna, we're beginning to see some, uh, bigger models thanks to, uh, Llama and Falcon, but even they pale in comparison to what the bigger players can do. This means the local models don't have the same capacity for intelligence.

However, a smaller, more focused model may be able to solve problems l- uh, better than a large general model. By the way, we don't actually know what size GPT-4 is.

That's rumors. Uh, only OpenAI knows. Next, let's talk about speed and capacity. Cloud models run specialized hardware with special configuration. Local models run on whatever hardware you can scrounge up, including rented hardware.

The further up the axis you go, the more speed and/or parallel inference you can do, but the more inaccessible it becomes. This end, a few hundred dollars. That end, a few hundred million dollars.

Next up, latency. Cloud models need the full prompt before they can start inference, and you have to wait for the message back, uh, back and forth. Local models can give you a response immediately.

You can feed the prompt as you go along. This is very important for conversations where you want the model to be able to process what you're saying as you say it.

And of course, you can't escape talking about cost. The cloud vendors will charge you a per token price. When running locally, it's entirely up to you how much it costs you to run the machine.

If the running cost of your, uh, model is less than the cost of running your workload through the cloud, you're going to make a profit, and if you're running on a machine you already own, well, that's basically free, right?

With the cloud, you have to offer, you have to use the models they offer you. Some vendors offer fine-tuning, but they often charge more than, uh, just using the regular model, and they often charge you for the pro- process of actually fine-tuning.

This means that it's not often cost-effective to, to actually do that. With local models, the sky's the limit. There are hundreds, potentially thousands of custom models that can suit any need you have.

Knowledge retrieval, storytelling, conversation, tool use. You name it, someone's probably already done it, and if they haven't, fine-tuning an existing model for your own use is easy enough. Special shout-out to Axolotl over there, which makes it easy to fine-tune, uh, models of any architecture.

And of course, privacy. [laughs] There are some questions you don't want to ask the internet. Local models let you privately embarrass yourself. [laughs]

Now, you might be wondering how it's actually possible to run these models locally. That, my friends, is possible with the power of quantization. If each model is billions of parameters, and those parameters are, are like individual numbers, how could you, how could you possibly run them on consumer hardware when there's only so much memory given for, uh,

available for a given, uh, performance level? Well, we can use quantization. Quantization lets you lossily compress a model while maintaining the majority of its maths. We can take the original model, here in blue, and squish it down to something much smaller using one of these green formats.

This is a secret sauce that makes it, uh, viable to run models locally. Smaller models aren't easier to store, aren't just easier to store. They can also run faster as your process, uh, uh, as your computer can process more of the model at any given moment.

But that's enough about local models. You've probably already h-heard much, uh, much that already. Let's talk about the actual library.

It all started with this man who built something you may have heard of. Of course, I'm referring to Llama CPP, and that's what it looked like on day one.

Look at the mere 98 stars. How pedestrian compared to today, where it's forty-two thousand stars. Uh, but let's go back to March when I first saw it. When I saw it, I had but one idea.

It's time to rewrite it in Rust. [laughs] For both the meme and because I wanted to do use it for other things. Well, I wanted to say I... Well, I said I wanted to do it, and I did.

But to the right here, [REDACTED:username] was also working on the same problem, and well, there was just one catch. He beat me at... He beat me to it, completely beat me to it.

I'm not afraid to admit it. Luckily, we came together, merged our projects, and I ended up as the maintainer of the resulting project, and that's how LLM was born.

So you might be wondering, why? If Llama CPP exists, why use La- uh, llm.rs? Well, with llm.rs, I had six principles in mind. It must be a library. When I first started in March, Llama CPP was not a library.

It was an application, and that made it impossible to reuse. It must not be coupled to, to an application. You must be able to customize its behavior. You must be able to go in and change every little bit of it to make it work for your application, and we sh- we shouldn't make any assumptions about how it's,

it's going to be used.

Uh, it should support a multitude of model architectures. Of course, llama.cpp supports Llama and now Falcon, but clearly there are more out there. Next up, it should be Rust native.

It should feel like using a Rust library. It shouldn't feel like using a, a library with bindings, and it should feel, work how you expect a Rust library to work.

Next up, backends. It should support all other, uh, all possible kinds of backends, so you can run it on your CPU, your GPU, or of course, your ML-powered toaster.

I'm sure that's gonna be a thing. But we have, we've s- we're gonna see it coming, I'm s- I swear. And finally, platforms. It should work the same whether it's on Windows, Linux, macOS, or something else.

It shouldn't have... You shouldn't have to change it significantly to make it work, because deployment has always been an issue.

Today, I'm proud to say we support a myriad of architectures, including the, uh, the darlings of the movement, Llama and Falcon. These architectures all use the same interface, so you don't have to worry about changing your code to use a different model.

This is made possible by the coordint- coordi- [laughs] concerted efforts of my co-contributors Lucas and Dan, who couldn't have done this without, as well, as well as many others.

Here's some sample code for the library. I won't go too much into it because it's quite dense, but the idea is that you load a model right there at the top.

You can see it's actually quite small. And with that model you create sessions which track an ongoing use of the model. You can have as many of these as you would like, but they do have a memory cost, so you want to be careful.

Once you have a session, you could pass, you can pass a prompt in and infer with the model to determine what comes next. You can k- keep reusing the same session, which is very useful for conversation.

You don't need to keep re-feeding the context. The last argument of the call, uh, of the function is the callback. That's where you actually get the tokens out. Um, it's worth noting that the function itself is actually a helper.

All it does is call the model in a loop with some boundary conditions. So if you wanna change the logic in some, uh, significant way, you can. We're not gonna stop you from doing that.

One last thing about this though. You see all the calls to default there? Those are all customization points. You can change pretty much anything about this. You can change how the model is loaded.

You can change how it'll do the inference. You can change how it'll sample. The entire point is you have the control you need to make the thing y- uh, you need to work.

Here's a quick demo of, uh, the library working with Llama 7, uh, billion on my MacBook CPU. It's reasonably fast, but it could be faster, right? Well, thanks to the power of

GPU acceleration, we have something that's much more usable. And believe me, it's even faster on NVIDIA GPUs. AMD and Intel, uh, support, uh, pending.

Now let's talk about what you can actually do with the library.

Let's start with three community projects to begin with. First, we've got Local AI. Local AI is a simple app that you can install to do inference locally. There's nothing magical about it.

It's just exactly what it says. I think that's really wonderful because it, it means anyone can download this app and get ready... Uh, get, be able to use local models without having to think about it.

Next up, LLMchain. It's, uh, LangChain but for Rust, and of course it supports inference with our library. And finally, we have Flowonium, which is a flowchart-based application where you can build your own workflows.

I think we've seen a few of that, few of those at this con- uh, this conference. And you can combine and, uh, create vi- uh, nodes to, uh, build the workflow you need.

And of course, it supports the library as an inference engine.

Now, I wouldn't be a very good library author if I didn't actually test my own library. So I'm gonna go through three applications. The first two are proofs of concept.

The first is llmcord. It's a Discord bot. [laughs]

You can see it's exactly what you'd expect. You send, uh, give it a prompt, it'll give you a response. Any hitches you see come from Discord limits, not from the actual, uh, inferencing itself.

You can see, bam, all there. When an issue, when a user, uh, issues a request for generation, it goes through this process here where the request goes through a generation thread, uh, with a channel.

That channel is then used, uh, to create a response task, and then that response task is r- responsible for sending the responses to the, uh, user. Now, the interesting thing is these sessions are created and thrown away immediately with each query.

But you don't need to do that. If you keep them around, you can actually use them for conversation.

And just to illustrate, this is just like the request response w- workflow you would use for anything. If I just take what I had there and drop the Discord bit and add in HTTP, you can see request, generation, response.

Easy. Next up, Alpa. I love using GitHub Copilot, but it's only available in my code editor, and it requires internet connection. Alpa is my attempt to solve this. It is autocomplete anywhere on your system just by taking what's left of your cursor and, uh, having, passing it to a model to type in.

And of course, you can use any model, including a model fine-tuned in your own writing. Ask me how I know. [laughs]

Alpa is also quite simple. In fact, it's so simple I don't really need to cover it. Listen for input, copy the, uh, the input, um, into a prompt, start generating, type out response.

Easy. Now, the first two examples were pretty simple. They're proofs of concept. But now I want to talk about an actual use case. This is a real world data extraction task.

Over the last few years, I've been working on a project to make a timeline from dates on Wikipedia because there are millions of pages and they all have dates, and you can build a world history from it.

However, these dates are often unstructured and more or less impossible to parse using traditional means. Like, yes, you can try using regex to extract the dates, but you can't get the context out in any meaningful sense, and there are some dates here that just don't make any sense at all.

So that's why, as is the theme of this conference, I threw a large language model at it. However, GPT-3 and 4 aren't perfect, even after rounds of prompt engineering.

You can see I tried here. And handling millions of dates is just too expensive and slow. So I decided I'd fine-tune my own model. I generated a representative data set using GPT-3, built a tool, uh, tool to go through the data set.

So pick out any data point, fix it up, and then correct the errors, build a new data set, and train a new model.

So we did that using Axolotl, which I mentioned earlier. Again, check out Axolotl for all your fine-tuning needs. Highly recommended. And now I have a small, fast, consistent model that can pass any data to, uh, sorry, any date to, and get back a structured, uh, representation, which I can then of course immediately parse using Rust.

And I can treat that as a black box. So I have a function there, fn parse, pass some dates, get some dates back. Simple. Now, let's quickly talk about the benefits of using local models and the library.

First off, deployments. Show of hands, who's had to deal with Python deployment hell?

Dependency hell even. Yeah, yeah, I know. It's, it's awful. You spend hours just trying to sort out your, your Conda, your PIP, your PIPenv. It's awful. With the library, you inherit Rust's excellent cross-platform support and build system, making it easy to ship self-enclosed support, uh, binaries to your platform.

No more making your users install Torch. As you might imagine, this unlocks use of desktop applications with models.

Next up, the ecosystem. Rust has one of the strongest ecosystems of, uh, of any native language. You can combine these libraries with LLMs to build all kinds of things.

It's what let me build a Discord bot, a system auto-completion utility, a data ingestion pipeline with a data set, a utility explorer, all in the same language. And I think if you use llm-rs, you can do the same thing with your, uh, task as well.

Of course, you also have control over how, uh, the model generates. I alluded to this earlier, but you can choose exactly how it samples tokens. Normally, when you use a cloud model, you have to get back the, uh, logits, the l- uh, probabilities.

But those probabilities are limited. Like, you have to keep going back and forth, and that's slow and expensive. With this, you can directly control what you are sampling.

Finally, let's talk about the innovation in the space. If you're here, you probably know there's a paper almost every single day. It's impossible to keep up with. Trust me, I've tried.

But it mean, but the use of local models means you can try this out before anyone else can. You can go through. You can try out some of these papers and be like, "Oh, wow, that's actually a worthwhile improvement."

And eventually, the cloud providers will provide them, but in the meantime, the control remains with you.

However, it's time to talk about the problems. There ain't no such thing as a free lunch, except if you're at a conference, of course. [laughs]

Let's talk about hardware again. I mentioned earlier that you can pretty much run any, uh, these things on almost any hardware, but that's kind of a lie. You still need some kind of power.

You c- you can only get so much out of your 10-year-old computer, your smartphone, or your Raspberry Pi. We're finding clever ways to improve this, like smaller models and better inferencing, but it's still something to be aware of.

Next, as with all things, the fast, cheap, good triad applies. You can, uh, make all kinds of trade-offs here, and you can see I've listed a couple of them here.

But fundamentally, you have to choose what are you willing to sacrifice in or- in order to serve your a- application. Are you willing to go for a bigger model to get better quality results at the cost of speed?

These are all decisions you have to make, and they're not always obvious. It's something you have to think about.

Next, there's no other way of putting this. The ecosystem churns. Innovation is a double-edged sword. When those changes come in, they can often break your existing workflows. I've helped alleviate this w- to some extent using the GGUF, uh, file format which helps standardize, but it's still a problem.

Some days you will just wake up, try your application with a new model, and it just won't work. There's nothing you can do except deal with it. Finally, a lot of the models in this space are open source.

They're free for you to use personally, but they have very strange clauses and exceptions. For most of us, this doesn't matter. You can just use the model personally. But it's a reminder that even though that these models are free, they're not capital F Free.

Luckily, there's been some recent change in this space with Mistral and StableLM giving you strong performance for small level, uh, sorry, a small size and being completely unburdened. But it's still a problem, and they're still, uh, much smaller than the big ones like Llama and Falcon.

Unfortunately, I've got to wrap things up here. There's only so much you can talk about in 18 minutes, I'm afraid. Local models are great, and I like, I'd like to think our library is too.

They're getting easier to run day by day with smaller, more powerful models. However, the situation isn't perfect, and there isn't always one obvious solution for your problem.

Thanks for listening. You can contact me by email or via Mastodon. The library can be found at, you guessed it, llm.rs or by scanning the QR code. Finally, we're always looking for contributors.

If you're interested in LLMs or Rust, feel free to reach out. Sponsorships are also very welcome because they help me try out new hardware, which is always necessary. And if you want to chat in person, I'll be hanging around the conference.

See you later. [audience cheering] [upbeat music]