# Harnessing the Power of LLMs Locally

Mithun Hunsur, Ambient

AI Engineer Summit 2023

[Talk page](https://ai.engineer/talks/MwqUYRQloGw-harnessing-power-llms-locally) · [Video](https://www.youtube.com/watch?v=MwqUYRQloGw)

## Harnessing the Power of LLMs Locally

Local inference puts model selection, session state, sampling, and deployment inside your application. Mithun Hunsur’s llm.rs examples show what that control enables—and what it costs.

Basic familiarity with language-model prompts and Rust functions, closures, and threads will help with the implementation examples.

### What changes when the model runs on your computer?

[0:15](https://www.youtube.com/watch?v=MwqUYRQloGw&t=15s)

What changes when an application runs its own language model instead of sending prompts to a hosted service? With **ChatGPT**, **Claude**, or **Bard**, the provider operates the model. With local inference, the model runs on your computer, giving your application control over the hardware, inference process, and data it sends into that process. Mithun Hunsur, introduces this approach through [llm.rs](https://github.com/rustformers/llm), the Rust inference library he maintains alongside his work building a game engine at Ambient. The recording describes the historical library; its repository is now archived and unmaintained.

### Model capacity meets available hardware

[1:11](https://www.youtube.com/watch?v=MwqUYRQloGw&t=71s)

The first constraint is model size. Hunsur uses parameter count as a rough proxy for capacity: the hosted models in his comparison—**GPT-3**, **GPT-4**, **Gopher**, and **PaLM 2**—occupy the large end of the chart, while **Llama** and **Falcon** represent the growing local-model ecosystem. Size is not a task-quality guarantee, however. A smaller model focused on one problem may solve it better than a larger general model. The chart’s GPT-4 size is explicitly speculative; Hunsur says its actual size is unknown outside OpenAI.

![Slide titled “Differences — Size” with a bar chart labeled “Parameter count (billions)” and a Local/Remote legend.](https://ai.engineer/api/talk-essay-media/a2f3235c4202fcfbf64d933c83489c9c49a6edb0f1fdc8dbed860b6ebc58040d)

Local and remote models compared by parameter count.

Hardware introduces a separate limit. Hosted inference uses specialized infrastructure, while local inference uses whatever equipment you can obtain, including rented machines. More capable infrastructure can improve generation speed, support more simultaneous inference, or both. Hunsur’s illustrative hardware-cost axis runs from a few hundred dollars to a few hundred million dollars. This is a comparison of infrastructure scale, not a measured price-performance benchmark.

### Latency, cost, customization, and privacy

[2:24](https://www.youtube.com/watch?v=MwqUYRQloGw&t=144s)

Latency is not just the speed at which a model generates tokens. In the hosted request pattern Hunsur describes, the application submits a completed prompt and waits for the network exchange. A local application can feed input into the model incrementally as it arrives. For conversation, that means processing what someone is saying while they are still saying it. Removing the network trip and overlapping prompt processing with input can reduce waiting, although inference itself still takes time.

The economic comparison is similarly workload-dependent:

| Question | Hosted inference | Local inference |
|---|---|---|
| What determines the bill? | Provider’s per-token price | Cost of operating the machine |
| Who supplies the hardware? | Provider | You, or a hardware rental service |
| When can local be cheaper? | Compare the workload’s API bill | When local operating cost is lower |

An existing computer reduces additional acquisition cost; it does not make electricity, machine time, or operation literally free. The relevant comparison is the cost of serving your workload, not simply whether you already own a laptop.

Model choice adds another reason to run locally. Hosted applications select from a provider’s catalog, and fine-tuning may introduce both training fees and higher usage charges. Local applications can choose custom models for retrieval, storytelling, conversation, or tool use, then fine-tune further when necessary. Hunsur recommends [Axolotl](https://github.com/axolotl-ai-cloud/axolotl) for that training work. Finally, local inference can keep private prompts on the machine instead of submitting them to a cloud model—a practical benefit even when the question is merely embarrassing.

### Fitting billions of parameters into memory

[3:58](https://www.youtube.com/watch?v=MwqUYRQloGw&t=238s)

Owning the inference process only helps if the model fits the available hardware. Billions of parameters mean billions of stored numerical values, and consumer machines have limited memory. **Quantization** addresses this by representing parameters at lower precision: a lossy compression that reduces the model’s memory requirement while preserving much of its useful capability.

The benefit extends beyond storing a smaller file. A smaller representation can let the computer process more of the model at a time, potentially improving inference speed. The trade-off is information loss, so reduced size and retained quality have to be considered together. Hunsur presents the mechanism rather than a particular compression ratio or speed benchmark.

### From a Rust rewrite to a reusable library

[4:52](https://www.youtube.com/watch?v=MwqUYRQloGw&t=292s)

The project began with Hunsur encountering llama.cpp in March. The repository snapshots he presents contrast 98 stars early on with roughly 42,000 at presentation time. His response was to rewrite it in Rust, partly for the familiar programming joke and partly because he wanted to reuse inference in other applications. Another developer finished a parallel implementation first. They merged their projects, and Hunsur became maintainer of the resulting llm library.

Why build another implementation? Hunsur describes the March version of llama.cpp as an application rather than a reusable library. His design starts with embedding inference in other software and develops six requirements:

- **Library first:** expose inference as functionality an application can call.
- **Customizable behavior:** avoid coupling the library to one application or assuming how it will be used.
- **Multiple architectures:** support more than one model family.
- **Idiomatic Rust:** make the interface feel native to Rust, rather than like foreign-language bindings.
- **Multiple backends:** allow CPU, GPU, and future compute implementations.
- **Portable deployment:** keep application behavior consistent across Windows, Linux, macOS, and other platforms.

Together, these requirements make inference a component whose behavior the application can shape.

![Library slide with llm.rs surrounded by customizable, library, models, rust-native, platforms, and backends.](https://ai.engineer/api/talk-essay-media/0b084303fb6795f1fcc56e34dbf800f725e0a47a1bbe05f8e623df65130927bb)

The llm.rs library’s six design themes.

The supported architectures, including Llama and Falcon, share an interface. Switching architectures therefore does not require rewriting the application’s integration around a completely different API. Hunsur credits Lucas, Dan, and the wider contributor community for making that breadth possible.

### Load a model, then keep state in sessions

[7:18](https://www.youtube.com/watch?v=MwqUYRQloGw&t=438s)

The demonstrated API separates the loaded model from an **inference session**, which tracks one ongoing use of that model. This is the historical rustformers API, not the provider interface now documented under the latest `llm` crate. Its application flow is straightforward:

1. Load the model.
2. Create a session for an ongoing interaction.
3. Pass a prompt into that session and run inference.
4. Receive generated tokens through a callback.

You can create multiple sessions, but each carries a memory cost. Keeping a session alive lets a conversation continue without feeding its previous context back into the model on every turn. Session lifetime therefore affects both memory consumption and conversational behavior.

The inference function is a convenience helper: it repeatedly calls the model until its boundary conditions say to stop. An application that needs different control flow can replace that loop. Likewise, the sample’s repeated `Default` values are customization points for model loading, inference, and sampling. **The defaults provide a short path to inference without making the default behavior mandatory.**

Hunsur then demonstrates **Llama 7B** running on his MacBook CPU before showing GPU acceleration. The GPU version makes the interaction visibly more usable in his presentation; he also reports faster performance on NVIDIA GPUs. These are qualitative comparisons, with no tokens-per-second result or complete benchmark configuration. AMD and Intel support is described as pending at the time of the demonstration.

### Three ways to package local inference

[8:54](https://www.youtube.com/watch?v=MwqUYRQloGw&t=534s)

The community projects show how the same inference engine fits different kinds of software:

- **[local.ai](https://github.com/louisgv/local.ai):** louisgv’s desktop application makes local inference available through a downloadable app, without requiring the user to build an integration.
- **llm-chain:** a Rust workflow library that Hunsur describes as LangChain for Rust, with llm.rs as an inference option.
- **Floneum:** a flowchart-based application in which users connect nodes into workflows, using the library as an inference engine.

The reusable part is model execution; the surrounding product can be a simple app, a programmatic workflow, or a visual editor.

### Separating generation from response delivery

[9:36](https://www.youtube.com/watch?v=MwqUYRQloGw&t=576s)

Hunsur’s first proof of concept, [llmcord](https://github.com/rustformers/llmcord), puts the library behind a Discord bot. A user sends a prompt and receives generated text. Hunsur attributes the visible hitches in the demonstration to Discord’s limits rather than inference. Underneath the chat interface, a generation thread and a channel connect the request to a response task responsible for sending output to the user.

![Flow diagram connecting User request to a Response channel above a Generation thread, then to a Response task and two response messages.](https://ai.engineer/api/talk-essay-media/47d7ab39dab210aafb5278aa936933c5734ea907ef87a23306f9c06dbee4e1b9)

The llmcord request and response pipeline.

The separation can be expressed directly in Rust. This adapter accepts a generation function that emits text chunks and a response function that delivers them. The channel carries output between the generation thread and the response side:

```rust
use std::sync::mpsc::{self, Sender};
use std::thread;

pub fn stream_response<G, R>(generate: G, mut respond: R)
where
    G: FnOnce(Sender<String>) + Send + 'static,
    R: FnMut(String),
{
    let (output, incoming) = mpsc::channel::<String>();
    let generation = thread::spawn(move || generate(output));

    for chunk in incoming {
        respond(chunk);
    }

    generation.join().expect("generation thread panicked");
}
```

An inference callback supplies chunks to `output`; `respond` handles delivery. Once generation finishes and its sender is dropped, the receiver loop ends. Keeping those responsibilities separate lets the response side accommodate a transport’s delivery behavior without putting that behavior inside model generation.

The bot creates and discards a session for each query. Retaining a session instead would let later requests continue the conversation. The transport is another replaceable part: remove Discord, put HTTP at the boundary, and the core flow remains request, generation, response. The session policy and the transport policy are independent application decisions.

### Autocomplete beyond the code editor

[10:45](https://www.youtube.com/watch?v=MwqUYRQloGw&t=645s)

Alpa, Hunsur’s second proof of concept, starts with a limitation he encounters using GitHub Copilot: completion is tied to his editor and requires an internet connection. Alpa takes text to the left of the cursor and uses a local model to continue it elsewhere in the operating system. Model selection remains open, including a model fine-tuned on the user’s own writing.

The application loop is small:

1. Listen for input.
2. Copy the relevant text into a prompt.
3. Generate a continuation with the model.
4. Type the response into the application.

The inference engine supplies text; the operating-system integration decides where that text comes from and where the completion goes.

### Turning Wikipedia dates into structured data

[11:20](https://www.youtube.com/watch?v=MwqUYRQloGw&t=680s)

The third application moves from proofs of concept to a data-extraction workload. Hunsur wants to build a world-history timeline from dates scattered across Wikipedia. The scale is millions of pages, but the harder problem is interpretation: date expressions are unstructured, and finding a date-like substring does not recover the context needed to understand it. Regular expressions can locate patterns without resolving what those patterns mean.

He first tries GPT-3 and GPT-4. Even after prompt engineering, the results remain imperfect, and processing millions of dates is too slow and expensive for his workload. That leads to a specialized model and a data preparation loop:

1. Generate a representative dataset with GPT-3.
2. Inspect examples in a custom tool.
3. Correct erroneous data points and assemble the corrected dataset.
4. Fine-tune a model with Axolotl.

The correction step matters: generated examples are material to review and repair before training.

The resulting model accepts a date expression and returns a structured representation that Rust can parse. Hunsur describes it as small, fast, and consistent, without giving measured accuracy or throughput. Once the inference and output parsing are wrapped behind an `fn parse`-style interface, the rest of the ingestion pipeline can treat the model as a component: pass dates in, receive structured dates back. **Specialization turns a broad language-model capability into a narrow application interface.**

### Shipping inference and controlling its behavior

[12:50](https://www.youtube.com/watch?v=MwqUYRQloGw&t=770s)

Embedding inference in Rust also changes distribution. Hunsur contrasts managing Python environments through conda, pip, and Pipenv with using Rust’s build system and platform support to ship self-contained application binaries. Users need not install Torch to run these applications, making local models easier to incorporate into desktop software.

The surrounding ecosystem matters as much as the model interface. Rust libraries let Hunsur build Discord integration, operating-system autocomplete, a data-ingestion pipeline, and tools for exploring and correcting datasets in the same language. Inference becomes another library in that application, rather than a separate service around which the entire program must be organized.

Control also extends to **token sampling**. Hunsur contrasts direct access to local generation with hosted interfaces that expose limited probability information and require repeated network exchanges to influence the process. Owning the inference loop lets application code decide how to sample at the point where the next token is selected.

That access creates room to experiment with new research before a hosted provider offers equivalent functionality. A developer can implement a technique, try it against the application’s problem, and decide whether it is an improvement. The opportunity is earlier experimentation—not an assurance that every new paper will help.

### Hardware, compatibility, and licensing still constrain the application

[14:29](https://www.youtube.com/watch?v=MwqUYRQloGw&t=869s)

Local inference does not remove hardware requirements. Hunsur explicitly qualifies the earlier suggestion that models can run on almost anything: an old computer, smartphone, or Raspberry Pi has limits that smaller models and better inference implementations can ease but not erase. The application still has to choose among speed, cost, and quality. A larger model may improve results while making the interaction slower; whether that is acceptable depends on the task.

Rapid innovation also creates compatibility churn. A new model can break a previously working workflow. Hunsur points to his work on **GGUF** standardization as one way to reduce that friction, not eliminate it. Format standardization should not be confused with support in this particular library: the archived repository’s main branch lacks GGUF support, while its experimental GGUF branch has limited Llama support and lacks quantization.

Licensing is a separate constraint. Downloadable weights and permission for personal use do not establish unrestricted rights; model licenses can contain consequential clauses and exceptions. Hunsur points to **[Mistral 7B](https://mistral.ai/news/announcing-mistral-7b/)** and **[StableLM](https://github.com/Stability-AI/StableLM)** as encouraging developments in smaller models, compared with the larger Llama and Falcon models in his discussion. Their licenses still need to be distinguished: Mistral 7B was released under Apache 2.0, while StableLM terms vary by checkpoint, including attribution/share-alike and noncommercial terms. The model family’s name alone is not enough to determine whether a release fits an application.

### More accessible models, application-specific choices

[16:20](https://www.youtube.com/watch?v=MwqUYRQloGw&t=980s)

Smaller, more capable models make local inference easier to adopt, and a reusable library makes that capability easier to embed. Neither development produces one obvious answer for every problem. The practical choice remains tied to the application: what it needs the model to do, how quickly it must respond, and which constraints its deployment can accept.

![Conclusion slide describing local LLMs as great, powerful, and controllable, saying llm.rs makes them easier to use, and ending with “but not perfect.”](https://ai.engineer/api/talk-essay-media/92c640616da5182a0b51a86d8bd62edd906e73577a43621d98cbdb3ffe04125b)

Local LLMs are becoming easier to run, but remain imperfect.

Hunsur closes by directing viewers to llm.rs and inviting contributions from people interested in Rust or language models. Sponsorship would help test additional hardware—a concrete requirement for a project aiming to support diverse machines. He offers email, Mastodon, and conversation at the conference as ways to get involved.

## Resources

- [llm: the historical Rust inference library](https://github.com/rustformers/llm) — Archived source for the library in the talk, including CLI examples, session persistence and model-format limitations.
- [local.ai](https://github.com/louisgv/local.ai) — Desktop application built around llm, with model downloads, notes and a streaming inference server.
- [llmcord](https://github.com/rustformers/llmcord) — Archived Rust Discord bot demonstrating integration with the llm inference library.
- [Axolotl](https://github.com/axolotl-ai-cloud/axolotl) — Fine-tuning framework with dataset preparation, configuration examples and training instructions. The repository documents its current capabilities.
- [Mistral 7B release announcement](https://mistral.ai/news/announcing-mistral-7b/) — The September 2023 announcement describes the 7.3B model, Apache 2.0 licensing and the team's benchmark methodology.
- [StableLM models and licenses](https://github.com/Stability-AI/StableLM) — Model-family documentation distinguishing base-model, tuned-model and source-code licenses.

## Slides

### What is llm.rs?

llm.rs is a Rust library for local LLM inference, where you own the model and run it on your computer.

[0:40](https://www.youtube.com/watch?v=MwqUYRQloGw&t=40s) [Slide describes llm.rs as all-in-one local inferencing.](https://assets.ai.engineer/sites/aiecode2025/sha256/ede12970c6d82f55497925fc147c9f70ac558c118ee34f5b459be99dacf3d18f.webp)

What?

llm.rs
all-in-one local inferencing

### Local models

Local models let you own the model and run it on your computer.

[1:01](https://www.youtube.com/watch?v=MwqUYRQloGw&t=61s) [Section slide titled Local models.](https://assets.ai.engineer/sites/aiecode2025/sha256/3cdef58ae4ddc2483e8a2227a899ac16719ace65f9ea86cb4b19b6812d467964.webp)

Local models

### Differences: Size

Remote models dominate the chart’s high end, though Falcon 180B is a large local model. The speaker cautions that GPT-4’s size is unknown and its charted value is based on rumors.

[1:12](https://www.youtube.com/watch?v=MwqUYRQloGw&t=72s) [Bar chart compares parameter counts for local and remote models; GPT-4 is marked with an asterisk.](https://assets.ai.engineer/sites/aiecode2025/sha256/ca22254c245c4498e6eb3f4fc0d487d2aed56284927f17d1e6189a33f46b85df.webp)

Differences
Size

Parameter count (billions)
Legend: Local, Remote
Y-axis ticks: 0, 200, 400, 600
Models: BERT, GPT-2, GPT-J, Llama 2 7B, Mistral 7B, Llama 2 13B, GPT-NeoX, Falcon 40B, Llama 2 70B, GPT-3, Falcon 180B, GPT-4*, Gopher, PaLM 2, M-T NLG

[1:34](https://www.youtube.com/watch?v=MwqUYRQloGw&t=94s) [Bar chart comparing parameter counts of local and remote language models.](https://assets.ai.engineer/sites/aiecode2025/sha256/c409ab9f7f2e81b3396f35551ccad7fd4c20d9a981e79cb8db780ceeb4f5a1ac.webp)

Differences
Size

Parameter count (billions)
Legend: Local, Remote
Y-axis ticks: 0, 200, 400, 600
Models: BERT, GPT-2, GPT-J, Llama 2 7B, Mistral 7B, Llama 2 13B, GPT-NeoX, Falcon 40B, Llama 2 70B, GPT-3, Falcon 180B, GPT-4*, Gopher, PaLM 2, M-T NLG

### Differences: Speed and Capacity

Hardware ranges from consumer processors and GPUs to cloud-scale systems. Moving right brings more speed and/or parallel inference, with costs ranging from a few hundred to a few hundred million dollars.

[2:01](https://www.youtube.com/watch?v=MwqUYRQloGw&t=121s) [Computing hardware ranges from consumer processors and graphics cards to cloud-scale infrastructure.](https://assets.ai.engineer/sites/aiecode2025/sha256/0acffe9aee80aaf9a03e82f09dc7403eb0b23114f9f1590107a0f73b0df47202.webp)

Differences
Speed/Capacity

AMD
RYZEN 5
5000 SERIES PROCESSOR
ASUS

Mere mortals
AI companies/cloud vendors

### Differences: Latency

The speaker contrasts local inference, which can process a prompt as it arrives, with a cloud round trip that waits for the full prompt before returning the next word.

[2:31](https://www.youtube.com/watch?v=MwqUYRQloGw&t=151s) [Diagram compares local and cloud paths for producing the next word over time.](https://assets.ai.engineer/sites/aiecode2025/sha256/9c26959c7cbf0217b75a022647fff2ea87358c425881e0d9b9ff3a80989ec9e4.webp)

Differences — Latency

Cloud
The quick brown fox jumps over
the

The quick brown fox jumps over
the
the

Time

### Differences: Cost

Cloud vendors charge per token; local costs depend on running the machine. The speaker compares that cost with sending the workload to the cloud.

[2:42](https://www.youtube.com/watch?v=MwqUYRQloGw&t=162s) [Per-token pricing versus running cost.](https://assets.ai.engineer/sites/aiecode2025/sha256/c4a1a3a2ec9140b02e7a5b77988e64c4409311619ee11f123c760e7b12d2104a.webp)

Differences — Cost

Per-token pricing
vs
Running cost

### Differences: Models

Cloud users choose from vendor offerings; local users can choose from many custom models or fine-tune an existing one. The speaker highlights Axolotl as a fine-tuning tool.

[3:09](https://www.youtube.com/watch?v=MwqUYRQloGw&t=189s) [A list of language models beside an axolotl illustration.](https://assets.ai.engineer/sites/aiecode2025/sha256/ff36481c4cd46246e4b0f35a1b6746b8c73891a94e775d3e6d5435b5e9190f54.webp)

Differences: Models

axolotl

And many more...

Interface text:
Models 1840
Collapse
Sort: Most Downloads

TheBloke/Wizard-Vicuna-7B-Uncensored-GPTQ
Text Generation • Updated 7 days ago • 383k • 84
TheBloke/vicuna-7B-v1.3-GPTQ
Text Generation • Updated Aug 21 • 224k • 10
TheBloke/MythoMax-L2-13B-GPTQ
Text Generation • Updated 7 days ago • 172k • 73
TheBloke/Llama-2-13B-chat-GPTQ
Text Generation • Updated 7 days ago • 105k • 273
TheBloke/Llama-2-7b-Chat-GPTQ
Text Generation • Updated 7 days ago • 83.5k • 148
TheBloke/Llama-2-70B-chat-GPTQ
Text Generation • Updated 7 days ago • 38.3k • 196
TheBloke/WizardCoder-Python-13B-V1.0-GPTQ
Text Generation • Updated 7 days ago • 35.9k • 56
TheBloke/falcon-40b-instruct-GPTQ
Text Generation • Updated Aug 21 • 30.5k • 193
TheBloke/OpenAssistant-Llama2-13B-Orca-v2-8K-...
Text Generation • Updated 7 days ago • 22.5k • 17
TheBloke/Falcon-180B-Chat-GPTQ
Text Generation • Updated 7 days ago • 22.3k • 56
TheBloke/gpt4-alpaca-lora_mlp-65B-GPTQ
Text2Text Generation • Updated Aug 21 • 19k • 14
TheBloke/vicuna-7B-v1.5-GPTQ
Text Generation • Updated 7 days ago • 17.5k • 9
TheBloke/Llama-2-7B-GPTQ
Text Generation • Updated 7 days ago • 16.2k • 55
TheBloke/WizardLM-Uncensored-Falcon-7B-GPTQ
Text Generation • Updated Aug 21 • 15.9k • 42

### Quantization

The speaker introduces quantization as a way to run models with billions of parameters on consumer hardware with limited memory.

[3:51](https://www.youtube.com/watch?v=MwqUYRQloGw&t=231s) [The word Quantization centered on a red background.](https://assets.ai.engineer/sites/aiecode2025/sha256/7ae4d09e7f70bc2d339e25b22477bea2306f17e61506e2c380cdb30ff31d04b6.webp)

Quantization

### Quantization: Processor-Memory Performance Gap

From 1980 to 2010, processor performance rises much faster than memory performance. The speaker introduces quantization as a way to run large models on memory-limited consumer hardware.

[4:07](https://www.youtube.com/watch?v=MwqUYRQloGw&t=247s) [A graph shows processor performance rising faster than memory performance from 1980 to 2010.](https://assets.ai.engineer/sites/aiecode2025/sha256/1338e07f660d7a1fe72d722210c50d1719d8e182993547caceab18e685e0cd2e.webp)

How? Quantization

Processor-Memory Performance Gap
Processor
Memory

Y-axis: Performance
Y-axis ticks: 1, 10, 100, 1,000, 10,000, 100,000
X-axis: Year
Timeline: 1980, 1985, 1990, 1995, 2000, 2005, 2010

### How? Quantization

Quantization lossily compresses a model: the slide shows a 30GB float32 model reduced to 15GB in float16, with smaller formats ranging from 3.52GB to 5.13GB. The speaker says smaller models can also run faster.

[4:16](https://www.youtube.com/watch?v=MwqUYRQloGw&t=256s) [A quantization hammer above a llama, alongside a diagram comparing floating-point and compressed model sizes.](https://assets.ai.engineer/sites/aiecode2025/sha256/b3c1ba2c8c9656fa4fb8a09a87ca04300e9fd5bc956b16bbec2f13c6f33df584.webp)

How? Quantization

quantization

float32
30GB

float16
15GB

gptq
4.16GB

awq
4.15GB

ggml 3-bit
3.52GB

ggml 5-bit
5.13GB

### llm.rs

The talk turns from local models to the llm.rs library.

[4:44](https://www.youtube.com/watch?v=MwqUYRQloGw&t=284s) [The text llm.rs centered on a red slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/9db9fca6715093aef2fdf009aabd7877f6e77f056b80798def337bb5b07fea3f.webp)

llm.rs

### When? llama.cpp at the start

On day one, llama.cpp had 98 stars and aimed to run LLaMA on a MacBook using 4-bit quantization.

[4:56](https://www.youtube.com/watch?v=MwqUYRQloGw&t=296s) [An early llama.cpp repository page showing 98 stars.](https://assets.ai.engineer/sites/aiecode2025/sha256/c198ae69503ad5c3f4cc22c26444176f278c0e8b1984e4e1083f99b183f1d40b.webp)

When? llama.cpp

llama.cpp
Inference of Facebook's LLaMA model in pure C/C++

Description
The main goal is to run the model using 4-bit quantization on a MacBook.

• Plain C/C++ implementation without dependencies
• Apple silicon first-class citizen - optimized via Arm Neon and Accelerate framework
• [illegible]

Interface text:
ggerganov / llama.cpp · Public
Notifications · Fork 2 · Star 98
Code · Issues 1 · Pull requests · Actions · Projects · Security · Insights
master · 1 branch · 0 tags · Go to file · Code
ggerganov: Fix a bug in the rope calculation · 3 hours ago · 5 commits
models · Final touches · 5 hours ago
.gitignore · Initial release · 5 hours ago
Makefile · Initial release · 5 hours ago
README.md · Update README.md · 4 hours ago
convert-pth-to-ggm... · Fix a bug in the rope calculation · 3 hours ago
ggml.c · Initial release · 5 hours ago
ggml.h · Initial release · 5 hours ago
main.cpp · Fix a bug in the rope calculation · 3 hours ago
quantize.cpp · Initial release · 5 hours ago
utils.cpp · Fix a bug in the rope calculation · 3 hours ago
utils.h · Fix a bug in the rope calculation · 3 hours ago

About
Port of Facebook's LLaMA model in C/C++
Readme · 98 stars · 2 watching · 2 forks
Releases · No releases published
Packages · No packages published
Languages · C 84.3% · C++ 13.5% · Makefile 1.3% · Python 0.9%

### When? llama.cpp later

The speaker contrasts llama.cpp’s 98 stars on day one with 41.6k in the later repository view.

[5:01](https://www.youtube.com/watch?v=MwqUYRQloGw&t=301s) [A later llama.cpp repository page showing 41.6 thousand stars.](https://assets.ai.engineer/sites/aiecode2025/sha256/163a51332f416534d202a9255dfc118e978360fd48796db3660c02ba07481b26.webp)

When?
llama.cpp

Interface text:
llama.cpp Public
Watch 433
Fork 5.9k
Starred 41.6k
master
60 branches
836 tags
Go to file
Add file
Code

AhmadTameem ggml : add RISC-V Vector Suppor... [illegible] 9 hours ago
1,317 commits
.devops — ci : Cloud-V for RISC-V builds (#3160) — 3 weeks ago
.github — cmake : make LLAMA_NATIVE flag actually use the in... — 11 hours ago
ci — ci : add LoRA test to CI (#2650) — 2 months ago
common — Work on the BPE tokenizer (#3252) — 20 hours ago
docs — docs : mark code as Bash (#3375) — last week
examples — main : consistent prefix/suffix coloring (#3425) — 9 hours ago
gguf-py — gguf : add BERT, MPT, and GPT-J arch info (#3408) — yesterday
grammars — speculative : add grammar support (#2991) — last month
media — media : add logos and banners — 6 months ago
models — Work on the BPE tokenizer (#3252) — 20 hours ago
pocs — build : enable more non-default compiler warnings (#... — 5 days ago
prompts — feature : support Baichuan serial models (#3009) — 3 weeks ago
scripts — cmake : fix transient definitions in find pkg (#3411) — 2 days ago
spm-headers — swift : Package compile breaks due to ggml-metal.m... — 4 months ago
tests — Work on the BPE tokenizer (#3252) — 20 hours ago
.clang-tidy — fix some warnings from gcc and clang-tidy (#3038) — last month
.dockerignore — docker : ignore Git files (#3314) — 2 days ago
.ecrc — Fix whitespace, add .editorconfig, add GitHub workfl... — 6 months ago
.editorconfig — server : add a subtle loading animation to the edit bo... — last month
.flake8 — hooks : setting up flake8 and pre-commit hooks (#16... — 4 months ago
.gitignore — Work on the BPE tokenizer (#3252) — 20 hours ago
.pre-commit-config.yaml — hooks : setting up flake8 and pre-commit hooks (#16... — 4 months ago
CMakeLists.txt — cmake : make LLAMA_NATIVE flag actually use the in... — 11 hours ago
LICENSE — Add LICENSE (#21) — 7 months ago
Makefile — Work on the BPE tokenizer (#3252) — 20 hours ago
Package.swift — swift : fix build on xcode 15 (#3387) — 5 days ago
README.md — llama.cpp : add documentation about rope_freq_bas... — 4 days ago

About
Port of Facebook's LLaMA model in C/C++
Readme
MIT license
Activity
41.6k stars
433 watching
5.9k forks
Report repository

Releases 815
b1317 Latest
9 hours ago
+ 814 releases

Packages 1
llama.cpp

Used by 11
+ 3

Contributors 366
+ 355 contributors

Languages
C 41.0%
C++ 31.4%
Cuda 11.0%
Python 5.9%
Metal 3.5%
Objective-C 2.8%
Other 4.4%

### Early Rust projects

The speaker says llama-rs beat his ggllama effort to the same problem. The projects later merged into LLM, which he maintained.

[5:26](https://www.youtube.com/watch?v=MwqUYRQloGw&t=326s) [Two Rust project repositories side by side, with a “wasted” overlay on the left.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf437394761bf13d36f707cd8e4c8687bc5d122d22b27c91b17b920eff8f03d1.webp)

When?
llama.cpp

wasted

Interface text:
Left repository:
philpax / ggllama  Public
Code  Issues  Pull requests  Actions  Projects  Security
main  2 branches  0 tags  Code
philpax  Update README.md  00836d5  3 minutes ago  28 commits
vendor  chore: update to latest llama
.gitignore  feat: initial commit
.gitmodules  feat: initial commit
Cargo.toml  feat: write completely untest[illegible]
LICENSE  feat: add LICENSE
README.md  Update README.md

Right repository:
setzer22 / llama-rs  Public  Sponsor
Code  Issues  Pull requests  Actions  Projects  Security
main  1 branch  0 tags  Go to file  Code
setzer22  Link profile  67855af  22 minutes ago  21 commits
.github  Add FUNDING.yml  1 hour ago
doc/resources  Update image  52 minutes ago
ggml-raw  It works, but there's something wrong in th...  17 hours ago
llama-rs  Restructure modules. Add clap
.gitignore  Hello world!
Cargo.lock  Restructure modules. Add clap
Cargo.toml  Hello world!
LICENSE  Update README and add licen[illegible]
README.md  Link profile

### The LLM project

Two developers merged their Rust projects to create LLM, and the speaker became its maintainer.

[5:31](https://www.youtube.com/watch?v=MwqUYRQloGw&t=331s) [A GitHub repository page describes an ecosystem of Rust libraries for large language models.](https://assets.ai.engineer/sites/aiecode2025/sha256/9abd10426a341e5be64248868ebcac2d8136925b885e30f9bc1007e21805851c.webp)

When?
llm

About
An ecosystem of Rust libraries for working with large language models

docs.rs/llm/latest/llm/

Interface text:
llm  Public
Edit Pins
Watch 45
Fork 278
Starred 5k
main
6 branches
1 tag
Go to file
Add file
Code
Your main branch isn't protected
Protect this branch from force pushing or deletion, or require status checks before merging. Learn more
Protect this branch
philpax Merge pull request #426 from Sanctum-...  84800b0  2 weeks ago  1,076 commits
.github/workflows  fix(ci): the docs state default toolchain and yet  3 months ago
.vscode  chore: fix some derps  5 months ago
binaries  map_or main.rs  3 weeks ago
crates  Merge pull request #426 from Sanctum-AI/m1-build-fix  2 weeks ago
doc  adjust doc  2 months ago
rust  ai  ml  llm  ggml
Readme
Apache-2.0, MIT licenses found
Activity
5k stars
45 watching
278 forks
Report repository

### LLM library principles

The speaker designed llm.rs as a customizable, Rust-native library, with models, backends, and platforms among its six principles. He wanted it to be reusable without coupling it to an application.

[5:39](https://www.youtube.com/watch?v=MwqUYRQloGw&t=339s) [LLM.rs surrounded by six principles: library, customizable, models, Rust-native, backends, and platforms.](https://assets.ai.engineer/sites/aiecode2025/sha256/6f20039b287a42e04ecde4e0cd50fad9a331479de9d8bf5ade6f18a841cf544d.webp)

Library

customizable
library
models
llm.rs
rust-native
platforms
backends

### Library overview

llm.rs aims to feel native to Rust and work across CPU and GPU backends and operating systems without significant code changes.

[6:41](https://www.youtube.com/watch?v=MwqUYRQloGw&t=401s) [The llm.rs library surrounded by customizable, library, models, Rust-native, platforms, and backends.](https://assets.ai.engineer/sites/aiecode2025/sha256/a903e6336f611f506f9fe97e97297143015fd8ab0ca8e485476892d814a84a71.webp)

Library

customizable
library
models
llm.rs
rust-native
platforms
backends

### Library architectures

llm.rs supports architectures including Llama and Falcon through the same interface, so users can switch models without changing their code.

[6:59](https://www.youtube.com/watch?v=MwqUYRQloGw&t=419s) [The llm.rs logo above two illustrations, alongside a list of model architectures from Llama to Bloom.](https://assets.ai.engineer/sites/aiecode2025/sha256/8efdff60898145b8f69b32303051f989bea7949bfe0f89955738346df5a784c3.webp)

Library
Architectures

llm.rs

llama
gpt-2
gpt-j
gpt-neox
falcon
mpt
bloom

### Library code example

The Rust example loads a model, starts a session, and runs inference on a prompt. A callback prints each token. Sessions can be reused for conversation, though each has a memory cost.

[7:16](https://www.youtube.com/watch?v=MwqUYRQloGw&t=436s) [Rust code loads a model, creates a session, and runs inference with a callback.](https://assets.ai.engineer/sites/aiecode2025/sha256/769803ffd3d91e84f597eae8b202b81f75cd9a8ca6b1b1687b9b4328e5638a89.webp)

Example

fn main() {
    let model_path = Path::new("llama-2-7b.gguf");
    let model = llm::load(model_path, llm::TokenizerSource::Embedded, Default::default(), llm::load_progress_callback_stdout).unwrap();

    let mut session = model.start_session(Default::default());
    session.infer::<Infallible>(
        model.as_ref(),
        &mut rand::thread_rng(),
        &llm::InferenceRequest {
            prompt: "Hello, world!",
            parameters: &llm::InferenceParameters::default(),
            play_back_previous_tokens: false,
            maximum_token_count: None,
        },
        // OutputRequest
        &mut Default::default(),
        |r| match r {
            llm::InferenceResponse::PromptToken(t) | llm::InferenceResponse::InferredToken(t) => {
                print!("{t}");
                std::io::stdout().flush().unwrap();
                Ok(llm::InferenceFeedback::Continue)
            }
            _ => Ok(llm::InferenceFeedback::Continue),
        },
    );
}

## Full transcript

[0:00](https://www.youtube.com/watch?v=MwqUYRQloGw&t=0s) [upbeat music] Right. Good day, everyone.

[0:15](https://www.youtube.com/watch?v=MwqUYRQloGw&t=15s) Good to see you all. Today, I'm here to tell you to har- how to harness the power of local LLMs using our Rust library. Quick intro. I'm Mithun, as you just heard, but I go by [REDACTED:username] online.

[0:27](https://www.youtube.com/watch?v=MwqUYRQloGw&t=27s) I hail from Australia, hence the accent, but I live in Sweden. I do a lot of things with computers, but my day job is at Ambient, where I build a game engine of the future.

[0:36](https://www.youtube.com/watch?v=MwqUYRQloGw&t=36s) Today though, I'm here to talk to you about llm.rs, a Rust library that I maintain.

[0:42](https://www.youtube.com/watch?v=MwqUYRQloGw&t=42s) So llm.rs or LLM between friends. I realized that I have to disambiguate when I started, when I signed this Simon's newsletter. [laughs] It's an all-in-one solution for local inference of LLMs, but what does that actually mean?

[0:55](https://www.youtube.com/watch?v=MwqUYRQloGw&t=55s) Well, most of the models we've discussed at this conference have been cloud models. You have ChatGPTs, your Claudes, your Bards. Local models offer another way, where you own the model and it runs on your computer.

[1:07](https://www.youtube.com/watch?v=MwqUYRQloGw&t=67s) So let's quickly go over what that actually means.

[1:11](https://www.youtube.com/watch?v=MwqUYRQloGw&t=71s) First up, size. Model size can be used as a rough proxy for the intelligence of the model. Most hosted models are really, really big. You can see that it's dominating the right-hand side of the chart there.

[1:24](https://www.youtube.com/watch?v=MwqUYRQloGw&t=84s) You have your GPT-3, your GPT-4, we'll get back to that. Your Gopher, your PaLM 2. These are all insanely big in comparison to the open source models we have.

[1:34](https://www.youtube.com/watch?v=MwqUYRQloGw&t=94s) We're gonna, we're beginning to see some, uh, bigger models thanks to, uh, Llama and Falcon, but even they pale in comparison to what the bigger players can do. This means the local models don't have the same capacity for intelligence.

[1:45](https://www.youtube.com/watch?v=MwqUYRQloGw&t=105s) However, a smaller, more focused model may be able to solve problems l- uh, better than a large general model. By the way, we don't actually know what size GPT-4 is.

[1:55](https://www.youtube.com/watch?v=MwqUYRQloGw&t=115s) That's rumors. Uh, only OpenAI knows. Next, let's talk about speed and capacity. Cloud models run specialized hardware with special configuration. Local models run on whatever hardware you can scrounge up, including rented hardware.

[2:11](https://www.youtube.com/watch?v=MwqUYRQloGw&t=131s) The further up the axis you go, the more speed and/or parallel inference you can do, but the more inaccessible it becomes. This end, a few hundred dollars. That end, a few hundred million dollars.

[2:24](https://www.youtube.com/watch?v=MwqUYRQloGw&t=144s) Next up, latency. Cloud models need the full prompt before they can start inference, and you have to wait for the message back, uh, back and forth. Local models can give you a response immediately.

[2:35](https://www.youtube.com/watch?v=MwqUYRQloGw&t=155s) You can feed the prompt as you go along. This is very important for conversations where you want the model to be able to process what you're saying as you say it.

[2:44](https://www.youtube.com/watch?v=MwqUYRQloGw&t=164s) And of course, you can't escape talking about cost. The cloud vendors will charge you a per token price. When running locally, it's entirely up to you how much it costs you to run the machine.

[2:55](https://www.youtube.com/watch?v=MwqUYRQloGw&t=175s) If the running cost of your, uh, model is less than the cost of running your workload through the cloud, you're going to make a profit, and if you're running on a machine you already own, well, that's basically free, right?

[3:06](https://www.youtube.com/watch?v=MwqUYRQloGw&t=186s) With the cloud, you have to offer, you have to use the models they offer you. Some vendors offer fine-tuning, but they often charge more than, uh, just using the regular model, and they often charge you for the pro- process of actually fine-tuning.

[3:18](https://www.youtube.com/watch?v=MwqUYRQloGw&t=198s) This means that it's not often cost-effective to, to actually do that. With local models, the sky's the limit. There are hundreds, potentially thousands of custom models that can suit any need you have.

[3:30](https://www.youtube.com/watch?v=MwqUYRQloGw&t=210s) Knowledge retrieval, storytelling, conversation, tool use. You name it, someone's probably already done it, and if they haven't, fine-tuning an existing model for your own use is easy enough. Special shout-out to Axolotl over there, which makes it easy to fine-tune, uh, models of any architecture.

[3:47](https://www.youtube.com/watch?v=MwqUYRQloGw&t=227s) And of course, privacy. [laughs] There are some questions you don't want to ask the internet. Local models let you privately embarrass yourself. [laughs]

[3:58](https://www.youtube.com/watch?v=MwqUYRQloGw&t=238s) Now, you might be wondering how it's actually possible to run these models locally. That, my friends, is possible with the power of quantization. If each model is billions of parameters, and those parameters are, are like individual numbers, how could you, how could you possibly run them on consumer hardware when there's only so much memory given for, uh,

[4:15](https://www.youtube.com/watch?v=MwqUYRQloGw&t=255s) available for a given, uh, performance level? Well, we can use quantization. Quantization lets you lossily compress a model while maintaining the majority of its smarts. We can take the original model, here in blue, and squish it down to something much smaller using one of these green formats.

[4:30](https://www.youtube.com/watch?v=MwqUYRQloGw&t=270s) This is a secret sauce that makes it, uh, viable to run models locally. Smaller models aren't easier to store, aren't just easier to store. They can also run faster as your process, uh, uh, as your computer can process more of the model at any given moment.

[4:44](https://www.youtube.com/watch?v=MwqUYRQloGw&t=284s) But that's enough about local models. You've probably already h-heard much, uh, much that already. Let's talk about the actual library.

[4:52](https://www.youtube.com/watch?v=MwqUYRQloGw&t=292s) It all started with this man who built something you may have heard of. Of course, I'm referring to llama.cpp, and that's what it looked like on day one.

[4:59](https://www.youtube.com/watch?v=MwqUYRQloGw&t=299s) Look at the mere 98 stars. How pedestrian compared to today, where it's forty-two thousand stars. Uh, but let's go back to March when I first saw it. When I saw it, I had but one idea.

[5:11](https://www.youtube.com/watch?v=MwqUYRQloGw&t=311s) It's time to rewrite it in Rust. [laughs] For both the meme and because I wanted to do use it for other things. Well, I wanted to say I... Well, I said I wanted to do it, and I did.

[5:21](https://www.youtube.com/watch?v=MwqUYRQloGw&t=321s) But to the right here, [REDACTED:username] was also working on the same problem, and well, there was just one catch. He beat me at... He beat me to it, completely beat me to it.

[5:30](https://www.youtube.com/watch?v=MwqUYRQloGw&t=330s) I'm not afraid to admit it. Luckily, we came together, merged our projects, and I ended up as the maintainer of the resulting project, and that's how LLM was born.

[5:41](https://www.youtube.com/watch?v=MwqUYRQloGw&t=341s) So you might be wondering, why? If llama.cpp exists, why use La- uh, llm.rs? Well, with llm.rs, I had six principles in mind. It must be a library. When I first started in March, llama.cpp was not a library.

[5:56](https://www.youtube.com/watch?v=MwqUYRQloGw&t=356s) It was an application, and that made it impossible to reuse. It must not be coupled to, to an application. You must be able to customize its behavior. You must be able to go in and change every little bit of it to make it work for your application, and we sh- we shouldn't make any assumptions about how it's,

[6:11](https://www.youtube.com/watch?v=MwqUYRQloGw&t=371s) it's going to be used.

[6:12](https://www.youtube.com/watch?v=MwqUYRQloGw&t=372s) Uh, it should support a multitude of model architectures. Of course, llama.cpp supports Llama and now Falcon, but clearly there are more out there. Next up, it should be Rust native.

[6:23](https://www.youtube.com/watch?v=MwqUYRQloGw&t=383s) It should feel like using a Rust library. It shouldn't feel like using a, a library with bindings, and it should feel, work how you expect a Rust library to work.

[6:30](https://www.youtube.com/watch?v=MwqUYRQloGw&t=390s) Next up, backends. It should support all other, uh, all possible kinds of backends, so you can run it on your CPU, your GPU, or of course, your ML-powered toaster.

[6:38](https://www.youtube.com/watch?v=MwqUYRQloGw&t=398s) I'm sure that's gonna be a thing. But we have, we've s- we're gonna see it coming, I'm s- I swear. And finally, platforms. It should work the same whether it's on Windows, Linux, macOS, or something else.

[6:49](https://www.youtube.com/watch?v=MwqUYRQloGw&t=409s) It shouldn't have... You shouldn't have to change it significantly to make it work, because deployment has always been an issue.

[6:56](https://www.youtube.com/watch?v=MwqUYRQloGw&t=416s) Today, I'm proud to say we support a myriad of architectures, including the, uh, the darlings of the movement, Llama and Falcon. These architectures all use the same interface, so you don't have to worry about changing your code to use a different model.

[7:08](https://www.youtube.com/watch?v=MwqUYRQloGw&t=428s) This is made possible by the coordint- coordi- [laughs] concerted efforts of my co-contributors Lucas and Dan, who I couldn't have done this without, as well, as well as many others.

[7:18](https://www.youtube.com/watch?v=MwqUYRQloGw&t=438s) Here's some sample code for the library. I won't go too much into it because it's quite dense, but the idea is that you load a model right there at the top.

[7:26](https://www.youtube.com/watch?v=MwqUYRQloGw&t=446s) You can see it's actually quite small. And with that model you create sessions which track an ongoing use of the model. You can have as many of these as you would like, but they do have a memory cost, so you want to be careful.

[7:35](https://www.youtube.com/watch?v=MwqUYRQloGw&t=455s) Once you have a session, you could pass, you can pass a prompt in and infer with the model to determine what comes next. You can k- keep reusing the same session, which is very useful for conversation.

[7:44](https://www.youtube.com/watch?v=MwqUYRQloGw&t=464s) You don't need to keep re-feeding the context. The last argument of the call, uh, of the function is the callback. That's where you actually get the tokens out. Um, it's worth noting that the function itself is actually a helper.

[7:57](https://www.youtube.com/watch?v=MwqUYRQloGw&t=477s) All it does is call the model in a loop with some boundary conditions. So if you wanna change the logic in some, uh, significant way, you can. We're not gonna stop you from doing that.

[8:06](https://www.youtube.com/watch?v=MwqUYRQloGw&t=486s) One last thing about this though. You see all the calls to default there? Those are all customization points. You can change pretty much anything about this. You can change how the model is loaded.

[8:15](https://www.youtube.com/watch?v=MwqUYRQloGw&t=495s) You can change how it'll do the inference. You can change how it'll sample. The entire point is you have the control you need to make the thing y- uh, you need to work.

[8:24](https://www.youtube.com/watch?v=MwqUYRQloGw&t=504s) Here's a quick demo of, uh, the library working with Llama 7, uh, billion on my MacBook CPU. It's reasonably fast, but it could be faster, right? Well, thanks to the power of

[8:40](https://www.youtube.com/watch?v=MwqUYRQloGw&t=520s) GPU acceleration, we have something that's much more usable. And believe me, it's even faster on NVIDIA GPUs. AMD and Intel, uh, support, uh, pending.

[8:51](https://www.youtube.com/watch?v=MwqUYRQloGw&t=531s) Now let's talk about what you can actually do with the library.

[8:54](https://www.youtube.com/watch?v=MwqUYRQloGw&t=534s) Let's start with three community projects to begin with. First, we've got Local AI. Local AI is a simple app that you can install to do inference locally. There's nothing magical about it.

[9:04](https://www.youtube.com/watch?v=MwqUYRQloGw&t=544s) It's just exactly what it says. I think that's really wonderful because it, it means anyone can download this app and get ready... Uh, get, be able to use local models without having to think about it.

[9:14](https://www.youtube.com/watch?v=MwqUYRQloGw&t=554s) Next up, llm-chain. It's, uh, LangChain but for Rust, and of course it supports inference with our library. And finally, we have Floneum, which is a flowchart-based application where you can build your own workflows.

[9:24](https://www.youtube.com/watch?v=MwqUYRQloGw&t=564s) I think we've seen a few of that, few of those at this con- uh, this conference. And you can combine and, uh, create vi- uh, nodes to, uh, build the workflow you need.

[9:32](https://www.youtube.com/watch?v=MwqUYRQloGw&t=572s) And of course, it supports the library as an inference engine.

[9:36](https://www.youtube.com/watch?v=MwqUYRQloGw&t=576s) Now, I wouldn't be a very good library author if I didn't actually test my own library. So I'm gonna go through three applications. The first two are proofs of concept.

[9:45](https://www.youtube.com/watch?v=MwqUYRQloGw&t=585s) The first is llmcord. It's a Discord bot. [laughs]

[9:49](https://www.youtube.com/watch?v=MwqUYRQloGw&t=589s) You can see it's exactly what you'd expect. You send, uh, give it a prompt, it'll give you a response. Any hitches you see come from Discord limits, not from the actual, uh, inferencing itself.

[9:58](https://www.youtube.com/watch?v=MwqUYRQloGw&t=598s) You can see, bam, all there. When an issue, when a user, uh, issues a request for generation, it goes through this process here where the request goes through a generation thread, uh, with a channel.

[10:12](https://www.youtube.com/watch?v=MwqUYRQloGw&t=612s) That channel is then used, uh, to create a response task, and then that response task is r- responsible for sending the responses to the, uh, user. Now, the interesting thing is these sessions are created and thrown away immediately with each query.

[10:28](https://www.youtube.com/watch?v=MwqUYRQloGw&t=628s) But you don't need to do that. If you keep them around, you can actually use them for conversation.

[10:33](https://www.youtube.com/watch?v=MwqUYRQloGw&t=633s) And just to illustrate, this is just like the request response w- workflow you would use for anything. If I just take what I had there and drop the Discord bit and add in HTTP, you can see request, generation, response.

[10:45](https://www.youtube.com/watch?v=MwqUYRQloGw&t=645s) Easy. Next up, Alpa. I love using GitHub Copilot, but it's only available in my code editor, and it requires internet connection. Alpa is my attempt to solve this. It is autocomplete anywhere on your system just by taking what's left of your cursor and, uh, having, passing it to a model to type in.

[11:04](https://www.youtube.com/watch?v=MwqUYRQloGw&t=664s) And of course, you can use any model, including a model fine-tuned in your own writing. Ask me how I know. [laughs]

[11:11](https://www.youtube.com/watch?v=MwqUYRQloGw&t=671s) Alpa is also quite simple. In fact, it's so simple I don't really need to cover it. Listen for input, copy the, uh, the input, um, into a prompt, start generating, type out response.

[11:20](https://www.youtube.com/watch?v=MwqUYRQloGw&t=680s) Easy. Now, the first two examples were pretty simple. They're proofs of concept. But now I want to talk about an actual use case. This is a real world data extraction task.

[11:31](https://www.youtube.com/watch?v=MwqUYRQloGw&t=691s) Over the last few years, I've been working on a project to make a timeline from dates on Wikipedia because there are millions of pages and they all have dates, and you can build a world history from it.

[11:40](https://www.youtube.com/watch?v=MwqUYRQloGw&t=700s) However, these dates are often unstructured and more or less impossible to parse using traditional means. Like, yes, you can try using regex to extract the dates, but you can't get the context out in any meaningful sense, and there are some dates here that just don't make any sense at all.

[11:53](https://www.youtube.com/watch?v=MwqUYRQloGw&t=713s) So that's why, as is the theme of this conference, I threw a large language model at it. However, GPT-3 and 4 aren't perfect, even after rounds of prompt engineering.

[12:02](https://www.youtube.com/watch?v=MwqUYRQloGw&t=722s) You can see I tried here. And handling millions of dates is just too expensive and slow. So I decided I'd fine-tune my own model. I generated a representative data set using GPT-3, built a tool, uh, tool to go through the data set.

[12:15](https://www.youtube.com/watch?v=MwqUYRQloGw&t=735s) So pick out any data point, fix it up, and then correct the errors, build a new data set, and train a new model.

[12:22](https://www.youtube.com/watch?v=MwqUYRQloGw&t=742s) So we did that using Axolotl, which I mentioned earlier. Again, check out Axolotl for all your fine-tuning needs. Highly recommended. And now I have a small, fast, consistent model that I can pass any data to, uh, sorry, any date to, and get back a structured, uh, representation, which I can then of course immediately parse using Rust.

[12:38](https://www.youtube.com/watch?v=MwqUYRQloGw&t=758s) And I can treat that as a black box. So I have a function there, fn parse, pass some dates, get some dates back. Simple. Now, let's quickly talk about the benefits of using local models and the library.

[12:50](https://www.youtube.com/watch?v=MwqUYRQloGw&t=770s) First off, deployments. Show of hands, who's had to deal with Python deployment hell?

[12:55](https://www.youtube.com/watch?v=MwqUYRQloGw&t=775s) Dependency hell even. Yeah, yeah, I know. It's, it's awful. You spend hours just trying to sort out your, your conda, your pip, your Pipenv. It's awful. With the library, you inherit Rust's excellent cross-platform support and build system, making it easy to ship self-enclosed support, uh, binaries to your platform.

[13:14](https://www.youtube.com/watch?v=MwqUYRQloGw&t=794s) No more making your users install Torch. As you might imagine, this unlocks use of desktop applications with models.

[13:21](https://www.youtube.com/watch?v=MwqUYRQloGw&t=801s) Next up, the ecosystem. Rust has one of the strongest ecosystems of, uh, of any native language. You can combine these libraries with LLMs to build all kinds of things.

[13:31](https://www.youtube.com/watch?v=MwqUYRQloGw&t=811s) It's what let me build a Discord bot, a system auto-completion utility, a data ingestion pipeline with a data set, a utility explorer, all in the same language. And I think if you use llm.rs, you can do the same thing with your, uh, task as well.

[13:45](https://www.youtube.com/watch?v=MwqUYRQloGw&t=825s) Of course, you also have control over how, uh, the model generates. I alluded to this earlier, but you can choose exactly how it samples tokens. Normally, when you use a cloud model, you have to get back the, uh, logits, the l- uh, probabilities.

[13:56](https://www.youtube.com/watch?v=MwqUYRQloGw&t=836s) But those probabilities are limited. Like, you have to keep going back and forth, and that's slow and expensive. With this, you can directly control what you are sampling.

[14:07](https://www.youtube.com/watch?v=MwqUYRQloGw&t=847s) Finally, let's talk about the innovation in the space. If you're here, you probably know there's a paper almost every single day. It's impossible to keep up with. Trust me, I've tried.

[14:16](https://www.youtube.com/watch?v=MwqUYRQloGw&t=856s) But it mean, but the use of local models means you can try this out before anyone else can. You can go through. You can try out some of these papers and be like, "Oh, wow, that's actually a worthwhile improvement."

[14:24](https://www.youtube.com/watch?v=MwqUYRQloGw&t=864s) And eventually, the cloud providers will provide them, but in the meantime, the control remains with you.

[14:29](https://www.youtube.com/watch?v=MwqUYRQloGw&t=869s) However, it's time to talk about the problems. There ain't no such thing as a free lunch, except if you're at a conference, of course. [laughs]

[14:37](https://www.youtube.com/watch?v=MwqUYRQloGw&t=877s) Let's talk about hardware again. I mentioned earlier that you can pretty much run any, uh, these things on almost any hardware, but that's kind of a lie. You still need some kind of power.

[14:48](https://www.youtube.com/watch?v=MwqUYRQloGw&t=888s) You c- you can only get so much out of your 10-year-old computer, your smartphone, or your Raspberry Pi. We're finding clever ways to improve this, like smaller models and better inferencing, but it's still something to be aware of.

[15:00](https://www.youtube.com/watch?v=MwqUYRQloGw&t=900s) Next, as with all things, the fast, cheap, good triad applies. You can, uh, make all kinds of trade-offs here, and you can see I've listed a couple of them here.

[15:08](https://www.youtube.com/watch?v=MwqUYRQloGw&t=908s) But fundamentally, you have to choose what are you willing to sacrifice in or- in order to serve your a- application. Are you willing to go for a bigger model to get better quality results at the cost of speed?

[15:18](https://www.youtube.com/watch?v=MwqUYRQloGw&t=918s) These are all decisions you have to make, and they're not always obvious. It's something you have to think about.

[15:26](https://www.youtube.com/watch?v=MwqUYRQloGw&t=926s) Next, there's no other way of putting this. The ecosystem churns. Innovation is a double-edged sword. When those changes come in, they can often break your existing workflows. I've helped alleviate this w- to some extent using the GGUF, uh, file format which helps standardize, but it's still a problem.

[15:41](https://www.youtube.com/watch?v=MwqUYRQloGw&t=941s) Some days you will just wake up, try your application with a new model, and it just won't work. There's nothing you can do except deal with it. Finally, a lot of the models in this space are open source.

[15:53](https://www.youtube.com/watch?v=MwqUYRQloGw&t=953s) They're free for you to use personally, but they have very strange clauses and exceptions. For most of us, this doesn't matter. You can just use the model personally. But it's a reminder that even though that these models are free, they're not capital F Free.

[16:06](https://www.youtube.com/watch?v=MwqUYRQloGw&t=966s) Luckily, there's been some recent change in this space with Mistral and StableLM giving you strong performance for small level, uh, sorry, a small size and being completely unburdened. But it's still a problem, and they're still, uh, much smaller than the big ones like Llama and Falcon.

[16:20](https://www.youtube.com/watch?v=MwqUYRQloGw&t=980s) Unfortunately, I've got to wrap things up here. There's only so much you can talk about in 18 minutes, I'm afraid. Local models are great, and I like, I'd like to think our library is too.

[16:30](https://www.youtube.com/watch?v=MwqUYRQloGw&t=990s) They're getting easier to run day by day with smaller, more powerful models. However, the situation isn't perfect, and there isn't always one obvious solution for your problem.

[16:38](https://www.youtube.com/watch?v=MwqUYRQloGw&t=998s) Thanks for listening. You can contact me by email or via Mastodon. The library can be found at, you guessed it, llm.rs or by scanning the QR code. Finally, we're always looking for contributors.

[16:49](https://www.youtube.com/watch?v=MwqUYRQloGw&t=1009s) If you're interested in LLMs or Rust, feel free to reach out. Sponsorships are also very welcome because they help me try out new hardware, which is always necessary. And if you want to chat in person, I'll be hanging around the conference.

[16:59](https://www.youtube.com/watch?v=MwqUYRQloGw&t=1019s) See you later. [audience cheering] [upbeat music]
