# Foundry Local: Cutting-Edge AI Experiences on Device with ONNX Runtime and Olive — Emma Ning, Microsoft

Emma Ning · Tsavo Knott

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/l614N5W60ls-foundry-local-cutting-edge-ai-experiences-on) · [Video](https://www.youtube.com/watch?v=l614N5W60ls)

## Foundry Local: Cutting-Edge AI Experiences on Device with ONNX Runtime and Olive

From unreliable conference Wi-Fi to private document summaries and receipt extraction, Foundry Local brings model management, hardware acceleration, and tool use onto the device.

Familiarity with command-line tools and JavaScript chat-completion requests will help you follow the application integration.

### When the network cannot be a dependency

[0:16](https://www.youtube.com/watch?v=l614N5W60ls&t=16s)

What happens to a cloud AI demo when the conference Wi-Fi fails? The model may be powerful, but the application still depends on reaching it. Microsoft product manager Emma Ning opens with that familiar problem: presenters hoping their connection survives the next request. Her [Foundry Local](https://github.com/microsoft/Foundry-Local) demonstration runs inference on the device instead.

Network reliability is only the first reason to move inference closer to the application. The other constraints concern what data can leave, what repeated requests cost, and how quickly the application must respond:

- **Privacy and security:** Legal documents and patient information may need to be processed entirely on-device.
- **Cost efficiency:** Ning asks developers to imagine a game deployed to millions of devices, generating hundreds of millions of inference calls each day. This is a workload scenario motivating local execution, rather than a measured cost comparison.
- **Real-time latency:** Interactive applications can have response requirements that leave little room for a cloud round trip.

**Local inference makes the device an execution environment, rather than just a client of a remote model.**

![The “Why local AI” slide shows four illustrated cards linking these motivations to warehouses, healthcare, video games, and creativity tools.](https://ai.engineer/api/talk-essay-media/5203a99bb9c6edbae8a740de25fddd747a3ac229bf8196dbe8ba316e13e63deb)

Four reasons for local AI: low bandwidth, privacy and security, cost efficiency, and real-time latency.

### From capable hardware to a usable local stack

[2:27](https://www.youtube.com/watch?v=l614N5W60ls&t=147s)

Local AI becomes practical when hardware, models, and runtime software improve together. Modern client GPUs and NPUs provide more compute; smaller models such as [**Phi-4 Mini**](https://huggingface.co/microsoft/Phi-4-mini-instruct) and small **DeepSeek** variants reduce the resources required for useful generation; runtime optimizations make better use of the available hardware. None of those pieces alone supplies an application developer with the whole experience.

Microsoft assembles Foundry Local from existing assets. Ning describes Azure AI Foundry, introduced at the previous year's Microsoft Ignite, as serving more than 70,000 organizations and offering more than 1,900 models at the time of the talk. The [Build 2025 announcement](https://blogs.microsoft.com/blog/2025/05/19/microsoft-build-2025-the-age-of-ai-agents-and-building-the-open-agentic-web/) places that model count in the cloud platform's partner-hosted and Microsoft-hosted catalog; it is not a count of models available for local execution.

[ONNX Runtime](https://onnxruntime.ai/docs/) supplies the cross-platform inference engine. Ning reports more than 10 million ONNX Runtime downloads per month and describes customer-reported acceleration over PyTorch, without presenting a quantified performance comparison. Windows contributes distribution and reach across millions of client devices and users. Foundry Local combines these assets into a managed path from model selection to on-device execution.

The stack separates execution from model management and application integration:

| Layer | Responsibility |
| --- | --- |
| ONNX Runtime | Accelerate inference across supported hardware |
| Foundry Local management service | Host and manage models on the client device |
| Model acquisition | Download open-source models from Azure AI Foundry on demand |
| CLI | Explore and run models locally |
| SDKs | Integrate local inference into applications |

This division lets a developer explore models through the CLI before embedding the selected model in an application.

At the time of this June 2025 talk, Foundry Local had recently been announced at Microsoft Build and was available on Windows and macOS. Ning emphasizes its Windows integration; the [launch announcement](https://blogs.windows.com/windowsdeveloper/2025/05/19/advancing-windows-for-ai-development-new-platform-capabilities-and-tools-introduced-at-build-2025/) distinguishes the installable preview from direct Windows 11 and Windows App SDK availability, which was still forthcoming. Hardware work with NVIDIA, Intel, AMD, and Qualcomm connects the runtime to their accelerators. Ning says more than 100 customers joined the private pre-preview and supplied feedback before the public announcement.

### Developer memory and sensitive speech

[6:46](https://www.youtube.com/watch?v=l614N5W60ls&t=406s)

Pieces provides one example of why packaging and lifecycle management matter alongside inference speed. Tsavo Knott, its CEO and co-founder, describes a product that gives developers artificial long-term memory across the operating system. Offline-first AI is central to that goal, and the team began exploring small language models across major platforms in late 2022. It moved from managing its own `llama.cpp` setup to Ollama, but still encountered difficulties with versioning, performance, and the end-user experience.

Knott reports a rapid move from access to Foundry Local documentation to a production-ready build, with improvements in memory management, time to first token, and tokens per second. The testimonial supplies no measurements, but identifies the operational concerns that a local inference platform needs to address: loading and managing models, starting responses promptly, and sustaining generation within a desktop application's resource budget.

A second customer example starts from a stricter data boundary: clients want generative AI but cannot send sensitive information to external services. The demonstration combines Foundry Local with a speech-to-text service that also runs locally. Its sample speech includes the name Maria Rodriguez and a redacted savings account number—the sort of input that makes the location of processing consequential. The customer highlights straightforward installation and model use, then describes hybrid applications in which only part of the workload runs locally.

### Discover hardware variants and prepare for offline use

[8:41](https://www.youtube.com/watch?v=l614N5W60ls&t=521s)

The CLI demonstration starts with Foundry Local already installed. Windows installation uses WinGet; macOS installation uses Homebrew. The first exploration command is:

```bash
foundry model list
```

The catalog exposes models together with variants optimized for different hardware. Ning's device shows CPU, CUDA, and integrated-GPU options. A Qualcomm NPU variant does not appear because that device lacks the corresponding NPU. Model discovery therefore includes hardware compatibility, not just a list of model families.

**Offline inference still requires preparing the device.** When a model is not cached, Foundry Local downloads it from the cloud before running it, which requires internet access. Ning has downloaded the models in advance and inspects them with:

```bash
foundry cache list
```

Her cache contains four models. Keeping several models locally allows her to compare response quality and performance without depending on the conference network for model acquisition.

### Compare speed and answer detail with the same prompt

[10:41](https://www.youtube.com/watch?v=l614N5W60ls&t=641s)

Ning first uses `foundry model run` to load a cached [**Qwen 2.5 1.5B**](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) model. Once its interactive session is ready, she asks, “What's ONNX Runtime?” The initial answer arrives quickly enough that she exits and repeats the request in verbose mode to inspect the generation rate.

For the Qwen 2.5 1.5B response to “What's ONNX Runtime?”, Ning reports approximately 90 tokens per second in verbose mode on her device. The hardware specification, quantization, execution provider, runtime build, measurement boundaries, and repeated trials are not supplied, so this number describes the displayed run rather than a portable performance expectation.

She then loads Phi-4 Mini in verbose mode and asks the same question. In this demonstration, the larger Phi-4 Mini model generates more slowly but supplies more detailed information. Ning prefers that answer for the application she wants to build.

| Model | Observed behavior in the demo | Selection implication |
| --- | --- | --- |
| Qwen 2.5 1.5B | Faster, briefer response | Useful candidate when speed and brevity matter |
| Phi-4 Mini | Slower, more detailed response | Ning's choice for the summarizer |

The comparison makes model selection tangible: run the intended style of request, inspect both the output and the execution behavior, and choose according to the application's needs. Here, detail in one answer drives the choice; the demonstration does not establish general model superiority.

### Summarize internal documents without uploading them

[13:06](https://www.youtube.com/watch?v=l614N5W60ls&t=786s)

The application problem is onboarding into an unfamiliar project. A team has long, detailed documents to read and wants high-level summaries to get oriented. Those documents are internal and cannot be uploaded to the cloud. Some teammates use Windows and others use Macs, so the solution needs both local inference and a cross-platform application.

The prepared summarization app accepts either a URL or a local file. Its settings tab lets the user select a model by ID. Ning chooses Phi-4 Mini, supplies the project document, and submits it for summarization. The response describes Foundry Local as a way to build cross-platform AI applications that run directly on-device. The workflow carries the earlier model experiment into a concrete task: choose a model, provide the document, and receive a summary inside the application.

### Connect the JavaScript app and run it on a Mac

[15:35](https://www.youtube.com/watch?v=l614N5W60ls&t=935s)

Foundry Local offers Python and JavaScript SDKs; the summarizer uses JavaScript. The demonstrated integration creates a Foundry Local manager, initializes it with the selected Phi-4 Mini model, and uses the manager's local endpoint to construct a chat client. The application then awaits a chat completion and displays the result. This is the talk's manager-and-endpoint integration; today's SDK examples emphasize in-process inference with optional serving, so the two integration styles should not be mixed.

![VS Code displays model initialization, client creation, and the beginning of a summarization prompt in the Brevity project.](https://ai.engineer/api/talk-essay-media/67ced142fb6bd2128885bfc179f176e54aaa3ef05ebdb4593a60cdf0c5be8d74)

JavaScript code initializes the model and creates a client using the Foundry Local endpoint.

Once the client points at the local service and the selected model is initialized, the summarization request can remain a small JavaScript function:

```javascript
async function summarizeDocument(client, modelId, documentText) {
  const completion = await client.chat.completions.create({
    model: modelId,
    messages: [
      {
        role: "system",
        content: "Summarize the document's main ideas and key details."
      },
      {
        role: "user",
        content: documentText
      }
    ]
  });

  return completion.choices[0].message.content;
}
```

The manager handles model initialization, while this function handles the application's request and response. The model ID must refer to the selected local model, and the client must use the Foundry Local endpoint.

Ning then packages the Windows project and shares it with a teammate who records the Mac experience. The teammate starts the same code with:

```bash
npm run start
```

The application and UI are unchanged. On the Mac, the teammate selects Qwen and submits the same document. Its summary is shorter than the earlier Phi-4 Mini output, echoing the difference seen in the CLI comparison. The handoff demonstrates reuse of the application across operating systems while leaving model choice available to the user.

### Give a local model tools through MCP

[17:55](https://www.youtube.com/watch?v=l614N5W60ls&t=1075s)

The final demonstration moves from generating a summary to using tools. Foundry Local's agent feature is in private preview in the recording. Its composition is straightforward: **one local model plus one or more MCP servers**. The model supplies the reasoning and tool selection; the configured servers supply the available operations.

Ning returns to the terminal and runs:

```bash
foundry agent list
```

The output shows three sample agents: a file organizer, an OCR agent, and a web interactor, with their server counts and model names. A developer can compose an agent by choosing a model from the catalog and the MCP servers needed for the task.

![Terminal tables list available MCP servers and the sample file organizer, OCR agent, and web interactor, alongside server counts and model names.](https://ai.engineer/api/talk-essay-media/6aae0aaef06e744bc33b99eb2c1803425129f4bc2402fb8dac58a9799389fb58)

Foundry Local lists MCP servers and three sample agents with their server counts and models.

For this walkthrough, Ning selects the existing OCR agent and inspects it with `foundry agent info`. It extracts text from images on the local device and combines a Phi-4 Mini variant with two servers: a filesystem MCP server and an OCR Mini MCP server. She then launches it with `foundry agent run`. These are the private-preview commands shown in the recording.

### Authorize access, find the receipt, and extract its total

[20:03](https://www.youtube.com/watch?v=l614N5W60ls&t=1203s)

Launching the agent does not immediately grant it unrestricted access. Startup first checks its dependencies and, if any are missing, offers to install them with permission. Ning has already installed the dependencies. She then works through the access prompts:

1. **Approve the filesystem MCP server.**
2. **Supply the allowed directory:** the folder containing the demonstration photos.
3. **Approve the OCR MCP server.**

After setup, the agent lists the tools provided by those servers, including filesystem and OCR operations. Dependency installation, server approval, and directory selection are separate parts of preparing the agent to act.

The user request is to find a receipt, process it, and obtain the total amount. The agent first selects file search to locate the receipt image. It then uses OCR to extract the text and produces the total. The completed terminal view shows the located file, extracted receipt text, and a final result of `Total: $8.39`. This is a concrete division of labor: filesystem tools locate the input, OCR turns image content into text, and the model uses those results to answer the request.

![Terminal output shows a receipt image search result, OCR text, and a final result reading “Total: $8.39.”](https://ai.engineer/api/talk-essay-media/7674baed4cb4512bc8ae2df19d5f6a3b69e30386ccadbecf59c90faaaf651704)

The receipt workflow shows the located file, extracted receipt text, and final total.

The receipt task also gives the closing capability warning a practical setting. Ning cautions that local models are generally less capable than cloud models and should not be expected to perform the same sophisticated model or agent work. A bounded task with relevant tools can still be useful, as the receipt workflow shows. She closes by pointing attendees to further information and a signup form for the agent private preview. The engineering opportunity is to choose work that fits the local model's capabilities while benefiting from on-device data handling and execution.

## Resources

- [Foundry Local SDK and examples](https://github.com/microsoft/Foundry-Local) — Current SDK quickstarts, model management, hardware acceleration and application samples.
- [Foundry Local at Build 2025](https://blogs.windows.com/windowsdeveloper/2025/05/19/advancing-windows-for-ai-development-new-platform-capabilities-and-tools-introduced-at-build-2025/) — The launch-era account of Foundry Local preview and its relationship to Windows ML and Windows AI APIs.
- [ONNX Runtime documentation](https://onnxruntime.ai/docs/) — Documentation for the inference runtime underlying Foundry Local.
- [Olive model optimization](https://github.com/microsoft/Olive) — Microsoft tooling for converting, quantizing and optimizing models for different hardware targets.
- [Phi-4-mini-instruct](https://huggingface.co/microsoft/Phi-4-mini-instruct) — Microsoft's model card for the instruction-tuned Phi-4 Mini model.
- [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) — The Qwen team's model card and usage guidance for its 1.5-billion-parameter instruction model.
- [Foundry Local with Agent Framework](https://learn.microsoft.com/en-us/agent-framework/integrations/by-component/model-providers/foundry-local) — Current guidance for local agents, model-dependent tool calling and local MCP integration.

## Slides

### Innovation partner and platinum sponsors

[0:10](https://www.youtube.com/watch?v=l614N5W60ls&t=10s) [AWS is the innovation partner; six platinum sponsor logos appear below.](https://assets.ai.engineer/sites/aiecode2025/sha256/64c58a4a27d2ca7b35caeb394f808d846769697c3ea967195436507428733cb1.webp)

INNOVATION PARTNER
aws

PLATINUM SPONSORS
Graphite
Windsurf
MongoDB®
daily™
augment code
WorkOS

### Audio Analyzer transcript masking

Audio Analyzer displays a Spanish transcript with sensitive fields masked. The customer describes a hybrid solution using Foundry Local and local speech-to-text.

[8:22](https://www.youtube.com/watch?v=l614N5W60ls&t=502s) [Audio Analyzer displays a Spanish transcript with sensitive fields masked.](https://assets.ai.engineer/sites/aiecode2025/sha256/1ce23880bc2d64dcd1e66e6802a627a8c489d94999d2bedd2af47b906e998ee8.webp)

Thank you to our 100+ Private Preview customers

Audio Analyzer
Audio Library

Segundo.wav
4/21/2025 · 605.59 KB · 00:19
Uploading
Analyze

Transcription
Analysis

Transcript
Edit transcript segments or export the full transcript
Sensitive data detected and masked with asterisks

00:00 - 00:15  Sensitive data  Edit
Hola mi nombre es ********[illegible] mi cuenta de ahorros es la ********[illegible].

00:15 - 00:19  Edit
Hice una transferencia el viernes pero todavía no se ha reflejado en la cuenta de destino.

00:30 - 00:19  Edit
Por favor revisar eso porque me urge que se complete.

00:45 - 00:19  Edit
Gracias.

Interface text:
Classified as Microsoft Highly Confidential
AIE
Capgemini
hp
iLink Digital
Thinking Beyond
Kingston Is With You
UiPath
Avaelgo
Part of IPS Group
PIECES.
softwareone
quantium
KPMG
avanade
Microsoft
smol ai

### Foundry Local availability

The card announces Foundry Local is “Available Today” for Windows and Mac and thanks 100+ private preview customers.

[8:38](https://www.youtube.com/watch?v=l614N5W60ls&t=518s) [A title card announces Foundry Local availability for Windows and Mac.](https://assets.ai.engineer/sites/aiecode2025/sha256/eefdb6ce959f6a292818a70334673bf71252b96fef5248502ed3b9ffb8328666.webp)

Thank you to our 100+ Private Preview customers

Foundry Local
Available Today
For Windows and Mac

Capgemini
HP
iLink Digital
Thinking Beyond
KINGSTON IS WITH YOU
UiPath
Avaelgo
Part of IPS Group
PIECES.
softwareOne
quantium
KPMG
avanade

Classified as Microsoft Highly Confidential
AIE
Microsoft
smol ai

### Foundry Local CLI

Windows uses winget and macOS uses Homebrew to install Foundry Local; both examples then run phi-3.5-mini with the same CLI command.

[8:47](https://www.youtube.com/watch?v=l614N5W60ls&t=527s) [Windows and macOS instructions show how to install Foundry Local and run a model.](https://assets.ai.engineer/sites/aiecode2025/sha256/21b01721555e66882e0fe872805c58114eca6af51916cb66f0c2113bdfacc636.webp)

Foundry Local CLI

On Windows
1. Open Windows Terminal
2. Install Foundry Local using winget
   winget install Microsoft.FoundryLocal
3. Run a model
   foundry model run phi-3.5-mini

On MacOS
1. Open Terminal
2. Install Foundry Local
   brew tap microsoft/foundrylocal
   brew install foundrylocal
3. Run a model
   foundry model run phi-3.5-mini

Classified as Microsoft Highly Confidential

Interface text:
AIE
Microsoft
smol ai

### Foundry Local model list and cache

Emma Ning lists model variants for different hardware, then uses foundry cache list to inspect four previously downloaded models.

[9:25](https://www.youtube.com/watch?v=l614N5W60ls&t=565s) [Command Prompt lists model variants, devices, file sizes and licenses.](https://assets.ai.engineer/sites/aiecode2025/sha256/017434a29365e597718bd631e8cc8197908924c1f34f60a02cf851f682d208ed.webp)

GPU  chat-completion  0.68 GB  apache-2.0  qwen2.5-0.5b-instruct-generic-gpu
CPU  chat-completion  0.80 GB  apache-2.0  qwen2.5-0.5b-instruct-generic-cpu

qwen2.5-1.5b
GPU  chat-completion  1.25 GB  apache-2.0  qwen2.5-1.5b-instruct-cuda-gpu
GPU  chat-completion  1.51 GB  apache-2.0  qwen2.5-1.5b-instruct-generic-gpu
CPU  chat-completion  1.78 GB  apache-2.0  qwen2.5-1.5b-instruct-generic-cpu

qwen2.5-coder-7b
GPU  chat-completion  4.73 GB  apache-2.0  qwen2.5-coder-7b-instruct-cuda-gpu
GPU  chat-completion  4.73 GB  apache-2.0  qwen2.5-coder-7b-instruct-generic-gpu
CPU  chat-completion  6.16 GB  apache-2.0  qwen2.5-coder-7b-instruct-generic-cpu

qwen2.5-coder-0.5b
GPU  chat-completion  0.52 GB  apache-2.0  qwen2.5-coder-0.5b-instruct-cuda-gpu
GPU  chat-completion  0.52 GB  apache-2.0  qwen2.5-coder-0.5b-instruct-generic-gpu
CPU  chat-completion  0.80 GB  apache-2.0  qwen2.5-coder-0.5b-instruct-generic-cpu

qwen2.5-coder-1.5b
GPU  chat-completion  1.25 GB  apache-2.0  qwen2.5-coder-1.5b-instruct-cuda-gpu
GPU  chat-completion  1.25 GB  apache-2.0  qwen2.5-coder-1.5b-instruct-generic-gpu
CPU  chat-completion  1.78 GB  apache-2.0  qwen2.5-coder-1.5b-instruct-generic-cpu

phi-4-mini
GPU  chat-completion  3.60 GB  MIT  Phi-4-mini-instruct-cuda-gpu
GPU  chat-completion  3.72 GB  MIT  Phi-4-mini-instruct-generic-gpu
CPU  chat-completion  4.80 GB  MIT  Phi-4-mini-instruct-generic-cpu

phi-4-mini-reasoning
GPU  chat-completion  3.15 GB  MIT  Phi-4-mini-reasoning-cuda-gpu
GPU  chat-completion  3.15 GB  MIT  Phi-4-mini-reasoning-generic-gpu
CPU  chat-completion  4.52 GB  MIT  Phi-4-mini-reasoning-generic-cpu

qwen2.5-14b
GPU  chat-completion  8.79 GB  apache-2.0  qwen2.5-14b-instruct-cuda-gpu
CPU  chat-completion  11.06 GB  apache-2.0  qwen2.5-14b-instruct-generic-cpu

qwen2.5-7b
GPU  chat-completion  4.73 GB  apache-2.0  qwen2.5-7b-instruct-cuda-gpu
GPU  chat-completion  5.20 GB  apache-2.0  qwen2.5-7b-instruct-generic-gpu
CPU  chat-completion  6.16 GB  apache-2.0  qwen2.5-7b-instruct-generic-cpu

qwen2.5-coder-14b
GPU  chat-completion  8.79 GB  apache-2.0  qwen2.5-coder-14b-instruct-cuda-gpu
GPU  chat-completion  8.79 GB  apache-2.0  qwen2.5-coder-14b-instruct-generic-gpu
CPU  chat-completion  11.06 GB  apache-2.0  qwen2.5-coder-14b-instruct-generic-cpu

Interface text:
Command Prompt
Search
ENG
3:41 PM
6/5/2025
AIE
Microsoft
smol ai

[10:22](https://www.youtube.com/watch?v=l614N5W60ls&t=622s) [Command prompt showing available models and a list of cached models.](https://assets.ai.engineer/sites/aiecode2025/sha256/c9463f5f01ef467b0a977cf933b5c5a355da4e9bdf8753b4410d946ce97862ac.webp)

GPU  chat-completion  4.73 GB  apache-2.0  qwen2.5-coder-7b-instruct-generic-gpu
CPU  chat-completion  6.16 GB  apache-2.0  qwen2.5-coder-7b-instruct-generic-cpu

qwen2.5-coder-0.5b
GPU  chat-completion  0.52 GB  apache-2.0  qwen2.5-coder-0.5b-instruct-cuda-gpu
GPU  chat-completion  0.52 GB  apache-2.0  qwen2.5-coder-0.5b-instruct-generic-gpu
CPU  chat-completion  0.80 GB  apache-2.0  qwen2.5-coder-0.5b-instruct-generic-cpu

qwen2.5-coder-1.5b
GPU  chat-completion  1.25 GB  apache-2.0  qwen2.5-coder-1.5b-instruct-cuda-gpu
GPU  chat-completion  1.25 GB  apache-2.0  qwen2.5-coder-1.5b-instruct-generic-gpu
CPU  chat-completion  1.78 GB  apache-2.0  qwen2.5-coder-1.5b-instruct-generic-cpu

phi-4-mini
GPU  chat-completion  3.60 GB  MIT  Phi-4-mini-instruct-cuda-gpu
GPU  chat-completion  3.72 GB  MIT  Phi-4-mini-instruct-generic-gpu
CPU  chat-completion  4.80 GB  MIT  Phi-4-mini-instruct-generic-cpu

phi-4-mini-reasoning
GPU  chat-completion  3.15 GB  MIT  Phi-4-mini-reasoning-cuda-gpu
GPU  chat-completion  3.15 GB  MIT  Phi-4-mini-reasoning-generic-gpu
CPU  chat-completion  4.52 GB  MIT  Phi-4-mini-reasoning-generic-cpu

qwen2.5-14b
GPU  chat-completion  8.79 GB  apache-2.0  qwen2.5-14b-instruct-cuda-gpu
CPU  chat-completion  11.06 GB  apache-2.0  qwen2.5-14b-instruct-generic-cpu

qwen2.5-7b
GPU  chat-completion  4.73 GB  apache-2.0  qwen2.5-7b-instruct-cuda-gpu
GPU  chat-completion  5.20 GB  apache-2.0  qwen2.5-7b-instruct-generic-gpu
CPU  chat-completion  6.16 GB  apache-2.0  qwen2.5-7b-instruct-generic-cpu

qwen2.5-coder-14b
GPU  chat-completion  8.79 GB  apache-2.0  qwen2.5-coder-14b-instruct-cuda-gpu
GPU  chat-completion  8.79 GB  apache-2.0  qwen2.5-coder-14b-instruct-generic-gpu
CPU  chat-completion  11.06 GB  apache-2.0  qwen2.5-coder-14b-instruct-generic-cpu

C:\Users\qining>foundry cache list
Models cached on device:
Alias  Model ID
phi-4-mini  Phi-4-mini-instruct-cuda-gpu
qwen2.5-1.5b  qwen2.5-1.5b-instruct-cuda-gpu
qwen2.5-1.5b  qwen2.5-1.5b-instruct-generic-cpu
[illegible]  qwen2.5-7b-instruct-cuda-gpu

Interface text:
Command Prompt
Search
ENG
3:42 PM
6/5/2025
AIE

### Running a cached model

Ning runs Qwen 2.5 1.5B from the local cache. Foundry loads the CUDA GPU variant and opens an interactive chat.

[11:12](https://www.youtube.com/watch?v=l614N5W60ls&t=672s) [Command prompt showing a cached model loaded successfully and interactive chat ready.](https://assets.ai.engineer/sites/aiecode2025/sha256/10aefde8be75ee51374257cd8865be7a88caf7d5956bf922c4fa341138a13feb.webp)

GPU  chat-completion  1.25 GB  apache-2.0  qwen2.5-coder-1.5b-instruct-generic-gpu
CPU  chat-completion  1.78 GB  apache-2.0  qwen2.5-coder-1.5b-instruct-generic-cpu

phi-4-mini
GPU  chat-completion  3.60 GB  MIT  Phi-4-mini-instruct-cuda-gpu
GPU  chat-completion  3.72 GB  MIT  Phi-4-mini-instruct-generic-gpu
CPU  chat-completion  4.80 GB  MIT  Phi-4-mini-instruct-generic-cpu

phi-4-mini-reasoning
GPU  chat-completion  3.15 GB  MIT  Phi-4-mini-reasoning-cuda-gpu
GPU  chat-completion  3.15 GB  MIT  Phi-4-mini-reasoning-generic-gpu
CPU  chat-completion  4.52 GB  MIT  Phi-4-mini-reasoning-generic-cpu

qwen2.5-14b
GPU  chat-completion  8.79 GB  apache-2.0  qwen2.5-14b-instruct-cuda-gpu
CPU  chat-completion  11.06 GB  apache-2.0  qwen2.5-14b-instruct-generic-cpu

qwen2.5-7b
GPU  chat-completion  4.73 GB  apache-2.0  qwen2.5-7b-instruct-cuda-gpu
GPU  chat-completion  5.20 GB  apache-2.0  qwen2.5-7b-instruct-generic-gpu
CPU  chat-completion  6.16 GB  apache-2.0  qwen2.5-7b-instruct-generic-cpu

qwen2.5-coder-14b
GPU  chat-completion  8.79 GB  apache-2.0  qwen2.5-coder-14b-instruct-cuda-gpu
GPU  chat-completion  8.79 GB  apache-2.0  qwen2.5-coder-14b-instruct-generic-gpu
CPU  chat-completion  11.06 GB  apache-2.0  qwen2.5-coder-14b-instruct-generic-cpu

C:\Users\qining>foundry cache list
Models cached on device:
Alias             Model ID
phi-4-mini        Phi-4-mini-instruct-cuda-gpu
qwen2.5-1.5b      qwen2.5-1.5b-instruct-cuda-gpu
qwen2.5-1.5b      qwen2.5-1.5b-instruct-generic-cpu
qwen2.5-7b        qwen2.5-7b-instruct-cuda-gpu

C:\Users\qining>foundry model run qwen2.5-1.5b
Model qwen2.5-1.5b-instruct-cuda-gpu was found in the local cache.
Loading model...
Model qwen2.5-1.5b-instruct-cuda-gpu loaded successfully

Interactive Chat. Enter /? or /help for help.
[illegible]de, please enter your prompt
[illegible]run

Interface text:
Command Prompt - foundry
Search
ENG
3:42 PM
6/5/2025
AIE
Microsoft
smol ai

### Asking about ONNX Runtime

Ning asks the locally cached Qwen 2.5 1.5B model what ONNX Runtime is. Its interactive chat response lists features including backend support, performance and model serving.

[11:21](https://www.youtube.com/watch?v=l614N5W60ls&t=681s) [Interactive chat displays a question about ONNX Runtime and a model-generated response.](https://assets.ai.engineer/sites/aiecode2025/sha256/27c5a22d6eab19bd99e0e6a88b35cac847f8106020a978c1b0f6f26c47ec8582.webp)

Models cached on device:
Alias  Model ID
phi-4-mini  Phi-4-mini-instruct-cuda-gpu
qwen2.5-1.5b  qwen2.5-1.5b-instruct-cuda-gpu
qwen2.5-1.5b  qwen2.5-1.5b-instruct-generic-cpu
qwen2.5-7b  qwen2.5-7b-instruct-cuda-gpu

C:\Users\qining>foundry model run qwen2.5-1.5b
Model qwen2.5-1.5b-instruct-cuda-gpu was found in the local cache.
Loading model...
Model qwen2.5-1.5b-instruct-cuda-gpu loaded successfully

Interactive Chat. Enter /? or /help for help.

Interactive mode, please enter your prompt
> what's onnx runtime
Onnx Runtime is a powerful and flexible library for running machine learning models. It provides several key features:

1. **Flexibility**: Supports multiple backends, including CPU, GPU (via CUDA), and TPU.
2. **Performance**: Optimized for high-performance computing environments.
3. **Ease of Use**: Provides easy-to-use APIs to load and run ONNX models efficiently.
4. **Scalability**: Designed to handle large-scale data processing tasks.
5. **Integration**: Seamlessly integrates with various ML frameworks like TensorFlow, PyTorch, and others.
6. **Compatibility**: Compatible with different types of ONNX models (e.g., neural network architectures).
7. **Documentation and Community Support**: Well-documented and has active community support.
8. **Cross-Platform Compatibility**: Works across multiple platforms and languages.
9. **Model Serving**: Simplifies the process of deploying trained models in production environments.

[illegible]s widely used in industries such as healthcare, finance, and autonomous vehicles due to its robust performance and eas[illegible]
[illegible]llows developers to focus more on model development rather than infrastructure setup.
[illegible]runtime

Interface text:
Command Prompt - foundry
Search
ENG
3:43 PM
6/5/2025
AIE
Microsoft
smol ai

[11:25](https://www.youtube.com/watch?v=l614N5W60ls&t=685s) [Command prompt showing a cached Qwen model responding to a question about ONNX Runtime.](https://assets.ai.engineer/sites/aiecode2025/sha256/26f3a91c8a9c29dca473124690a9adb37fbe31d1da57059f954672af0f9899ad.webp)

Models cached on device:
Alias  Model ID
phi-4-mini  Phi-4-mini-instruct-cuda-gpu
qwen2.5-1.5b  qwen2.5-1.5b-instruct-cuda-gpu
qwen2.5-1.5b  qwen2.5-1.5b-instruct-generic-cpu
qwen2.5-7b  qwen2.5-7b-instruct-cuda-gpu

C:\Users\qining>foundry model run qwen2.5-1.5b
Model qwen2.5-1.5b-instruct-cuda-gpu was found in the local cache.
Loading model...
Model qwen2.5-1.5b-instruct-cuda-gpu loaded successfully

Interactive Chat. Enter /? or /help for help.

Interactive mode, please enter your prompt
> what's onnx runtime
Onnx Runtime is a powerful and flexible library for running machine learning models. It provides several key features:

1. **Flexibility**: Supports multiple backends, including CPU, GPU (via CUDA), and TPU.
2. **Performance**: Optimized for high-performance computing environments.
3. **Ease of Use**: Provides easy-to-use APIs to load and run ONNX models efficiently.
4. **Scalability**: Designed to handle large-scale data processing tasks.
5. **Integration**: Seamlessly integrates with various ML frameworks like TensorFlow, PyTorch, and others.
6. **Compatibility**: Compatible with different types of ONNX models (e.g., neural network architectures).
7. **Documentation and Community Support**: Well-documented and has active community support.
8. **Cross-Platform Compatibility**: Works across multiple platforms and languages.
9. **Model Serving**: Simplifies the process of deploying trained models in production environments.

[illegible]s widely used in industries such as healthcare, finance, and autonomous vehicles due to its robust performance and eas[illegible]
[illegible]llows developers to focus more on model development rather than infrastructure setup.

Interface text:
Command Prompt - foundry
Search
ENG
3:43 PM
6/5/2025
AIE
aws

### Foundry model cache and Qwen response

Foundry lists cached models, then loads Qwen 2.5 1.5B from the local cache and answers a question about ONNX Runtime.

[12:03](https://www.youtube.com/watch?v=l614N5W60ls&t=723s) [A terminal lists cached models and shows Qwen answering a question about ONNX Runtime.](https://assets.ai.engineer/sites/aiecode2025/sha256/d205dd9ca6f76f00cf379eb4fca80ac2b83d376c81848ff0835a4efbc0449100.webp)

C:\Users\qining>foundry cache list
Models cached on device:
Alias                 Model ID
phi-4-mini            Phi-4-mini-instruct-cuda-gpu
qwen2.5-1.5b          qwen2.5-1.5b-instruct-cuda-gpu
qwen2.5-1.5b          qwen2.5-1.5b-instruct-generic-cpu
qwen2.5-7b            qwen2.5-7b-instruct-cuda-gpu

C:\Users\qining>foundry model run qwen2.5-1.5b
Model qwen2.5-1.5b-instruct-cuda-gpu was found in the local cache.
Loading model...
Model qwen2.5-1.5b-instruct-cuda-gpu loaded successfully

Interactive Chat. Enter /? or /help for help.

Interactive mode, please enter your prompt
> what's onnx runtime
Onnx Runtime is a powerful and flexible library for running machine learning models. It provides several key features:

1. Flexibility: Supports multiple backends, including CPU, GPU (via CUDA), and TPU.
2. Performance: Optimized for high-performance computing environments.
3. Ease of Use: Provides easy-to-use APIs to load and run ONNX models efficiently.
4. Scalability: Designed to handle large-scale data processing tasks.
5. Integration: Seamlessly integrates with various ML frameworks like TensorFlow, PyTorch, and others.
6. Compatibility: Compatible with different types of ONNX models (e.g., neural network architectures).
7. Documentation and Community Support: Well-documented and has active community support.
8. Cross-Platform Compatibility: Works across multiple platforms and languages.
9. Model Serving: Simplifies the process of deploying trained models in production environments.

[illegible]s widely used in industries such as healthcare, finance, and autonomous vehicles due to its robust performance and eas[illegible]
[illegible]llows developers to focus more on model development rather than infrastructure setup.

Interface text:
Command Prompt - foundry
Search
ENG
3:43 PM
6/5/2025
AIE
Microsoft
smol ai

### Phi-4 Mini ONNX Runtime demo

Ning loads Phi-4 Mini from the local cache in verbose mode and asks it the same ONNX Runtime question previously given to Qwen. The new response begins in a foreground terminal.

[12:18](https://www.youtube.com/watch?v=l614N5W60ls&t=738s) [A foreground terminal reports that Phi-4 Mini loaded successfully and shows the beginning of an interactive prompt.](https://assets.ai.engineer/sites/aiecode2025/sha256/5cfefa5272cdb400109beea177f4220566c147631b4121ec09ea77837e75ce17.webp)

C:\Users\qining>foundry cache list
Models cached on device:
Alias                 Model ID
phi-4-mini            Phi-4-mini-instruct-cuda-gpu
qwen2.5-1.5b          qwen2.5-1.5b-instruct-cuda-gpu
qwen2.5-1.5b          qwen2.5-1.5b-[illegible]
qwen2.5-7b            qwen2.5-7b-in[illegible]

C:\Users\qining>foundry model run qwen2.5-1.5b
Model qwen2.5-1.5b-instruct-cuda-gpu was found [illegible]
Loading model...
Model qwen2.5-1.5b-instruct-cuda-gpu loaded [illegible]
Interactive Chat. Enter /? or /help for help.
Interactive mode, please enter your prompt
> what's onnx runtime
Onnx Runtime is a powerful and flexible li[illegible]
1. **Flexibility**: Supports multiple backends [illegible]
2. **Performance**: Optimized for high-perform[illegible]
3. **Ease of Use**: Provides easy-to-use APIs [illegible]
4. **Scalability**: Designed to handle large-s[illegible]
5. **Integration**: Seamlessly integrates with [illegible]
6. **Compatibility**: Compatible with differen[illegible]
7. **Documentation and Community Support**: We[illegible]
8. **Cross-Platform Compatibility**: Works acr[illegible]
9. **Model Serving**: Simplifies the process o[illegible]
[illegible]s widely used in industries such as healthcare, finance, and autonomous vehicles due to its robust performance and eas[illegible]
[illegible]llows developers to focus more on model development rather than infrastructure setup.

Foreground terminal:
C:\Users\qining>foundry model run phi-4-mini --verbose
Model Phi-4-mini-instruct-cuda-gpu was found in the local cache.
Unloading existing models. Use --retain true to keep additional models loaded.
Loading model...
Time cost to load model: 00:00:10.3936905
Model Phi-4-mini-instruct-cuda-gpu loaded successfully
Interactive Chat. Enter /? or /help for help.
Interactive mode, please enter your prompt
> what's

Interface text:
Command Prompt - foundry
Microsoft Windows [Version 10.0.22631.5335]
(c) Microsoft Corporation. All rights reserved.
Search
ENG
3:44 PM
6/5/2025
AIE
Microsoft
smol ai

[12:31](https://www.youtube.com/watch?v=l614N5W60ls&t=751s) [Two command windows show local models and the beginning of a Phi-4 Mini answer about ONNX Runtime.](https://assets.ai.engineer/sites/aiecode2025/sha256/f85a190b172168dbcaf5e243e3620cd5680c545d90e6199bd84fe5e43a77ddaf.webp)

C:\Users\qining>foundry cache list
Models cached on device:
Alias                 Model ID
phi-4-mini            Phi-4-mini-instruct-cuda-gpu
qwen2.5-1.5b          qwen2.5-1.5b-instruct-cuda-gpu
qwen2.5-1.5b          qwen2.5-1.5b-[illegible]
qwen2.5-7b            qwen2.5-7b-in[illegible]

C:\Users\qining>foundry model run qwen2.5-1.5b
Model qwen2.5-1.5b-instruct-cuda-gpu was found [illegible]
Loading model...
Model qwen2.5-1.5b-instruct-cuda-gpu loaded [illegible]
Interactive Chat. Enter /? or /help for help.
Interactive mode, please enter your prompt
> what's onnx runtime
Onnx Runtime is a powerful and flexible li[illegible]
1. **Flexibility**: Supports multiple backends [illegible]
2. **Performance**: Optimized for high-perform[illegible]
3. **Ease of Use**: Provides easy-to-use APIs [illegible]
4. **Scalability**: Designed to handle large-s[illegible]
5. **Integration**: Seamlessly integrates with [illegible]
6. **Compatibility**: Compatible with differen[illegible]
7. **Documentation and Community Support**: We[illegible]
8. **Cross-Platform Compatibility**: Works acr[illegible]
9. **Model Serving**: Simplifies the process o[illegible]
[illegible]s widely used in industries such as healthcare, finance, and autonomous vehicles due to its robust performance and eas[illegible]
[illegible]llows developers to focus more on model development rather than infrastructure setup.

Foreground terminal:
C:\Users\qining>foundry model run phi-4-mini --verbose
Model Phi-4-mini-instruct-cuda-gpu was found in the local cache.
Unloading existing models. Use --retain true to keep additional models loaded.
Loading model...
Time cost to load model: 00:00:10.3936905
Model Phi-4-mini-instruct-cuda-gpu loaded successfully
Interactive Chat. Enter /? or /help for help.
Interactive mode, please enter your prompt
> what's onnx runtime
ONNX Runtime is an open-source library for running machine learning models that have been converted to the ONNX (Open Neural Network Exchange) format. ONNX is a format for representing deep learning models that allows them to be shared between different frameworks, tools, and platforms. ON

Interface text:
Command Prompt - foundry
Microsoft Windows [Version 10.0.22631.5335]
(c) Microsoft Corporation. All rights reserved.
Search
ENG
3:44 PM
6/5/2025
AIE
aws

## Full transcript

[0:00](https://www.youtube.com/watch?v=l614N5W60ls&t=0s) [upbeat music] Hello, everybody.

[0:16](https://www.youtube.com/watch?v=l614N5W60ls&t=16s) Uh, my name is Emma. I'm a program mani-- uh, product manager at Microsoft. It's a pleasure to talk to you today about Foundry Local, which enables developers to easily build up cross-platform applications powered by local AI.

[0:35](https://www.youtube.com/watch?v=l614N5W60ls&t=35s) So let's get started. Uh, the first question is, if the cloud AI is so powerful, why do we need local AI?

[0:46](https://www.youtube.com/watch?v=l614N5W60ls&t=46s) So here are four key reasons based on our conversations and observations with our customers. So first of all, how does cloud AI work in an environment with low network bandwidth or even offline access?

[1:04](https://www.youtube.com/watch?v=l614N5W60ls&t=64s) Many of you have experienced the bad internet during this conference, right? And a common sentence we have heard from many speakers with live demo is, "Oh, finger crossed. Hopefully the Wi-Fi connection is good."

[1:20](https://www.youtube.com/watch?v=l614N5W60ls&t=80s) So it's not their fault, uh, it's required by cloud AI. But it's not a concern at all for my session because my live demo runs entirely locally. So reason number two, privacy and security.

[1:36](https://www.youtube.com/watch?v=l614N5W60ls&t=96s) Many companies work with very sensitive data, uh, such as legal document and patient information. They need to process that da- data, uh, entirely locally without anything ever leaving device.

[1:53](https://www.youtube.com/watch?v=l614N5W60ls&t=113s) Right. And reason number three, cost efficiency. Um, think about, uh, game applications, uh, which is deployed to millions of devices. With hundreds of millions inference call every day, it's just not sustainable.

[2:10](https://www.youtube.com/watch?v=l614N5W60ls&t=130s) And reason number three, real-time latency. So many AI applications needs to respond in real time, and it's not just, uh, not possible if we wait on cloud. So that's why we need a local AI.

[2:27](https://www.youtube.com/watch?v=l614N5W60ls&t=147s) Then the next question, whether local AI is ready now.

[2:33](https://www.youtube.com/watch?v=l614N5W60ls&t=153s) So thanks, uh, thanks to the decades of progress in computing, hardware has become more and more powerful. So many client devices are equipped with modern GPUs, NPUs, capable of running advanced AI models.

[2:49](https://www.youtube.com/watch?v=l614N5W60ls&t=169s) Meanwhile, model companies are keeping publishing more and more models, which are leaner, faster, more optimized for local inference, such as, uh, Phi-4 Mini and, uh, DeepSeek, uh, small variants.

[3:07](https://www.youtube.com/watch?v=l614N5W60ls&t=187s) And we are also seeing more and more advanced, um, uh, state-of-art optimization techniques introduced at runtime level. So this convergence now makes the local AI a reality.

[3:22](https://www.youtube.com/watch?v=l614N5W60ls&t=202s) Then how do we build up the solution?

[3:25](https://www.youtube.com/watch?v=l614N5W60ls&t=205s) So Microsoft already, uh, has many, uh, great assets. Uh, Azure AI Foundry, uh, introduced, um, last year at, uh, Microsoft Ignite, uh, has been, uh, trusted by over seventy thousand organizations with over one, uh, thousand and nine hundred models.

[3:44](https://www.youtube.com/watch?v=l614N5W60ls&t=224s) And the ONNX Runtime, our cross-platform high-performance on-device inference engine, have now seen over ten million downloads per month. And our customers are pretty happy with the significant, uh, performance acceleration provided by ONNX Runtime compared to PyTorch.

[4:05](https://www.youtube.com/watch?v=l614N5W60ls&t=245s) And lastly, let's not forget Windows. The scale and the reach of Windows on client devices are massive. So when we think about, uh, democratizing AI, um, the millions of, um, you know, devices, uh, and the millions of customers using Windows devices really matter to us.

[4:27](https://www.youtube.com/watch?v=l614N5W60ls&t=267s) So we are not starting from scratch. We are bringing all those advanced assets into Foundry Local, an optimized end-to-end solution for seamless on-device AI.

[4:42](https://www.youtube.com/watch?v=l614N5W60ls&t=282s) So at the bottom, as you can see, it uses ONNX Runtime to accelerate performance across various kinds of hardware. On the top, uh, we are introducing a new Foundry Local management service, which host and manage model on your, on devi- on client device.

[5:03](https://www.youtube.com/watch?v=l614N5W60ls&t=303s) It also connects to Azure AI Foundry to download open source models on demand.

[5:11](https://www.youtube.com/watch?v=l614N5W60ls&t=311s) And for user experience, we provide Foundry Local CLI, which allows you easily explore models on device, and we also provide SDK so that developers can easily, uh, integrate Foundry Local into your own applications, um, from cloud to local, from different hardware.

[5:32](https://www.youtube.com/watch?v=l614N5W60ls&t=332s) So Foundry Local was officially announced just one month ago at Microsoft Build conference. It's available on both Windows and macOS. On the Windows, it is integrated into the platform, which makes the experience even simpler for Windows AI developer.

[5:55](https://www.youtube.com/watch?v=l614N5W60ls&t=355s) As I just mentioned, uh, Foundry Local accelerates performance across different kinds of silicon. We have been, uh, closely working with hardware vendors, including NVIDIA, Intel, AMD, Qualcomm, to integrate their hardware accelerators into Foundry Local, ensuring the best-in-class performance that you can get on their hardware.

[6:22](https://www.youtube.com/watch?v=l614N5W60ls&t=382s) So before our official announcement, uh, over 100 customers joined our private pre-preview. They have shared valuable feedback on how easy Foundry Local is to use and how good the performance it is.

[6:38](https://www.youtube.com/watch?v=l614N5W60ls&t=398s) So let's hear some of their feedback. [upbeat music]

[6:46](https://www.youtube.com/watch?v=l614N5W60ls&t=406s) Hey there. Tsavo here, CEO and co-founder at Pieces, where we've been on an ambitious journey to give developers artificial long-term memory across the OS. Now, offline-first AI is core to this vision, and in late 2022, we began to explore small language models running at the edge on all major platforms.

[7:05](https://www.youtube.com/watch?v=l614N5W60ls&t=425s) But between rolling our own llama.cpp and then to Ollama, frustrations around versioning, performance, and end-user experience still remained. That was until our recent partnership with Microsoft on their new Foundry Local platform, an end-to-end AI inference solution that offers ease of use and high performance across different hardware.

[7:26](https://www.youtube.com/watch?v=l614N5W60ls&t=446s) In no time, our team went from documentation access to a production-ready build with noticeable improvements in memory management, time to first token, and tokens per second. If you're looking to deploy on-device models, you can't go wrong with Foundry Local. [upbeat music]

[7:42](https://www.youtube.com/watch?v=l614N5W60ls&t=462s) We have been working on AI projects for our customers for several years now. Some clients want to use the latest AI technologies but are restricted from using external services when the information they want to process contains sensitive data.

[7:56](https://www.youtube.com/watch?v=l614N5W60ls&t=476s) Foundry Local is a perfect solution for these scenarios, as it allows us to easily run gen AI models locally. Here, we can see a solution that combines Foundry Local with a speech-to-text service- Hello, my name is Maria Rodriguez.

[8:09](https://www.youtube.com/watch?v=l614N5W60ls&t=489s) My savings account number is [REDACTED:account_number]- [upbeat music]

[8:14](https://www.youtube.com/watch?v=l614N5W60ls&t=494s) ... which also runs locally. One aspect we were really impressed by was the simplicity of the installation and the ease of using the models. With Foundry Local, we can now create hybrid solutions where part of the solution can be run locally. [upbeat music]

[8:28](https://www.youtube.com/watch?v=l614N5W60ls&t=508s) It's been our privilege to work with these customers to improve Foundry Local together. All right, I've talked enough. Who wants to see live demos? [laughs]

[8:41](https://www.youtube.com/watch?v=l614N5W60ls&t=521s) Okay, uh, let's do that. Um, [upbeat music] so first of all, we can, uh, let's see our CLI experience. So our Windows platform, you can, uh, install Foundry Local package using winget, and on macOS, uh, you can use Homebrew commands.

[9:01](https://www.youtube.com/watch?v=l614N5W60ls&t=541s) Um, so I have already installed Foundry Local, so...

[9:12](https://www.youtube.com/watch?v=l614N5W60ls&t=552s) So first, uh, we want to see what models supported by Foundry Local. So we can type foundry

[9:23](https://www.youtube.com/watch?v=l614N5W60ls&t=563s) model list. So as you can see, it supports many popular, uh, generative AI models, and for each model, you can get, uh, you can get different variants optimized for different hardware.

[9:39](https://www.youtube.com/watch?v=l614N5W60ls&t=579s) So, uh, you can see we have, uh, optimization version for CPU, for CUDA, for integrated GPU. Uh, we also provide NPU variants because, um, my device doesn't, um, contain Qualcomm NPU, so that variant doesn't show up.

[9:56](https://www.youtube.com/watch?v=l614N5W60ls&t=596s) Okay, so we want to run some models, right? And, um, if you haven't downloaded model, uh, for, before, the Foundry Local will download the model from the cloud and then run the model.

[10:08](https://www.youtube.com/watch?v=l614N5W60ls&t=608s) Um, it requires internet. But, uh, I have already pre-installed, uh, pre-downloaded model, so we don't need that. Uh, so we're gonna see what model I have already downloaded using foundry cache list.

[10:24](https://www.youtube.com/watch?v=l614N5W60ls&t=624s) So as you can see, I have downloaded, uh, four models here. So I want to, like, during our experiments, I want, uh, we w- might want to explore different models to see, um, the quality, to see the performance, then decide which model we want to use to build up our application, right?

[10:41](https://www.youtube.com/watch?v=l614N5W60ls&t=641s) So firstly, um, I want to try our foundry

[10:47](https://www.youtube.com/watch?v=l614N5W60ls&t=647s) model run Qwen two point five one, one point five billion model.

[11:04](https://www.youtube.com/watch?v=l614N5W60ls&t=664s) Since I have already downloaded this model, so the model loading is pretty quick. Uh, should be, uh, pretty quick. [laughs] Okay, uh, the model is set up. You can talk to the model directly.

[11:15](https://www.youtube.com/watch?v=l614N5W60ls&t=675s) So let's, uh, ask a simple question. What's ONNX Runtime?

[11:22](https://www.youtube.com/watch?v=l614N5W60ls&t=682s) Oh, it's pretty quick, right? Um, so I think, uh, we may want to see the, mm, latency number. So let's, uh, exit here and rerun it with, sorry,

[11:37](https://www.youtube.com/watch?v=l614N5W60ls&t=697s) verbose mode. And same question. Okay. So here we get around the ninety tokens per second. Um, we also want to try a different model, so let's do that.

[11:53](https://www.youtube.com/watch?v=l614N5W60ls&t=713s) Um, we want to try foundry model run,

[12:03](https://www.youtube.com/watch?v=l614N5W60ls&t=723s) uh, this Phi-4 Mini. Also with verbose mode.

[12:19](https://www.youtube.com/watch?v=l614N5W60ls&t=739s) So it's loading the model. Okay, model is set up. Same question what's ONNX Runtime. So Phi-4 Mini is, um, more advanced than, like, Qwen model.

[12:39](https://www.youtube.com/watch?v=l614N5W60ls&t=759s) The model size is, um, bigger than, uh, the Qwen model. So, uh, I would say in terms of the performance, it, uh, is a bit smaller than Qwen model.

[12:49](https://www.youtube.com/watch?v=l614N5W60ls&t=769s) But, uh, in terms of the quality, as you can see, uh, Phi-4 Mini, uh, can provide more detailed information. Uh, all right. So, uh, personally, uh, I vote, um, Phi-4 Mini, so I want to use this model to build up an application.

[13:06](https://www.youtube.com/watch?v=l614N5W60ls&t=786s) So what application do we want to build? Uh, I guess many of you have such experience. Your team move to a new organization either to ramp up the existing project very quickly, and there are many long, detailed document, uh, documents, uh, you need to read.

[13:25](https://www.youtube.com/watch?v=l614N5W60ls&t=805s) And it's very time-consuming to read every words, right? So you may want some high-level summarized version of, um, all of this project so you can quickly ramp up. And...

[13:36](https://www.youtube.com/watch?v=l614N5W60ls&t=816s) But this project is an internal project. You cannot upload all those documents to cloud. And meanwhile you have, you know, some of your team members are using Windows, some of your team members are using Mac.

[13:49](https://www.youtube.com/watch?v=l614N5W60ls&t=829s) So you want to build a application cross-platform powered by local AI. So let's do that.

[13:58](https://www.youtube.com/watch?v=l614N5W60ls&t=838s) So I have this application setting up. Um, so what it does, um, so we can run it first.

[14:14](https://www.youtube.com/watch?v=l614N5W60ls&t=854s) Oh, um, and let's click the existing conversation.

[14:28](https://www.youtube.com/watch?v=l614N5W60ls&t=868s) All right, um, so the app is setting up. Basically, it is used to summarize content. Uh, you can give it, uh, your URL, or you give the local file, so it can do summarization.

[14:40](https://www.youtube.com/watch?v=l614N5W60ls&t=880s) And it also has a setting tab. You can choose the model you want to run. Uh, as mentioned before, I prefer Phi-4 Mini, so because I want to get some, uh, more detailed information.

[14:54](https://www.youtube.com/watch?v=l614N5W60ls&t=894s) So, uh, we can put, uh, this model ID

[15:00](https://www.youtube.com/watch?v=l614N5W60ls&t=900s) here. Then, uh, we'll pass it with our project document, and let it to give me some high-level information.

[15:21](https://www.youtube.com/watch?v=l614N5W60ls&t=921s) So I will hit summarization. So as you can see, the summar-uh, summarized version is coming out and, uh, it says, "Foundry Local is useful to build up cross-platform AI applications that run directly on device."

[15:35](https://www.youtube.com/watch?v=l614N5W60ls&t=935s) That's pretty cool. And, uh, then let's take a look at the code quickly. So as you can see, uh, we, in terms of SDK, we, um, provide a Python SDK and a JavaScript, uh, JavaScript, uh, SDK.

[15:52](https://www.youtube.com/watch?v=l614N5W60ls&t=952s) So here we use a JavaScript one. So we, uh, create the Foundry Local manager and, uh, we...

[16:02](https://www.youtube.com/watch?v=l614N5W60ls&t=962s) And we initial this manager. Oh, um, just a second. What I say?

[16:17](https://www.youtube.com/watch?v=l614N5W60ls&t=977s) Yeah. So initial the manager with the model name. Um, so as you can see, I passed Phi-4 Mini here. And then, um, it use, uh, Foundry Local endpoint to create a client, and then you just wait, uh, for the chat to, uh, be complete, uh, to be completed, and output the result.

[16:36](https://www.youtube.com/watch?v=l614N5W60ls&t=996s) So, um, this is the application, uh, running on Windows. Uh, I, some of, um, one team member is using Mac, so I want, um, he to use on my app as well.

[16:49](https://www.youtube.com/watch?v=l614N5W60ls&t=1009s) So I package the whole project and share to him, and let's see what his experience is.

[16:59](https://www.youtube.com/watch?v=l614N5W60ls&t=1019s) He take the project and record a demo for me. So as you can see, the exactly, uh, same code and, uh, he just, uh, use the same command in npm run start to start this, um, application.

[17:15](https://www.youtube.com/watch?v=l614N5W60ls&t=1035s) And exactly UI, exactly application and, oh, he choose Qwen model. Um, maybe he like this model more. And he also use the same document, uh, documents I used, uh, in my previous demo, and hit Summarize button.

[17:35](https://www.youtube.com/watch?v=l614N5W60ls&t=1055s) So, uh, as we, uh, see in the previous demo, uh, Qwen model is kind of, uh, provide more brief information, um, than, uh, Phi-4 Mini model. So as you can see here, it also shows the summarization is more, like, shorter than what, uh, Phi-4 Mini provides.

[17:55](https://www.youtube.com/watch?v=l614N5W60ls&t=1075s) Okay. So we build up cross-platform applications. Um, is that all my demo? Of course not. We forgot one important thing, agent, right? So everybody talks about agent, so we-- So do I.

[18:11](https://www.youtube.com/watch?v=l614N5W60ls&t=1091s) So Foundry Local enables you to easily create and bu- and run a local agent using local model and MCP servers. Um, this feature is still in private preview, but I want to give you a slip, um, gi- give you a quick, uh, look so you know, uh, how it works.

[18:33](https://www.youtube.com/watch?v=l614N5W60ls&t=1113s) So let's back to our, uh, CMD. So, uh, we can u-

[18:41](https://www.youtube.com/watch?v=l614N5W60ls&t=1121s) uh, use foundry agent list to show all the available agents, uh, in a Foundry Local. As you can see, we have built up, uh, three sample agents here. So in terms of the concept, an agent in a Foundry Local, um, consists of one model and one or more MCP servers based on your

[19:05](https://www.youtube.com/watch?v=l614N5W60ls&t=1145s) need. So you can use, um, one model from the list and pick up, uh, whatever MCP server you like to create your own agent. But here, um, we want to run the existing one.

[19:17](https://www.youtube.com/watch?v=l614N5W60ls&t=1157s) So I'm interested in this OCR agent, so, uh, let's see, uh, foundry agent info, uh, to know what it can do.

[19:35](https://www.youtube.com/watch?v=l614N5W60ls&t=1175s) Okay, so it can extract the text from images in your local device and this agent, um, contains one, uh, model, which is the Phi-4 Mini Meta V1 and the two MCP servers.

[19:49](https://www.youtube.com/watch?v=l614N5W60ls&t=1189s) One is file system MCP server, one OCR Mini MCP server. So let's run this, uh, agent, foundry agent run.

[20:03](https://www.youtube.com/watch?v=l614N5W60ls&t=1203s) So this command will, uh, check the dependencies of this agent first. If, um, the dependency hasn't been installed before, it will, uh, installed with your permission. So I have already installed all those, uh, dependencies, so, uh, it just run.

[20:20](https://www.youtube.com/watch?v=l614N5W60ls&t=1220s) Okay. The agent is setting up. Uh, it ask for permission to use this MCP server, so let's say yes.

[20:29](https://www.youtube.com/watch?v=l614N5W60ls&t=1229s) And he ask a directory you want it to access. Uh, I give it, um, the demo photo. And he also ask a permission for the OCR MCP server.

[20:45](https://www.youtube.com/watch?v=l614N5W60ls&t=1245s) I will say yes. So all right, so it-- from here you can get all the tools supported by this agent. So literally the tools, um, provided in the MCP servers.

[21:00](https://www.youtube.com/watch?v=l614N5W60ls&t=1260s) So you can get some tools related to file system, tools related to OCR. So, uh, what we want it to do, so here

[21:11](https://www.youtube.com/watch?v=l614N5W60ls&t=1271s) is the user requirement. I want it to find my receipt, um, process it, and get the total amount from it. So let's see, uh, whether it can complete this task or not.

[21:26](https://www.youtube.com/watch?v=l614N5W60ls&t=1286s) So it starts to thinking because it needs to figure out which tool to use. Okay, uh, the first tool to use is search file, because it nee- he, it needs to find the receipt and then, uh, he figure out to use the, um,

[21:42](https://www.youtube.com/watch?v=l614N5W60ls&t=1302s) after search, it'll use the OCR one to, uh, extract the text and then get the output, get the total amount. Okay, that's cool. So that's all my demo. Um, so finally, I know it's a little bit over time, but I just quickly wrap up.

[22:00](https://www.youtube.com/watch?v=l614N5W60ls&t=1320s) So Foundry Local enables you to build up, um, applications powered by local AI. And, uh, one best practice, um, so local model generally are not, uh, ca- uh, not that capable, uh, as cloud model.

[22:18](https://www.youtube.com/watch?v=l614N5W60ls&t=1338s) So, um, you cannot expect it to do the fancy work that cloud model or cloud agent can do, but it has unlock a lot of potential. So I leave that to you guys to explore.

[22:32](https://www.youtube.com/watch?v=l614N5W60ls&t=1352s) Uh, if you want to get more information, uh, here is the link, and you want to, uh, try out our agent, uh, feature, you can sign up our, you know, our private preview form.

[22:44](https://www.youtube.com/watch?v=l614N5W60ls&t=1364s) All right, uh, thanks, everyone. [outro music]
