# Making Open Models 10x faster and better for Modern Application Innovation

Lin Qiao, Fireworks · Dmytro (Dima) Dzhulgakov

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/C1CXwRYbwuQ-making-open-models-10x-faster-better-modern) · [Video](https://www.youtube.com/watch?v=C1CXwRYbwuQ)

## Making Open Models 10x faster and better for Modern Application Innovation

Specialized models need specialized serving. Fireworks connects workload-aware inference, shared LoRA deployment, and function calling into a practical path from prototype to production.

Familiarity with language-model inference, tokens, retrieval-augmented generation, and API calls will help you follow the serving and function-calling examples.

### What does a customer-support chatbot actually need?

[0:15](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=15s)

What should a customer-support application pay for: broad general knowledge, or expertise in its own domain? That question anchors Dmytro “Dima” Dzhulgakov’s discussion of production inference at Fireworks. Presenting in place of cofounder Lin Qiao, he describes a team drawn from Meta’s PyTorch leads and Google AI veterans, with roughly a decade of production AI experience and five years of his own work maintaining PyTorch. Open deep-learning tools provide the precedent for the team’s focus on open generative models.

An audience show of hands reveals substantial open-model usage in the room. Dima nevertheless describes the broader production market at the time as dominated by proprietary models, often with frontier capabilities. **GPT-4** and **Claude** can handle many different tasks, but their breadth and serving infrastructure do not necessarily fit each application equally well. Batch inference over a narrow domain and a real-time voice assistant have different requirements, even when they call the same model.

The customer-support chatbot makes the mismatch concrete: it does not need knowledge of 150 Pokémon or the ability to write poetry. It needs reliable answers about the business it supports. **The useful unit of optimization is the application’s task, not the model’s total range of capabilities.** Paying for abilities the application never uses can make a broadly excellent model an expensive fit.

![GPT-4 branches to Chat, Code generation, and Agents above the statement: Best quality off-the-shelf, but high costs and poor UX in production.](https://ai.engineer/api/talk-essay-media/11bcaa6d27f1685cef30788a771685f50fa928df0b5de08669f86f4d843b4a66)

Big models: broad capabilities with high costs and poor production UX.

### Repeated calls amplify latency and cost

[3:06](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=186s)

Large-model response times become especially visible in voice assistants. Agentic applications compound the problem: a system such as Devin may reason through several steps and call the model repeatedly before completing a task. Each dependent call adds another wait. Dima reports that smaller **Llama** or **Gemma** models can match or exceed quality on a narrow task while running up to 10 times faster.

For function calling, Dima cites a Berkeley benchmark and reports similar quality from fine-tuned **Llama 3** at 10× the speed. The accompanying slide shows GPT-4 at 1.97 seconds and Llama 3 at 0.18 seconds, with a Berkeley Function-Calling Leaderboard credit. The recording does not specify the hardware, concurrency, request lengths, or exact leaderboard snapshot behind this comparison; those conditions matter when applying the result to another workload.

![Latency slide shows GPT-4 at 1.97 seconds and Llama 3 at 0.18 seconds, with a 10x arrow and a Berkeley Function-Calling Leaderboard source credit.](https://ai.engineer/api/talk-essay-media/356215569f3825ddb6a3c8a4a9ade56a9ffd647b158e8682e6b24673bac1f32d)

Latency comparison for GPT-4 and Llama 3.

Traffic makes the economic question equally concrete. Dima’s example uses a 5,000-token prompt, 10,000 users, and 20 calls per user per day, mentioning GPT-4 and **GPT-4o**. The input volume is:

```math
\begin{aligned}
T_{\text{input/day}}
&= 5{,}000 \times 10{,}000 \times 20 \\
&= 1{,}000{,}000{,}000\ \text{tokens/day}.
\end{aligned}
```

At GPT-4o’s [May 2024 launch price](https://community.openai.com/t/announcing-gpt-4o-in-the-api/744700) of $5 per million input tokens, that is $5,000 per day for input alone; output length is unspecified. Dima’s spoken estimate of approximately $10,000 per day would amount to $300,000 per 30-day month or $3.65 million per year, rather than several million monthly. The underlying motivation remains: repeated calls at substantial traffic make smaller, customizable models economically interesting.

### Open weights make adaptation possible, not automatic

[4:39](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=279s)

Open models offer two routes to domain fit: choose an existing specialization or create one. Dima points to Gemma adaptations for Indian languages, models specialized for code or medicine, and the tens of thousands of variants available on Hugging Face. Access to weights makes it possible to tune quality for the application’s particular requirements.

That flexibility brings three kinds of operational work:

- **Setup and maintenance:** Find GPUs, select a framework, download the model, and tune its performance. Model updates can require repeating the process.
- **Workload optimization:** Low latency and high throughput are different objectives. Short and long prompts also favor different settings; Dima reports multiple-fold improvements from choosing those settings effectively.
- **Production reliability:** Cloud GPUs need operational attention. Enterprise serving adds scaling, telemetry, and observability beyond simply getting a model to answer a request.

These are the problems Fireworks sets out to absorb for application developers.

### Optimize the serving stack around a latency budget

[6:38](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=398s)

Fireworks’ approach spans custom **CUDA kernels**, runtime behavior, deployment, and service orchestration. Dima describes customization across that stack for each workload’s cost and latency requirements. A RAG application with long, repeatable prompts is one example: caching can avoid repeated work, while runtime and deployment settings can raise throughput without giving up the required response time.

Dima reports that Fireworks ranked fastest under Artificial Analysis’s Long Prompt selection at the time. He also describes Fireworks as the fastest provider serving **SDXL**, and as the sole provider serving **SD3** because Stability’s API routed to its servers. These are historical provider claims; [Stability’s SD3 API announcement](https://stability.ai/news-updates/stable-diffusion-3-api) confirms the delivery partnership, but does not establish the separate speed ranking or universal exclusivity.

A more useful optimization target than maximum speed alone is **minimum cost subject to an application’s latency constraint**. Dima gives the example of an interactive application that must generate its required number of tokens in under two seconds. The output length is application-specific, so the constraint must be evaluated with that length fixed:

```math
\begin{aligned}
\min_{\text{serving configuration}}\quad
& C_{\text{serving}} \\
\text{subject to}\quad
& L_{\text{required output}} < 2\ \text{seconds}.
\end{aligned}
```

Dima reports that tuning the latency cutoff and settings across the stack can increase throughput multiple-fold. Higher throughput means fewer GPUs are needed to serve the same traffic, reducing cost while preserving the response-time requirement.

### Serve custom models without a GPU for every variant

[8:50](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=530s)

The model catalog described in the talk includes Gemma, Llama, automatic speech recognition, and text-to-speech models. Dima also names **Yi-Large**, whose US serving on Fireworks he describes as having launched the previous week. Beyond hosting models, Fireworks produces function-calling fine-tunes and releases work combining vision and language models. Developers can bring a fine-tuning dataset collected elsewhere or build one from feedback gathered during live serving.

Training a custom variant creates another question: how should it be served economically? With **LoRA**, customization can be represented by an adapter associated with a base model. Dima says Fireworks can deploy thousands of LoRA variants on the same GPU. Sharing serving resources lets the platform offer per-token serverless inference without a fixed serving charge for each deployed variant, rather than requiring developers to reserve separate hardware for every customization.

### The model becomes one part of the system

[10:33](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=633s)

A capable model still does not constitute a complete application. Hallucinations motivate grounding through RAG or external knowledge bases. Gaps across modalities motivate chaining different model types. Applications that act on the world also need tools and external APIs. Dima uses the term **compound AI system**, crediting Databricks with popularizing it, for this combination of models, retrieval, and actions.

In the architecture he describes, a central function-calling model interprets and triages the user’s request. It can route work to domain-specialized models, invoke a code interpreter, or call external APIs. Function calling—initially popularized by OpenAI, as Dima notes—is the connection between the language model and those external capabilities.

Fireworks fine-tunes models specifically for that coordinating role. The demonstration uses **FireFunction**, the latest release of which Dima places about two weeks before the talk. Its job combines ordinary conversation with deciding when and how to request a tool operation.

### From cloud providers to a stock chart

[12:59](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=779s)

The demo begins with freeform conversation: the user can ask what the application can do, and it describes its capabilities. Those capabilities include querying stock quotes, plotting charts, and generating images through external tools. The interesting work is translating a loosely specified request into a sequence of concrete calls.

A request for a bar chart of the Big Three cloud providers requires several steps:

1. **Resolve the entities.** Interpret the providers as AWS, GCP, and Azure, then connect the services to their publicly traded parent companies—for example, Azure to Microsoft.
2. **Retrieve the data.** Call stock-price tools for the relevant companies.
3. **Combine the results.** Pass the retrieved prices to the chart-plotting API.

Dima describes the chart appearing after this sequence runs in the background. The model supplies the interpretation and coordination; the external tools supply the stock data and render the chart.

The next request adds Oracle to the existing graph. The application must preserve the earlier chart context, resolve the new company, and regenerate the chart with the additional entry. A compact JSON representation of the requested edit could look like this:

```json
{
  "chart_type": "bar",
  "existing_companies": ["Amazon", "Alphabet", "Microsoft"],
  "company_to_add": "Oracle"
}
```

This represents the requested change, not retrieved prices or an executed tool result. It makes the contextual requirement explicit: the earlier companies remain part of the chart.

Then the user changes topics and requests a cat photo. Now the stock-chart context should stop influencing the task: Oracle does not belong in the generated cat image. Useful conversational memory therefore requires both retention and selective disregard. The demonstration combines FireFunction with other hosted models, including SDXL; Dima makes the application available as open-source demo code, with [Fireworks Demo Apps](https://github.com/fw-ai/forge/blob/main/README.md) providing a companion entry point.

### Connect model decisions to structured interfaces

[14:58](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=898s)

The function-calling model’s weights are available on Hugging Face, allowing developers to choose hosted Fireworks inference or local execution. The [FireFunction v2 model repository](https://huggingface.co/fireworks-ai/llama-3-firefunction-v2) distributes its weights under the Llama 3 license. Weight availability gives developers deployment choices; it should not be read as unrestricted licensing.

Fireworks also provides **structured generation** through JSON mode and grammar mode, which Dima compares to the capabilities discussed in the preceding Outlines talk. These modes constrain the form of generated output so that model decisions can pass through structured interfaces to the rest of the application. The architecture diagram places FireFunction in JSON/Grammar mode above two destinations: Fireworks inference—including language, audio, embedding, image, multimodal, and tuned models—and external tools and APIs.

![Compound AI Inference Engine diagram places FireFunction in JSON/Grammar mode above Fireworks Inference and Tools and APIs. Inference includes LLM, audio, embedding, image, multi-modal, and tuned models.](https://ai.engineer/api/talk-essay-media/fbc4870fea38cba594f30fb68fadd043f5c06589f969734c4e6aca65adba14ca)

FireFunction combines model inference with tools and APIs.

### From playground to dedicated serving

[15:25](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=925s)

The starting point is the Fireworks model catalog and playground. From there, the deployment options described in the talk progressively expose more control over serving:

| Offering | Billing and infrastructure | Customization |
|---|---|---|
| Serverless | Per-token pricing; no hardware setup | Hosted fine-tunes and imported LoRA adapters |
| On-demand | Dedicated hardware | More workload settings and custom models |
| Enterprise | Discounted long-term contracts | Personalized hardware and performance tuning |

Serverless keeps the first deployment simple without excluding fine-tuning. On-demand provides dedicated resources as workload-specific control becomes useful. At larger volumes, the enterprise offering adds help tuning the hardware and serving configuration. These are the historical product choices described in the recording.

### Production workloads and existing clients

[16:50](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=1010s)

Dima reports that Fireworks was serving more than 150 billion tokens per day as of his last check. He names Quora’s Poe as a chatbot example, Sourcegraph and Cursor as users of some code-assistant functionality where latency matters, and Upstage and Liner as builders of assistants and agents. These examples connect the earlier serving requirements to applications already handling production traffic.

For developer adoption, Dima cites the previous year’s LangChain State of AI material, describing Fireworks as a leading source for models after Hugging Face. His suggested entry point remains practical: try Llama or Gemma in the playground, then build with FireFunction or a fine-tune suited to the application.

The final integration point is **OpenAI API compatibility**. Dima describes keeping familiar clients and using frameworks such as LangChain or LlamaIndex when moving to Fireworks. That preserves the application’s existing integration approach while opening up choices over model specialization and serving configuration.

![Slide reads Make the switch with a 1-line code change, with a red outline around a model setting in a code example and LangChain and LlamaIndex logos below.](https://ai.engineer/api/talk-essay-media/0dd6d9c11951a7aa1ba75ac5090570963caed04b6321c474363b4a9f6fb174ed)

A highlighted model setting alongside LangChain and LlamaIndex integration options.

## Resources

- [FireFunction v2 release and evaluation](https://fireworks.ai/blog/firefunction-v2-launch-post) — Historical release explaining the Llama 3 fine-tune, conversational tool use, benchmark results and serving measurements.
- [FireFunction v2 model repository](https://huggingface.co/fireworks-ai/llama-3-firefunction-v2) — Model weights and loading examples for the Llama 3-based function-calling model, distributed under the Llama 3 license.
- [Fireworks Demo Apps](https://github.com/fw-ai/forge/blob/main/README.md) — Collection of Fireworks demonstrations with an entry point to the Functional chat application.
- [Stability's SD3 API announcement](https://stability.ai/news-updates/stable-diffusion-3-api) — Stability's announcement of SD3 and SD3 Turbo API delivery in partnership with Fireworks.

## Slides

### Presenting Sponsor

Microsoft is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=5s) [Microsoft presenting sponsor slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/5d3d51b7f4395ac7638521872389073b482489fa8b12ec0e10957e6f0474a758.webp)

PRESENTING SPONSOR

Microsoft

### Innovation and Platinum Sponsors

AWS is the innovation sponsor; MongoDB, Google Cloud, and Neo4j are platinum sponsors.

[0:10](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=10s) [AWS innovation sponsor; MongoDB, Google Cloud and Neo4j platinum sponsors.](https://assets.ai.engineer/sites/aiecode2025/sha256/62ce08afe4cb2429e2822aaa781ef1a595d43f100b09b8528b663310ba0fad78.webp)

INNOVATION SPONSOR

aws

PLATINUM SPONSORS

MongoDB®
Google Cloud
neo4j

### Fireworks AI: Customized Production-Ready Inference

Dmytro Dzhulgakov introduces Fireworks AI’s focus on productionizing and customizing open-source models for inference.

[0:15](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=15s) [Title slide on customized inference with open-source models, presented by Dmytro Dzhulgakov.](https://assets.ai.engineer/sites/aiecode2025/sha256/3592988e5953db96e9dc9895a8acf56ff6717a3b04af353590248cae3e8791ae.webp)

Fireworks AI

Customized, production-ready inference with open-source models

Dmytro (Dima) Dzhulgakov, CTO & co-founder

Fireworks AI

### Fireworks: Productionizing Open-Source LLMs

Four portraits introduce the Fireworks team. Its founding team includes PyTorch leads from Meta and Google AI veterans.

[0:49](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=49s) [Four Fireworks team portraits with references to PyTorch and Google AI.](https://assets.ai.engineer/sites/aiecode2025/sha256/392e1e353e2b5ecdd5451a854657d0c266f475d28620b12519978245f29c7765.webp)

Fireworks - productionizing open-source LLMs

Fireworks Team

PyTorch & Google AI

4

### Going Big or Small?

The speaker introduces the trade-off between big and small models, noting that proprietary models dominate production inference and often lead in many domains.

[1:39](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=99s) [Purple title slide reading “Going Big or Small?”](https://assets.ai.engineer/sites/aiecode2025/sha256/3d19d8597ace0d515ee8c0372451b6918b8a091a918e26e483fe6c5711702fbc.webp)

Going Big or Small?

7

### Latency

The slide compares GPT-4 at 1.97 seconds with Llama 3 at 0.18 seconds. The speaker says latency matters for agentic applications that call a model many times.

[3:06](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=186s) [Latency slide comparing GPT-4 at 1.97 seconds with Llama 3 at 0.18 seconds, labeled 10x.](https://assets.ai.engineer/sites/aiecode2025/sha256/afa8b7369005aaa8bd846a73f7f6ef9fbcd6a933adfffe24db52ebcf8b65e302.webp)

Latency

Slow processing times hinder real-time performance

Impact: Low user engagement & product adoption

GPT-4 vs Llama 3 Latency (sec)
Y-axis ticks: 0.0, 0.5, 1.0, 1.5, 2.0
GPT-4: 1.97
Llama 3: 0.18
10x

Source: Berkeley Function-Calling Leaderboard

10

### Domain Adaptability

General-purpose models can lack specialized knowledge. Open models offer many specialized variants, and their open weights let developers customize them for a particular use case.

[4:43](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=283s) [General-purpose models may lack specialized knowledge, producing generic responses; the slide cites over 50,000 open-source GenAI models.](https://assets.ai.engineer/sites/aiecode2025/sha256/6b18f1e874066d5859ed50f58017e8b8e00d778cb84453ae0d57fce281ec8b0e.webp)

Domain Adaptability

General-purpose models typically lack knowledge in specialized areas

Impact: Generic & irrelevant responses, no moat

50k+
Open-source GenAI models

12

### Challenges Using Open Source Models in Production

Running open models requires repeated setup and maintenance, tuning for workload needs, and production infrastructure such as scaling and observability.

[5:09](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=309s) [Three production challenges: complex setup, lack of optimization, and lack of production readiness.](https://assets.ai.engineer/sites/aiecode2025/sha256/46e6a1bc52d50445ac82fbe565d89a6058ee2cc22ba44f5aa3c89a53b444d760.webp)

Using open source models in production is still challenging

Complex Set-up
Not Optimized
Not Production-Ready

16

### Fireworks AI Inference Engine

Fireworks introduces its fast and efficient AI inference engine.

[6:32](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=392s) [Purple title slide introducing Fireworks AI's fast and efficient AI inference engine.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a20afbc0a0d8fd390de8e9f35be9db92892cd411547590021239220b2c2e135.webp)

Fireworks AI
Fast and Efficient AI Inference Engine

17

### The Fastest Serving Stack, Customized to You

Fireworks built its serving stack from CUDA kernels through deployment orchestration, then tunes hardware and software for each workload’s cost and latency requirements.

[6:37](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=397s) [FireAttention serving stack features include quantization, speculative decoding, semantic caching, and distributed inference, with hardware and software tuned to the workload.](https://assets.ai.engineer/sites/aiecode2025/sha256/bb97bc6060ff644f23fcd8df7d82cd3673e58b1237e406f9bbc2d675949b468b.webp)

The fastest serving stack, customized to you

• FireAttention serving stack
  ◦ Quantization without quality impact
  ◦ Speculative decoding
  ◦ Semantic caching
  ◦ Distributed inference
• Customized serving for you
  ◦ Hardware and software stack tuned for your workload and goals

18

### Image Generation Speed

In ArtificialAnalysis.ai’s SDXL 1.0 benchmark, Fireworks generates one image in 1.1 seconds, compared with 9.8 seconds for Replicate. The slide marks the difference as 9x.

[7:52](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=472s) [Bar chart compares SDXL image-generation times, showing Fireworks at 1.1 seconds and other providers at 1.9 to 9.8 seconds.](https://assets.ai.engineer/sites/aiecode2025/sha256/13fd5c473a88d46dac2ccddda9033b2896d2c609bf3464fefab663c334c375bb.webp)

Image Generation Speed

Generation Time
Generation time: Seconds to generate 1 image, Lower is better

Independent benchmark from ArtificialAnalysis.ai
SDXL 1.0 Image Generation

ArtificialAnalysis.ai
9x

SDXL 1.0, Fireworks: 1.1
SDXL 1.0 (SDXL), fal.ai: 1.9
SDXL 1.0, Together.ai: 4.5
SDXL 1.0, Amazon Bedrock: 6.3
SDXL 1.0, Stability.ai: 6.6
SDXL 1.0, Replicate: 9.8

### More Speed and Output for Your Dollar

Fireworks FP8 leads throughput at all three latency cutoffs; the slide claims 8-15X higher throughput. The speaker links higher throughput to fewer GPUs and lower cost.

[8:07](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=487s) [Grouped bars compare vLLM and Fireworks throughput at one-, two- and three-second cutoffs; the slide claims 8–15 times higher throughput.](https://assets.ai.engineer/sites/aiecode2025/sha256/448b15380bc89f5a2bf87a240614c3680ace20b383f1cbbfd0a8285bf9fd8643.webp)

More speed and output for your $

8-15X higher throughput

Legend: vLLM Quant; vLLM FP16; Fireworks FP16; Fireworks FP8
Y-axis: requests per second
Y-axis ticks: 0, 20, 40, 60, 80
X-axis: 1 sec cutoff; 2 sec cutoff; 3 sec cutoff

21

### Best Quality Small Models

Fireworks presents support for 100+ models from multiple providers, including Llama 3, Whisper, SD3 and Gemma.

[8:48](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=528s) [Model providers and examples include Llama 3, Whisper, SD3, Gemma, Phi-3 Vision and Mixtral; the slide states that more than 100 models are available.](https://assets.ai.engineer/sites/aiecode2025/sha256/d5ce3411477078da4b99eba330657635e3da2baf1ba1150190bbf03a9bf7718b.webp)

Best Quality Small Models

Meta — llama3
whisper
stability.ai — SD3
Google — gemma
Microsoft — phi-3-vision
MISTRAL AI_ — mixtral

100+ top quality models

### Best Application Quality Customized

Fireworks offers open-source, custom and proprietary models, plus fine-tuning with customer data for specific use cases.

[9:19](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=559s) [Diagram shows open-source and custom models, Fireworks proprietary models, and customer data used to create fine-tuned models.](https://assets.ai.engineer/sites/aiecode2025/sha256/743b1c3e13353b40581014464253a708f489b319fe42b09ec4e3e4c1325a7eda.webp)

Best Application Quality Customized
Use the best model for each use case

100+ OSS & Custom Models

Fireworks Proprietary Models

Your data
Fine-tuned models

Sticky, differentiated value

### Customize Small Models for Production

Fireworks batches requests across base and fine-tuned models using shared base weights. The slide claims 1000x more efficient LoRA serving.

[9:54](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=594s) [Diagram contrasts separate batches for base and fine-tuned models with a shared cross-model batch; the slide claims 1,000 times more efficient LoRA serving.](https://assets.ai.engineer/sites/aiecode2025/sha256/9bc40f0d7701538451fa8de0fc5ac81d130c5021e3fa8bad095ffb22f6278582.webp)

Customize Small Models for Production

Naive serving (model=GPU)
request stream
batch (base)
base weights
batch (A)
fine-tuned A
batch (B)
fine-tuned B

Fireworks w/ cross-model batching
request stream
batch (cross-model)
A
B
base weights

1000x more efficient LoRA serving!

### Single Model is Limited!

Models can hallucinate, have specialized strengths, and have limited or outdated knowledge. The speaker suggests grounding responses in external knowledge and chaining models across modalities.

[10:35](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=635s) [Slide identifies hallucination, specialized strengths, and limited or outdated knowledge as model limitations.](https://assets.ai.engineer/sites/aiecode2025/sha256/6a3cf803c7a3f8640dacda34696543b456e71eaf8f219b759d4707e6434c8860.webp)

Single Model is Limited!

LLM   Audio
Embedding
Image   Multi-modal

API

Hallucination
Specialized strength
Limited/outdated knowledge

26

### Compound AI System

A single model gives way to connected models, data and tools. The speaker describes products combining RAG, function calling and external tools.

[11:15](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=675s) [Diagram contrasts a single neural network with connected models, data, and tools.](https://assets.ai.engineer/sites/aiecode2025/sha256/ba268edec0db98cd0db46d90b8a96064ce2d4f671a0a733d1bc13dd6eb519c6a.webp)

Compound AI System

Increasingly many new AI results are from compound systems.

27

### Compound AI Inference Engine

FireFunction sits above Fireworks inference models and external tools and APIs. The speaker describes a central function-calling agent coordinating user requests across them.

[11:41](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=701s) [FireFunction connects Fireworks inference models with tools and APIs.](https://assets.ai.engineer/sites/aiecode2025/sha256/9e8c88cd70649a103281c4c896f33dcfc2050678497525299ad030ee7155d05c.webp)

Compound AI Inference Engine

FireFunction
(JSON/Grammar mode)

LLM    Audio
Embedding
Image    multi-modal
Tuned models
Fireworks Inference

API
Tools and APIs

### What is function calling?

Function calling connects models to external information and tools. A stock-price exchange illustrates the idea, which the slide links to real-time agents, RAG, tool use and model routing.

[12:26](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=746s) [Slide explains connecting models to external information and tools, using a stock-price example.](https://assets.ai.engineer/sites/aiecode2025/sha256/27915d4f8985f24ed64b5695d2d8fa3b9bf1a89d3efb180e1739de0dc1b1b4f8.webp)

What is function calling?

• Functions give models the ability to interface with external information and tools

What's the stock price of Nvidia?
The stock price of Nvidia is 130.78.

Transformative, “compound” AI:
Real-time agents, RAG and tool usage, model routing

31

### FireFunction V2 demo capabilities

In the demo, FireFunction answers a general chat question by listing available tasks: image generation, chart rendering, stock information, and news search.

[13:06](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=786s) [Chat response lists image generation, charting, stock information, and news search.](https://assets.ai.engineer/sites/aiecode2025/sha256/ef0f23ecb422d3c4103253f93ffbff0771b00a4b56028dbe1345a4d59226117d.webp)

Compound AI System Demo with FireFunction V2

What can you do for me?

I can assist you with various tasks such as generating images from text descriptions, rendering charts from numeric data, obtaining the latest price and volume information for a given stock ticker symbol, and searching for recent news articles related to a query. Please let me know what you need help with.

Interface text:
show available functions
Type a message
33

### FireFunction V2 function schemas

The demo exposes schemas for image generation and chart rendering. FireFunction must translate a user’s request into calls to external tools.

[13:10](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=790s) [The demo displays function definitions for image generation and chart rendering.](https://assets.ai.engineer/sites/aiecode2025/sha256/b54a452967161cb5885544ec009c31fdc36a94931f58ae7793e6a045a3a50746.webp)

Compound AI System Demo with FireFunction V2

[
  {
    type: "function",
    function: {
      name: "generateImage",
      description: "Generates an image from a text description. The tool output should be displayed as ![text](image_url) where the image url includes blob: prefix.",
      parameters: {
        type: "object",
        properties: {
          prompt: {
            description: "description of the image",
            type: "string"
          },
          negative_prompt: {
            description: "concepts that should be excluded from the generated image",
            type: "string"
          }
        },
        required: [
          "prompt"
        ]
      }
    }
  },
  {
    type: "function",
    function: {
      name: "renderChart",
      description: "Generates chart from numeric data. The chart is rendered by Chart.js, a popular open-source charting library. The tool output should be displayed as ![text](image_url) where the [illegible]

Interface text:
hide available functions
Type a message
33

### Stock chart for three cloud providers

The demo queries stock prices for three major cloud providers and passes them to a chart plotting API. The resulting chart puts MSFT well above AMZN and GOOGL.

[13:56](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=836s) [Bar chart compares stock prices for Microsoft, Amazon, and Alphabet.](https://assets.ai.engineer/sites/aiecode2025/sha256/5ac2ed9390c381e14f9323c503f2e395e4ba0c284178ffc20d9356e4625a6cfe.webp)

Compound AI System Demo with FireFunction V2

Generate a bar chart with stock prices of the top 3 cloud providers.

Stock Prices
Y-axis ticks: 0, 50, 100, 150, 200, 250, 300, 350, 400, 450
X-axis labels: MSFT, AMZN, GOOGL

Interface text:
show available functions
show function calls
Type a message
33

### Stock chart with Oracle added

The demo adds Oracle to the existing stock chart, using the earlier conversation to understand which graph to update.

[14:10](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=850s) [Bar chart compares stock prices for Microsoft, Amazon, Alphabet, and Oracle.](https://assets.ai.engineer/sites/aiecode2025/sha256/fe1f17ee8380f97af2e418699150020e60bbf0e3daa447fcc2206285693e2641.webp)

Compound AI System Demo with FireFunction V2

Add Oracle to it.

Stock Prices
Y-axis ticks: 0, 50, 100, 150, 200, 250, 300, 350, 400, 450
X-axis labels: MSFT, AMZN, GOOGL, ORCL

Interface text:
show available functions
show function calls
Thanks! Just for fun,
33

## Full transcript

[0:00](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=0s) [upbeat music] Hello, everyone.

[0:15](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=15s) So my name is Dima. Um, as mentioned, uh, unfortunately my co- uh, co-founder Lin, who was, uh, on the schedule couldn't make it today because of some personal emergency.

[0:24](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=24s) So you, you got me and as you saw, we don't have yet AI to figure out video, uh, projection, but we have AI for a lot of other things.

[0:32](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=32s) Uh, so today I'm gonna talk about Fireworks AI and generally gonna con- continue this theme which Caitlin started about open models, uh, and how we, uh, basically focus on productionization and customization of, uh, open source models, uh, in inference at Fireworks.

[0:48](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=48s) But first, uh, as a introduction, uh, what's our background? Uh, so f- the founding team of Fireworks, uh, comes from PyTorch leads at Meta and some veterans from Google AI.

[0:59](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=59s) So we combined have like probably decade of experience in productionizing AI in some of the biggest companies in the world and I myself personally have been core maintainer of PyTorch, uh, for the past five years.

[1:11](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=71s) So topic of open source is really close to my heart and since we kind of led this revolution of open source tool chain for deep learning through our work through, on PyTorch and some of the Google technologies, we really believe that open source, uh, models are the future also for like, for, for GenAI application and our focus

[1:32](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=92s) at Fireworks is precisely on that. Um, so

[1:38](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=98s) I mean, uh, how many people in the audience actually like use GPT and deploy it in for production?

[1:45](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=105s) And how many people-- how many folks use open models, um, or deploy to production? Oh, okay. So I was about to convince you that share of open source models is gonna grow over time, but it looks like in this audience it's already [laughs] already sizable, but nevertheless, um, so why, why basically this trade-off?

[2:04](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=124s) Why go big or, uh, why go small? Um, currently still like bulk of production inference is still based on proprietary models and, uh, the catch is that those are really good models and they are often fr- frontier in ma- in, in many domains.

[2:18](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=138s) Uh, however, the catch is that it's one model which is good in many, many things and it's often served, uh, in the same way, uh, regardless of the use case which means that maybe if you have batch inference, uh, on some narrow domain or you have some super real-time use case, uh, where you need to, you need

[2:35](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=155s) to do like voice, uh, assistant or something, uh, those are often s-served from the same infrastructure without customization. In terms of model capabilities, it also means, yeah, like GPT-4 is great, uh, or, uh, Claude is great and can handle a lot of things, but you are often paying a lot for additional capabilities which are not needed in

[2:52](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=172s) particular use case. You don't really need customer support chatbot to know about 150 Pokémons or be able to write, write you poetry, uh, but you really want it to be really good in the particular, uh, narrow domain.

[3:06](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=186s) So this, uh, can, uh, this kind of discrepancy for large models leads to several issues. One, as I mentioned, is high latency because using a big model means, uh, longer response times, uh, which is particularly important for real-time use cases like voice assistants.

[3:22](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=202s) It gets more and more important with agentic stuff because, uh, for stuff like, um, for example, n-next [laughs] time is gonna be a Devin, right? Like you, you need to do a lot of steps for like something like agent-like application to do reasoning and call the model many times, so latency is really, really important.

[3:38](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=218s) And often you see that you can pick smaller models like Llama or Gemma [laughs] which we, which we just talk about and achieve the, for a narrow domain, uh, same or better quality while being, you know, up to 10 times faster.

[3:51](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=231s) Uh, for example, for some of the function calling use cases like externally benchmark from, uh, from Berkeley, yeah, like the, you get similar performance from fine-tuned Llama 3 at 10X the speed.

[4:03](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=243s) Cost is also, uh, is, is also an issue if you're running a big model for, uh, on a lot of traffic. You know, even if you have perhaps, I don't know, 5K tokens prompt and 10,000 users and each of them calls LLM 20 times per day, you know, on GPT-4, even on GPT-4o, it probably adds up to

[4:20](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=260s) like 10K per day or something like several million per month. Also se- several million per year which is a sizable cost of a startup. You can easily cut that with much smaller models and that often we see as a, uh, as a kind of motivation for reaching out for smaller and more customizable models.

[4:39](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=279s) Uh, but really the, uh, like where open models shine is, uh, domain adaptability and that comes, uh, in two aspects. First, um, there is so many different fine-tunes and customizations.

[4:50](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=290s) Uh, I think Caitlin was mentioning about, you know, Gemma built Indian languages adaptations like there are models specialized for code or for medicine. If you head to Hugging Face, there are like tens of thousands of diff- different model variants and because the weights are open you can always customize to your particular use case and tune and, uh,

[5:07](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=307s) tune quality specifically for, for what you need. So open source models are great, so what are the challenges? Uh, the challenges really come from three areas. Uh, first like what we usually see when people try to use, you know, open model, something like Gemma or whatever, uh, or, or Llama 2B, uh, you run into com- complicated setup

[5:27](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=327s) and maintenance, right? You need to go and find GPU somewhere. You need to figure out which frameworks to run on those. You need to like download your models, maybe do some performance optimization and tuning and you kind of have to repeat this process end to end every time the model gets updated or new version is released, et

[5:43](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=343s) cetera. Uh, on optimization itself, uh, there is especially for LLMs, but generally for GenAI models, there are many attributes, uh, and settings which are really de- really dependent on your use case and requirements.

[5:56](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=356s) Somebody needs low latency, somebody needs high throughput. Prompts can be short, prompts can be long, et cetera, and choosing the optimal settings across the stack is actually not trivial and as I show later, in many cases you can get multiple X improvements from doing, from doing this efficiently.

[6:12](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=372s) And finally, like just getting it production ready is actually hard. Uh, if y- as you sc- kind of go from experimentation to production, even just babysitting GPUs on public clouds is not, not as easy because GPUs are finicky and not always reliable.

[6:26](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=386s) Uh, but getting to enterprise scale requires, you know, sca- all the scalability technology, telemetry, observability, et cetera. So those are, uh, things which we focus on, uh, solving, uh, at Fireworks.

[6:38](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=398s) So starting with efficiency, we built, uh, our own custom service stack, which we believe is one of the fastest, if not the fastest. Uh, we did it, uh, did it from the ground up, meaning from writing our own, you know, CUDA kernels all the way to customizing how the s- uh, stuff gets deployed and orchestrated on the

[6:56](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=416s) service level, and that brings multiple optimizations. But most importantly, we really focus on customizing the service stack to your needs, uh, which basically means for your custom workload and for your custom cost and latency, uh, latency requirements, we can, we can tune it for, uh, for those settings.

[7:14](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=434s) What does it mean in practice? And what does customization means in practice? Uh, for example, many use cases use RAG and, uh, use very long prompts. Uh, so there are many settings you can tune actually on the runtime level and the deployment level to optimize for long prompts, which often can be repeatable, so caching is useful, or

[7:31](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=451s) just m- tune in settings so the throughput is higher while maintaining latency. So this is i- independently benchmarkable. If you go to, uh, you know, Artificial Analysis and select Long Prompt, where Fireworks actually is the fastest, even faster than some of the other providers which are over there at Expo booth.

[7:47](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=467s) Uh, and, uh, we don't only focus, we don't only focus on LLM inference. Uh, we are focused on many modalities. Uh, as an example, for image generation, we are the fastest provider serving SDXL.

[7:58](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=478s) We're also the only provider serving SD3, uh, uh, in, in, uh, Stability's new model because their API actually routes to our servers. Um, and finally, as I mentioned, with like L- LLMs, like customiza- especially for LLMs, customization matters a lot.

[8:14](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=494s) Uh, one requirement, like one problem we have to think about performance of LLMs often it's useful for use cases is to think about maxi- like minimizing cost under a particular latency constraint.

[8:23](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=503s) We often have customers come and say like, "Hey, I need to like have this, my interactive application. I need to generate that many tokens under two seconds." And that's where, that's really where like cross-stack optimizations shine, uh, whereby, uh, tuning the particular like latency cutoff and changing many settings, you can deliver much higher throughput, multiple times higher

[8:43](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=523s) throughput, uh, which bas- higher throughput basically means fewer GPUs and lower cost.

[8:50](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=530s) Uh, in terms, in terms of, uh, model support, we support, support best quality open source models. Uh, you know, we heard about Gemma now, obviously, uh, uh, Llama's, uh, some of the, uh, ASR and text-to-speech models, uh, pretty much from, from many providers.

[9:05](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=545s) We also work with model developers. For exam- uh, for example, Yi-Large, uh, in US is also served on, uh, on Fireworks, launched, uh, laun-launched, uh, last week. And, uh, as a kind of pla- platform capabilities, as I mentioned, we have a lot of op- open source models to st- uh, to get you started or cus- uh, customized

[9:23](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=563s) ones. We do some of the fine-tuning of those models, uh, in-house. So I'm gonna talk a little bit about function calling specialized models, uh, later on, or we do some of the, uh, vision-language models fusion, uh, ourselves, which we re-release as well.

[9:38](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=578s) And of course, the key for open source, um, mo- open model development is that it can tune for a particular use case. So we do provide a platform for fine-tuning, uh, whether you're bringing your dataset collected elsewhere or collecting it live, uh, with the feedback wh-when served on our platform.

[9:56](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=596s) Uh, specifically on customization is like one, uh, interest... One interesting feature which a lot of people expe- starting to experiment with models, uh, find interesting is if you try to fi-fine-tune and deploy the resulting model, how, uh, how to serve it efficiently.

[10:10](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=610s) Uh, it turns out if you do LoRA fine-tuning, which a lot of folks do, uh, you can do, uh, s- smart tricks and deploy multiple LoRA models on the same GPU, uh, actually thousands of them, which means that we can give you still serverless inference with paying for, per token, even if you have like thousands of m-

[10:26](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=626s) uh, model variants, uh, sitting and deployed there without having to pay any fixed cost.

[10:33](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=633s) Of, of course, single model is all great, uh, but what we see increasingly more and more in applications is model is not the product, right, uh, uh, by itself.

[10:43](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=643s) You need a kind of bigger system, uh, in order to solve target application. And the reason for that is because, uh, models by themselves tend to hallucinate, so you need some grounding, and that's where like RAG or a- access to external, uh, knowledge bases comes in.

[10:57](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=657s) Uh, also, we don't have, you know, yet an industry magical multimodal, uh, AI across all the modalities, so often you have to kind of chain multiple types of models.

[11:07](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=667s) And of course, you have all these like external tools and external actions which, uh, uh, kind of end-to-end applications might want to do in agentic form. Uh, so I think the term which I really like, which was basically popularized by Databricks, is like compound AI system.

[11:22](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=682s) But basically, re- increasingly seeing like transition from just the model being the product to kind of this combination of maybe like RAG and function calling and external tools, et cetera, built together as the product, and that's pretty much the direction which we kind of see, uh, th-this field mo- uh, moving along, uh, o- o- over time.

[11:40](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=700s) So what does it mean from, uh, from kind of our perspective, what we, what we do in this case? Uh, so, uh, we see kind of as a function calling like agent as a, at the core of this, uh, emerging architecture, which might be connected to either domain specialized, uh, models, uh, uh, se- served on our platform

[11:58](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=718s) di- directly or maybe tuned for partic- different needs, and connected to external tools, uh, maybe it's a code interpreter or maybe it's like external APIs somewhere, uh, with really like this kind of central agentic, uh, view, uh, kind of central, central model kind of coordinating and trying to triage the, uh, user, user requirements if it's, for example,

[12:17](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=737s) a chatbot or something. Uh, you probably all, uh, heard about like function calling, you know, popularized by OpenAI initially. That's, uh, that's basically the same idea. Uh, so yeah, the function calling is really like a how to- How to connect, uh, LLM to external tools and exter- and, and external elements.

[12:36](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=756s) What does it mean in practice? So we actually, uh, uh, fo- focus on fine-tuning models specifically for function calling. So we released a series of models like that, like the latest one, FireFunction, which was released two, uh, two weeks ago.

[12:48](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=768s) And, uh, what you can do with that is, uh... Oh. Okay, if I manage to click.

[12:57](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=777s) If I manage to click on this button. [laughs]

[12:59](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=779s) Uh, w- what it means is, like, you can build, uh, applications which kind of combine freeform general chat, uh, capabilities with function calling. So in this case, uh, this is, you know, this FireFunction has some, uh, chat capabilities.

[13:13](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=793s) So you can see, you can, like, ask it what, what you can you do, and it has, like, some self-reflection to tell you what it can do. It's also connected in this demo app to a bunch of, uh, external tools.

[13:22](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=802s) So it can, uh, query, uh, like, stock quotes. It can plot some charts, all those, like, external APIs. Uh, it can, um, also gener- gen- generate images. But what it really needs to figure out is how to translate user query into, do complex reasoning, translate it into function calls.

[13:40](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=820s) So for example, if you ask it to generate a bar chart with top three, uh, like with stocks of top cloud providers, like the Big Three, it actually needs to do several steps, right?

[13:51](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=831s) It needs to understand that, like, top three cloud providers means, you know, AWS, GCP, and, uh, and Azure, right? And Azure is o- owned by Microsoft. It needs to, uh, then go do function calls querying their stock prices.

[14:04](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=844s) And finally, it needs to combine those information and send it to chart plotting API, which is what just happened, uh, in the, in the background. Uh, another important aspect which you have to do for, like, ef- efficient, uh, kind of function calling chat capabilities, you need to have contextual awareness.

[14:20](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=860s) So if I ask it to add particular, uh, if I ask it to add Oracle to this graph, it needs to understand what I'm referring to and, like, still keeps the previous context and regenerates the image.

[14:30](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=870s) And finally, you know, if I switch to a partic- to a different topic, it kind of needs to drop the previous context and understands that, like, "Hey, this is less, uh, this historical context is less important.

[14:40](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=880s) I'm gonna start from scratch." So there is no, like, Oracle in that cat photo or whatever. Uh, so you know, this particular demo is, uh, is, is actually open source.

[14:48](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=888s) You can, like, go to our GitHub and try it out. It's built with FireFunction, and it builds with, like, a few other mo- a few other models, including, like, SDXL, which are run on our platform.

[14:58](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=898s) Uh, the model itself for function calling is actually open source. Uh, it's on Hugging Face. I mean, you can of course call it on, uh, at, at Fireworks with, for optimal speeds, but you can also, uh, run it locally if you want.

[15:09](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=909s) It uses a bunch of, uh, you know, functionality on our platform. Uh, for example, like structured generation, uh, with, like, uh, with JSON mode or grammar mode, which I think was similar to some of the previous talks from, like, Outlines guys, uh, which were talking here yesterday.

[15:25](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=925s) Uh, yeah, so finally, try, try it out. And generally, like, how to get started on Fireworks. So if you head, head out to fireworks.ai/models, you'll, uh, you'll find a lot of, uh, open, open source, open base models, which I mentioned about.

[15:38](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=938s) They're available in the playground. In terms of product offering, uh, we have this kind of range which can take you from early prototyping all the way to enterprise scale.

[15:47](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=947s) So you can start with serverless inference, which is, you know, not different from, uh, getting to OpenAI Pi- uh, OpenAI playground or something, where you pay per token. Uh, it's a constant price.

[15:57](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=957s) You don't need to worry about, like, hardware settings or anything. As I mentioned, you can still do fine-tuning, so you can, uh, you can do hosted fine-tune on our platform.

[16:05](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=965s) You can bring your own LoRA adapter and still serve it serverless. As you kind of graduate to, like, maybe, like, a startup, and you graduate to, uh, more production scale, uh, you might want to go to on-demand, where it, where it's more, like, dedicated hardware with more settings and modifications, uh, for your use case.

[16:20](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=980s) Uh, you can bring your own custom model fine-tuned from scratch or do it on our platform. And finally, as you kind of, if you scale up, uh, to bigger volume and want to go to enterpr- enterpr- enterprise level, where it's kind of discounted long-term, uh, long-term contracts, and we also will help you to kind of personalize hardware

[16:37](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=997s) setup and do some of those tuning, uh, for performance, which I, uh, which I talked about earlier. And in terms of use cases, I mean, we're running production for m- uh, for many, many companies, ranging from small startups to big enterprises.

[16:50](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=1010s) We're serving, like, last time I checked, like, more than 150 billion tokens per day. So, you know, companies like Quora built chatbots like Poe. Uh, Sourcegraph and Cursor, which I think, I think Cursor had a talk here yesterday.

[17:03](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=1023s) They use us for, like, some of their code assistant functionality, and their, like, latency is really important. Uh, as you can imagine, you know, folks like Upstage and Liner are building, like, different assistants and agents, uh, on top of that.

[17:15](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=1035s) So, uh, we are definitely production ready. Go try, try it out. Uh, finally, we care a lot about developers, you, uh, you guys. Um, so actually, this is external numbers from, uh, like, last year, LangChain State of AI stuff, where turns out we are one of the, like, a- after Hugging Face, the most popular platform for where

[17:34](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=1054s) people pull models, which is great. Which is very, was very nice to, uh, nice to hear. And again, for, like, for getting started, just, uh, you know, head out to, uh, head out to our website.

[17:44](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=1064s) You can go in the, go play in the playground, uh, right away. So for example, you can run, you know, Llama or Gemma or whatever at the, at the top speeds.

[17:53](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=1073s) Um, and kind of go start building from there. I'm really excited to see what you can build with open models of FireFunction or some stuff which you can, uh, which you can fine-tune on, uh, on your own.

[18:04](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=1084s) And yeah, last point, we are, as I mentioned, OpenAI API compatible, so you can still use your, you know, your, your favorite tools, uh, the same clients, or you can use frameworks like LangChain or LlamaIndex or et cetera.

[18:16](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=1096s) So yeah, really excited to, uh, to kind of, uh, to be here and talk, tell a little bit about open source, uh, open source models and how we at Fireworks are, uh, focusing on productionizing that and scaling it up.

[18:30](https://www.youtube.com/watch?v=C1CXwRYbwuQ&t=1110s) Uh, go try it out, and you can also find us at the booth, uh, at the expo. Thank you. [upbeat music]
