# Customized, production ready inference with open source models: Dmytro (Dima) Dzhulgakov

Dmytro (Dima) Dzhulgakov, Fireworks AI

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/ePMvfa8vgL8-customized-production-ready-inference-open-source) · [Video](https://www.youtube.com/watch?v=ePMvfa8vgL8)

## Customizing open-model inference for production

Production inference means matching model capabilities and serving infrastructure to a workload, then connecting models to the tools and context that make an application useful.

Familiarity with language-model inference, fine-tuning, and basic API calls will help; the article explains the roles of RAG, LoRA, and function calling as they arise.

### What does your application actually need from a model?

[0:32](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=32s)

What does it take to turn an open model into a production service tuned for a particular application? For Dmytro (Dima) Dzhulgakov, that question follows naturally from building the tools used to train models. He describes Fireworks AI’s founding team as former PyTorch leads at Meta and Google AI veterans, and his own background as five years maintaining PyTorch. The precedent is an open toolchain that developers can adapt; Fireworks applies that approach to model customization and production inference.

An audience poll finds substantial open-model use in the room, although it says nothing about the market as a whole. The choice still has a real trade-off: proprietary frontier models offer strong capabilities across many domains, making them attractive starting points for production applications.

![Slide linking GPT-4 to chat, code generation, and agents, with the statement “Best quality off-the-shelf, but high costs and poor UX in production.”](https://ai.engineer/api/talk-essay-media/757b4823c0e03317597cd5f3ca60ef1d0456f7e6d5df192b637ddbce2414122f)

Big models: broad capabilities, but high costs and poor production UX.

The mismatch appears when both the model and its serving infrastructure are broader than the task requires. Batch inference over a narrow domain and a real-time voice assistant can receive the same serving treatment despite needing different things. **GPT-4** and **Claude** can handle many tasks, but a customer-support chatbot does not need Pokémon trivia or poetry-writing ability. It needs reliable answers within its support domain. **Model capability and serving configuration are separate things to customize.**

### Repeated calls multiply latency and cost

[3:06](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=186s)

Large-model response times are especially visible in voice assistants, where a delay interrupts the interaction. Agentic applications make the problem more pronounced: reasoning and tool use can require repeated model calls before the user receives a result. A delay that seems tolerable for one call can become costly across a sequence of dependent steps.

Dzhulgakov says smaller **Llama** or **Gemma** models can match or improve quality in a narrow domain while running up to 10× faster. His concrete example is a fine-tuned **Llama 3** function-calling model with similar performance on a Berkeley benchmark and a claimed 10× speed advantage. The accompanying slide labels GPT-4 and Llama 3; the talk does not establish the exact benchmark snapshot, prompt and output lengths, concurrency, or timing definition. Treat this as a particular specialization claim, not a general speed ratio between the model families.

![Latency chart shows GPT-4 at 1.97 seconds and Llama 3 at 0.18 seconds, with a 10× annotation and Berkeley Function-Calling Leaderboard source. Text links slow processing to low user engagement and product adoption.](https://ai.engineer/api/talk-essay-media/377644919470b9762d63c27fff0566ea93b3cdeac6b1dd5620af9cc99fab982e)

The latency slide compares GPT-4 and Llama 3, highlighting a 10× difference.

Traffic creates a second multiplier. The illustrative workload uses 5,000-token prompts, 10,000 users, and 20 calls per user per day—one billion input tokens daily. Dzhulgakov estimates roughly $10,000 per day, or several million dollars per year, for a large-model service. That is a rough total rather than a bill derivable from the stated inputs: output length is unspecified. The economic motivation remains clear even without adopting the estimate as a price quote: unnecessary model capacity becomes expensive when purchased repeatedly at scale.

### Open weights bring flexibility—and operating responsibilities

[4:38](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=278s)

Domain adaptation is the stronger reason to consider open models. Language adaptations of Gemma, code models, and medical models illustrate how specialization can change what a model is useful for. Hugging Face offers a broad collection of variants, and access to weights lets developers continue tuning for their own task instead of accepting a fixed general-purpose model.

That flexibility introduces three distinct responsibilities:

- **Setup and maintenance:** Find GPU capacity, choose a framework, download the model, and optimize it. New model releases can require repeating that work.
- **Workload optimization:** Choose settings for the actual combination of latency, throughput, and prompt length. A configuration suitable for short interactive requests need not suit long-context processing.
- **Production operation:** Handle unreliable GPUs and add the scaling, telemetry, and observability needed beyond an experiment.

A model that produces a good answer locally has solved only part of the deployment problem.

### Tune the stack around a latency requirement

[6:38](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=398s)

Fireworks’ approach spans custom CUDA kernels, the inference runtime, deployment, and service orchestration. The purpose of controlling those layers is to tune them together for a customer’s workload, cost target, and latency requirement. The optimization target is therefore an application configuration, not just an isolated model benchmark.

Long-prompt retrieval-augmented generation, or **RAG**, makes the distinction concrete. Repeated prompt content creates opportunities for caching. Runtime and deployment settings can also be adjusted to increase throughput while maintaining latency. Dzhulgakov cites Fireworks as the fastest provider under Artificial Analysis’s Long Prompt selection at the time; that historical ranking is not a promise about every prompt distribution or today’s providers.

The same serving focus extends beyond language models. Dzhulgakov describes Fireworks as the fastest **SDXL** provider and the sole provider serving **Stable Diffusion 3**, with Stability’s API routing to Fireworks servers. These are his claims about the offering at the time. Stability’s [API announcement](https://stability.ai/news-updates/stable-diffusion-3-api) confirms the serving partnership, but does not establish exclusive hosting or a fastest-provider result.

**Minimize cost subject to the application’s latency constraint.** That is a more useful objective than maximizing speed without regard to expense. Dzhulgakov’s customer example asks for a specified amount of generated text in under two seconds; the token count is not supplied. Once that cutoff is known, settings across the stack can be tuned to increase the amount of work completed while still meeting it. He reports multiple-fold throughput improvements from such tuning, without a universal multiplier. For a fixed workload, higher throughput means less GPU capacity is required, which lowers serving cost.

### Serve specialized variants without duplicating everything

[8:50](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=530s)

The model catalog described in the talk includes Gemma, Llama, speech recognition, and text-to-speech models. Fireworks also works directly with model developers; Dzhulgakov gives **Yi-Large** serving in the United States as a recent example. Alongside those models, the company produces function-calling fine-tunes and vision-language model fusions. Customization can use a dataset brought to the platform or feedback collected from live serving.

Fine-tuning creates another infrastructure question: how should all the resulting variants be served? With **LoRA**, variants can share the base model rather than each requiring a separate full-model deployment. Dzhulgakov describes deploying thousands of LoRA variants on a shared GPU, enabling per-token serverless billing without a fixed deployment charge for each variant. The talk does not specify adapter sizes, simultaneous residency, or concurrency, so the count should not be read as a guarantee that thousands of variants can all run concurrently.

### A coordinating model connects knowledge, models, and tools

[10:33](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=633s)

**A model alone is rarely the whole product.** Grounding through RAG or external knowledge bases helps address hallucination. Different modalities can require different models chained together. Tools and external APIs let an application act beyond generating text. Dzhulgakov uses the term **compound AI system**, whose popularization he attributes to Databricks, for this combination of components.

In the architecture he presents, a central function-calling model interprets and triages the user’s request. It can connect to domain-specialized models, a code interpreter, or external APIs. The central model supplies coordination: deciding which capability is needed and translating a conversational requirement into a tool request. Function calling, popularized by OpenAI, is the interface between the language model and those external components. Fireworks fine-tunes models specifically for that role; **Firefunction** is the recently released example introduced here.

![Compound AI Inference Engine diagram places FireFunction, labeled JSON/Grammar mode, around two groups: Fireworks Inference with model modalities and tuned models, and Tools and APIs with database, package, and API icons.](https://ai.engineer/api/talk-essay-media/93771a36d6ede30521ed17783eef0340646e7fa0b6d4c4d7185c78a467a3f2b2)

FireFunction surrounds Fireworks Inference and external tools in the compound AI architecture.

### From cloud providers to a stock chart—and then a cat

[12:59](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=779s)

The demonstration combines ordinary conversation with function calling. A user can ask what the assistant can do and receive a conversational description of its capabilities. Behind that chat interface are tools for querying stock quotes, plotting charts, and generating images. The model’s job is to translate an informal request into the appropriate sequence of calls.

The first substantive request asks for a bar chart of the stocks associated with the Big Three cloud providers. Producing it requires several dependent steps:

1. **Resolve the entities.** Interpret the cloud providers as AWS, GCP, and Azure, then associate those services with the companies whose stocks can be queried. Dzhulgakov explicitly calls out Azure’s ownership by Microsoft.
2. **Retrieve the data.** Call the stock-query tools for the relevant companies.
3. **Construct the chart.** Combine the returned information and pass it to the chart-plotting API.

The request does not supply company identifiers, prices, or chart data. Those must be resolved or retrieved before plotting. Dzhulgakov reports that this sequence has completed in the background.

The follow-up—adding Oracle to the graph—tests a different capability. The assistant must resolve which graph the user means, retain its existing context, and regenerate it with the additional company. A new request for a cat image tests the opposite behavior: the earlier stock discussion should no longer determine the tool request. The conversation can remain in history without Oracle leaking into the cat image. **Useful context handling means preserving relevant state and excluding irrelevant state.**

The [Functional Chat demo](https://github.com/fw-ai/forge/tree/main/apps/functional_chat) is open source and combines Firefunction with other models, including SDXL. Dzhulgakov says the function-calling model is also available on Hugging Face, so it can run locally as well as through Fireworks. The platform supplies structured generation through JSON mode and grammar mode. These constrain the form of a generated request; the demonstrated task still requires the model to choose the right entities, tools, and conversational context.

### Move from the playground to dedicated infrastructure

[15:25](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=925s)

The starting point is the [Fireworks model catalog](https://fireworks.ai/models) and its playground. The deployment progression presented in the talk adds control as workload requirements become clearer:

| Offering in the talk | Billing or commitment | Deployment control |
| --- | --- | --- |
| Serverless | Fixed per-token pricing | No hardware configuration |
| On-demand | Dedicated hardware | More workload-specific settings |
| Enterprise | Discounted long-term contracts | Personalized hardware and tuning |

In the historical serverless offering, users could fine-tune on the platform or bring a LoRA adapter and retain per-token serving. On-demand added dedicated hardware and support for custom models brought to, or fine-tuned on, the platform. Enterprise agreements added assistance with hardware selection and performance tuning. **That LoRA deployment policy has changed:** [documentation checked in August 2026](https://docs.fireworks.ai/fine-tuning/deploying-loras) requires dedicated on-demand deployments for both imported and platform-trained LoRAs, rather than serverless deployment.

### Production workloads and familiar integration tools

[16:50](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=1010s)

Dzhulgakov reports that Fireworks was serving more than 150 billion tokens per day when he last checked. He names Quora’s Poe as a chatbot example, Sourcegraph and Cursor for code-assistant functionality, and Upstage and Liner for assistants and agents. Coding assistance makes the latency requirement particularly tangible: a response must arrive quickly enough to remain useful within the developer’s ongoing work. These are the speaker’s production examples and volume report, not audited measurements.

The closing adoption slide cites [LangChain State of AI 2023](https://www.langchain.com/blog/langchain-state-of-ai-2023), placing Fireworks after Hugging Face in the displayed open-source list. The underlying report uses anonymized LangSmith user metadata, not a market-wide survey: Fireworks led its API-access open-source-provider category and ranked sixth among LLM providers overall. That scope matters when interpreting the slide as evidence of developer adoption.

![Slide states that 33,000 developers have run on Fireworks since August 2023 and cites Langchain State of AI Report 2023. A ranked list places Hugging Face first, fireworks.ai second, and ollama third.](https://ai.engineer/api/talk-essay-media/892739b960147f0443f846ff41abb611b9338d3fc04c410014f39f09a5d29643)

The adoption slide reports 33,000 developers and lists Fireworks second under “Most Common OSS Models.”

The final integration point is OpenAI API compatibility. Dzhulgakov presents it as a way to reuse familiar clients and tools, including LangChain and LlamaIndex, while changing the model and its serving configuration underneath. The application can begin with a familiar interface and add specialized models, function calling, and workload-specific infrastructure as its requirements become concrete.

## Resources

- [Functional Chat demo source](https://github.com/fw-ai/forge/tree/main/apps/functional_chat) — Chat application combining stock quotes, chart generation, and SDXL images through function calls, with setup instructions.
- [Firefunction-v2 training and evaluation](https://fireworks.ai/blog/firefunction-v2-launch-post) — Historical release report describing the Llama 3 fine-tune, conversational capabilities, function-calling evaluations, and serving measurements.
- [FireAttention V2 performance methodology](https://fireworks.ai/blog/fireattention-v2-long-context-inference) — Long-context inference experiments showing how precision, hardware, generation length, and latency limits affect throughput.
- [LangChain State of AI 2023](https://www.langchain.com/blog/langchain-state-of-ai-2023) — Historical LangSmith usage report covering model providers, retrieval, agents, and evaluation practices.
- [Current LoRA deployment guide](https://docs.fireworks.ai/fine-tuning/deploying-loras) — Dedicated deployment instructions and performance trade-offs for merged adapters and shared multi-LoRA serving.
- [Current structured-output guide](https://docs.fireworks.ai/structured-responses/structured-response-formatting) — Examples and limitations for enforcing JSON schemas and custom grammars in model responses.

## Slides

### Presenting Sponsor

[0:05](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=5s) [Presenting sponsor Microsoft logo on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/d642d9146bb4ff07413c128b19d250bb9baf2e4006e3e5db00ccb6f1c6af8947.webp)

PRESENTING SPONSOR

Microsoft

### Innovation and Platinum Sponsors

AWS is the innovation sponsor; MongoDB, Google Cloud, and Neo4j are platinum sponsors.

[0:10](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=10s) [AWS is the innovation sponsor; MongoDB, Google Cloud and Neo4j are platinum sponsors.](https://assets.ai.engineer/sites/aiecode2025/sha256/7043e357ba3806d12f74cdc4be64c4a8862f4a00210cd3a23a506b6325da8755.webp)

INNOVATION SPONSOR
aws

PLATINUM SPONSORS
MongoDB®
Google Cloud
neo4j

### Fireworks AI: Customized, Production-Ready Inference with…

Dmytro Dzhulgakov introduces Fireworks AI’s focus on customizing and productionizing open-source models for inference.

[0:15](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=15s) [Title slide for Dmytro Dzhulgakov’s talk on customized inference with open-source models.](https://assets.ai.engineer/sites/aiecode2025/sha256/ba1e32648a72819740ab8e55a3baf191f93c4d89dc18f01a99884515fd1acd87.webp)

Fireworks AI

Customized, production-ready inference with open-source models

Dmytro (Dima) Dzhulgakov, CTO & co-founder

Fireworks AI

### Fireworks: Productionizing Open-Source LLMs

Dzhulgakov introduces the Fireworks founding team: PyTorch leads from Meta and Google AI veterans. He describes himself as a PyTorch core maintainer for the previous five years.

[0:49](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=49s) [Four team portraits with the labels Fireworks Team and PyTorch & Google AI.](https://assets.ai.engineer/sites/aiecode2025/sha256/a31078586901be59a5f9f5b1f7c5755ffc10e7381a7cf7d31906bc78886f2405.webp)

Fireworks - productionizing open-source LLMs

Fireworks Team

PyTorch & Google AI

4

### Going Big or Small?

Dzhulgakov opens the model-size tradeoff by asking who deploys GPT or open models in production. He notes that open-model use already looks sizable in this audience.

[1:39](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=99s) [Purple section slide asking, “Going Big or Small?”](https://assets.ai.engineer/sites/aiecode2025/sha256/3d8c572a178bf19d195244472b9309eb696ee8d655ffd08606a2ff6815a5bee8.webp)

Going Big or Small?

7

### Latency

The slide compares GPT-4 at 1.97 seconds with Llama 3 at 0.18 seconds. Dzhulgakov argues that latency matters especially for real-time voice assistants and agents that call models repeatedly.

[3:06](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=186s) [A chart compares GPT-4 latency of 1.97 seconds with Llama 3 latency of 0.18 seconds and labels the difference 10x.](https://assets.ai.engineer/sites/aiecode2025/sha256/f13bac7afa10e82e637b1ca6ab2b3826c33f595f63e819a4018607e591396762.webp)

Latency

Slow processing times hinder real-time performance

Impact: Low user engagement & product adoption

GPT-4 vs Llama 3 Latency (sec)
Y-axis ticks: 0.0, 0.5, 1.0, 1.5, 2.0
GPT-4: 1.97
Llama 3: 0.18
10x

Source: Berkeley Function-Calling Leaderboard

10

### Domain Adaptability

Open-weight models can be customized for specialized uses such as code or medicine. The slide points to 50k+ open-source GenAI models alongside the risk of generic, irrelevant responses.

[4:43](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=283s) [The slide discusses specialized knowledge and says there are over 50,000 open-source generative AI models.](https://assets.ai.engineer/sites/aiecode2025/sha256/cd07cdf4bf2ebe6299cd072368b4d4f89c83a1fb659c3de25acb202c8474442a.webp)

Domain Adaptability

General-purpose models typically lack knowledge in specialized areas

Impact: Generic & irrelevant responses, no moat

50k+
Open-source GenAI models

12

### Using open source models in production is still challenging

Running open models involves finding GPUs and repeating setup as models change, tuning for latency or throughput, and making infrastructure reliable and observable in production.

[5:09](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=309s) [Three illustrated challenges are complex setup, lack of optimization, and lack of production readiness.](https://assets.ai.engineer/sites/aiecode2025/sha256/2cafaf91c701737ca66e57ac039da86d6489afe5aa070913123fcafc71536618.webp)

Using open source models in production is still challenging

Complex Set-up
Not Optimized
Not Production-Ready

16

### Fireworks AI: Fast and Efficient AI Inference Engine

Fireworks introduces its inference engine after identifying scalability and observability as challenges for production deployments.

[6:32](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=392s) [Purple title slide introducing Fireworks AI's inference engine.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a894c15de4b3241dcad83671fc022e0de765fe7e2498dcd2e6d626be4553a5e.webp)

Fireworks AI
Fast and Efficient AI Inference Engine

17

### The fastest serving stack, customized to you

FireAttention combines quantization, speculative decoding, semantic caching and distributed inference. Fireworks tunes the hardware and software stack to each workload’s cost and latency needs.

[6:37](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=397s) [FireAttention serving stack features include quantization, speculative decoding, semantic caching and distributed inference, with hardware and software tuned to the workload.](https://assets.ai.engineer/sites/aiecode2025/sha256/7e64a3adc59a35b7488ceeb053a290dcfe3a2ffaaf68b7c9043e2cd76b68d113.webp)

The fastest serving stack, customized to you

• FireAttention serving stack
  ◦ Quantization without quality impact
  ◦ Speculative decoding
  ◦ Semantic caching
  ◦ Distributed inference
• Customized serving for you
  ◦ Hardware and software stack tuned for your workload and goals

18

### LLM with RAG

In an Artificial Analysis benchmark using Mixtral 8x7b Instruct, 10 parallel queries and a 10k-token prompt, Fireworks takes 1.4 seconds to output 100 tokens; Groq takes 12.2 seconds, about 9x as long.

[7:12](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=432s) [Bar chart attributed to Artificial Analysis compares response times for Mixtral 8x7b Instruct with a 10,000-token prompt; Fireworks is shown at 1.4 seconds and Groq at 12.2 seconds.](https://assets.ai.engineer/sites/aiecode2025/sha256/a710fc575fa9929f78bd36e8fe60c017feae55d7cf9184d65ba1e36a10055183.webp)

LLM with RAG

Total Response Time
Seconds to Output 100 Tokens; Lower is better

Independent benchmark from ArtificialAnalysis.ai
Mixtral 8x7b Instruct
10 parallel queries
10k token prompt

Fireworks: 1.4
Together.ai: 2.6
Mistral: 2.7
Amazon: 5.5
Replicate: 7.6
Groq: 12.2

9x
ArtificialAnalysis.ai

### Image Generation Speed

Artificial Analysis’s SDXL 1.0 benchmark shows Fireworks generating one image in 1.1 seconds, versus 9.8 seconds for Replicate—roughly a 9x difference.

[7:52](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=472s) [Bar chart attributed to Artificial Analysis compares SDXL 1.0 image generation times, showing Fireworks at 1.1 seconds and Replicate at 9.8 seconds.](https://assets.ai.engineer/sites/aiecode2025/sha256/e3a54591133b20bb9ffe74733036ecdcfda16da7bc5405328ab9350cc0d991d0.webp)

Image Generation Speed

Generation Time
Generation time: Seconds to generate 1 image, Lower is better

Independent benchmark from ArtificialAnalysis.ai
SDXL 1.0 Image Generation

SDXL 1.0, Fireworks: 1.1
SDXL 1.0 (SDXL), fal.ai: 1.9
SDXL 1.0, Together.ai: 4.5
SDXL 1.0, Amazon Bedrock: 6.3
SDXL 1.0, Stability.ai: 6.6
SDXL 1.0, Replicate: 9.8

9x
ArtificialAnalysis.ai

### More speed and output for your dollar

Fireworks FP8 leads the chart at all three latency cutoffs. Dzhulgakov says tuning for a latency constraint can increase throughput, requiring fewer GPUs and lowering cost.

[8:07](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=487s) [Bar chart compares vLLM and Fireworks throughput at one-, two-, and three-second cutoffs; the slide claims 8–15 times higher throughput.](https://assets.ai.engineer/sites/aiecode2025/sha256/d8b0b7f7a42e28b470008bb381d95fd7457bea04387aa863ba8bd748254a47d3.webp)

More speed and output for your $

8-15X higher throughput

Legend: vLLM Quant; vLLM FP16; Fireworks FP16; Fireworks FP8
Y-axis: requests per second
Y-axis ticks: 0, 20, 40, 60, 80
X-axis: 1 sec cutoff; 2 sec cutoff; 3 sec cutoff

21

### Best Quality Small Models

Fireworks presents 100+ top-quality models, with examples from Meta, OpenAI, Stability AI, Google, Microsoft, and Mistral.

[8:48](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=528s) [Examples of models from Meta, OpenAI, Stability AI, Google, Microsoft, and Mistral, with a claim of over 100 models.](https://assets.ai.engineer/sites/aiecode2025/sha256/a53a431ee59c69777ef76c8ba8254918022f50b80c896a87d63de82dd7c62ada.webp)

Best Quality Small Models

Meta — llama3
OpenAI — whisper
stability.ai — SD3
Google — gemma
Microsoft — phi-3-vision
MISTRAL AI_ — mixtral

100+ top quality models

### Best Application Quality Customized

Fireworks offers 100+ open-source and custom models, proprietary models, and fine-tuning with customer data collected elsewhere or through live feedback.

[9:19](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=559s) [Diagram presents open-source and custom models, Fireworks proprietary models, and models fine-tuned with customer data.](https://assets.ai.engineer/sites/aiecode2025/sha256/d6fdfa70742458951bcded9edc149f9e154e026e4dbd19ad12dea28a9ad4bbde.webp)

Best Application Quality Customized
Use the best model for each use case

100+ OSS & Custom Models
Fireworks Proprietary Models
Your data
Fine-tuned models

Sticky, differentiated value

## Full transcript

[0:00](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=0s) [on-hold electronic music] Hello, everyone.

[0:15](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=15s) So my name is Dima. Um, as mentioned, uh, unfortunately my co- uh, co-founder Lin, who was, uh, on the schedule couldn't make it today because of some personal emergency.

[0:24](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=24s) So you, you get me. And as you saw, we don't have yet AI to figure out video, uh, projection, but we have AI for a lot of other things.

[0:32](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=32s) Uh, so today I'm gonna talk about Fireworks AI and generally gonna con- continue this theme which Caitlin started about open models, uh, and how we, uh, basically focus on productionization and customization of, uh, open source models, uh, in inference at Fireworks.

[0:48](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=48s) But first, uh, as a introduction, uh, what's our background? Uh, so f- the founding team of Fireworks, uh, comes from PyTorch leads at Meta and some veterans from Google AI.

[0:59](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=59s) So we combined have, like, probably decade of experience in productionizing AI in some of the biggest companies in the world. And I myself personally has been core maintainer of PyTorch, uh, for the past five years, so topic of open source is really close to my heart.

[1:15](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=75s) And since we kind of led this revolution of open source toolchain for deep learning through our work through, on PyTorch and some of the Google technologies, we really believe that open source, uh, models are the future also for, for,

[1:30](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=90s) for GenAI application, and our focus at Fireworks is precisely on that. Um, so

[1:38](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=98s) I mean, uh, how many people in the audience actually, like, use GPT and deploy it in pr- production?

[1:45](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=105s) And how many people-- how many folks use open models, um, or deploy to production? Oh, okay. So I was about to convince you that share of open source models is gonna grow over time, but it looks like in this audience it's already [laughs] already sizable.

[1:59](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=119s) But nevertheless, um, so why, why basically this trade-off? Why go big o- uh, why go small? Um, currently still, like, bulk of production inference is still based on proprietary models and, uh, the catch is that it-- those are really good models and they are often fr- frontier in ma- in, in many domains.

[2:18](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=138s) Uh, however, the catch is that it's one model which is good in many, many things, and it's often served, uh, in the same way, uh, regardless of the use case.

[2:26](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=146s) Which means that maybe if you have batch inference, uh, on some narrow domain or you have some super real-time use case, uh, where you need to, you need to do, like, voice, uh, assistant or something, uh, those are often s- served from the same infrastructure without customization.

[2:41](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=161s) In terms of model capabilities, it also means, yeah, like GPT-4 is great, uh, or, uh, Claude is great and can handle a lot of things, but you are often paying a lot for additional capabilities which are not needed in particular use case.

[2:53](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=173s) You don't really need customer support chatbot to know about 150 Pokémons or be able to write, write you poetry, uh, but you really want it to be really good in the particular, uh, narrow domain.

[3:06](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=186s) So this, uh, can, uh, this kind of discrepancy for large models leads to several issues. One, as I mentioned, is high latency because using a big model means, uh, longer response times, uh, which is particularly important for real-time use cases like voice assistants.

[3:22](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=202s) It gets more and more important with agentic stuff because, uh, for stuff like, um, for example, N- Next is gonna be a Devin, right? Like you, you need to do a lot of steps for, like, something like agent-like application to do reasoning and call the model many times.

[3:36](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=216s) So latency is really, really important. And often you see that you can pick smaller models like Llama or Gemma [laughs] which we, which we just talk about and achieve the, for a narrow domain, uh, same or better quality while being, you know, up to 10 times faster.

[3:51](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=231s) Uh, for example, for some of the function calling use cases, like externally benchmark from, uh, from Berkeley, yeah, like the... You get similar performance from fine-tuned Llama 3 at 10X speed.

[4:03](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=243s) Cost is also, uh, is, is also an issue. If you're running a big model for, uh, on a lot of traffic, you know, even if you have perhaps, I dunno, 5K tokens prompt and 10,000 users and each of them calls LLM 20 times per day, you know, on GPT-4, even on GPT-4o, it probably adds up to, like,

[4:20](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=260s) 10K per day or something like several million per month. Also several, several million per year, which is a sizable cost of a startup. You can easily cut that with much smaller models and that often we see as a, uh, as a kind of motivation for reaching out for smaller and more customizable models.

[4:38](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=278s) Uh, but really the, uh, like, where open models shine is, uh, domain adaptability and that comes, uh, in two aspects. First, um, there is so many different fine-tunes and customizations.

[4:49](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=289s) Uh, I think Caitlin was mentioning about, you know, Gemma built Indian languages adaptations. Like, there are models specialized for code or for medicine. If you head to Hugging Face, there are, like, tens of thousands of diff- different model variants and because the weights are open you can always customize to your particular use case and tune and, uh,

[5:07](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=307s) tune quality specifically for, for what you need. So open source models are great, so what are the challenges? Uh, the challenges really come from three areas. Uh, first, like, what we usually see when people try to use, you know, open model, something like Gemma or whatever, uh, or, or Llama today, uh, you run into com- complicated setup

[5:27](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=327s) and maintenance, right? You need to go and find GPU somewhere. You need to figure out which frameworks to run on those. You need to, like, download your models, maybe do some performance optimization and tuning and you kind of have to repeat this process end to end every time the model gets updated or new version is released, et

[5:43](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=343s) cetera. Uh, on optimization itself, uh, there is, especially for LLMs but generally for GenAI models, there are many attributes, uh, and settings which are really de- really dependent on your use case and requirements.

[5:56](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=356s) Somebody needs low latency, somebody needs high throughput. Prompts can be short, prompts can be long, et cetera. And choosing the optimal settings across the stack is actually not trivial and as I show later, in many cases you can get multiple X improvements from doing, from doing this efficiently.

[6:12](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=372s) And finally, like, just getting it production ready is actually hard. Uh, if y- as you sc- kind of go from experimentation to production, even just babysitting GPUs on public clouds is not, not as easy because GPUs are finicky and not always reliable.

[6:26](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=386s) Uh, but getting to enterprise scale requires, you know, sca- all the scalability technology, telemetry, observability, et cetera. So those are, uh, things which we focus on, uh, solving, uh, at Fireworks.

[6:38](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=398s) So starting with efficiency, we built, uh, our own custom service stack, which we believe is one of the fastest, if not the fastest. Uh, we did it, uh, did it from the ground up, meaning from writing our own, you know, CUDA kernels all the way to customizing how the s- uh, stuff gets deployed and orchestrated on the

[6:56](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=416s) service level, and that brings multiple optimizations. But most importantly, we really focus on customizing the service stack to your needs, uh, which basically means for your custom workload and for your custom cost and latency, uh, latency requirements, we can, we can tune it for, uh, for those settings.

[7:14](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=434s) What does it mean in practice? And what does customization means in practice? Uh, for example, many use cases use RAG and, uh, use very long prompts. Uh, so there are many settings you can tune actually on the runtime level and the deployment level to optimize for long prompts, which often can be repeatable, so caching is useful or

[7:31](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=451s) just m- tuning settings so the throughput is higher while maintaining latency. So this is i- independently benchmarkable. If you go to, uh, you know, Artificial Analysis and select Long Prompt, where Fireworks actually is the fastest, even faster than some of the other providers which are over there at Expo booth.

[7:47](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=467s) Uh, and, uh, we don't only focus-- we don't only focus on LLM inference. Uh, we are focused on many modalities. Uh, as an example, for image generation, we are the fastest provider serving SDXL.

[7:58](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=478s) We're also the only provider serving SD3, uh, uh, in, in, uh, Stability's new model because their API actually routes to our servers. Um, and finally, as I mentioned, with the, like, LL- LLMs, like customiza- especially for LLMs, customization matters a lot.

[8:14](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=494s) Uh, one requirement, like one problem we have to think about performance of LLMs, often it's useful for use cases, is to think about maxi- like minimizing cost under a particular latency constraint.

[8:23](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=503s) We often have customers come and say, like, "Hey, I need to, like, have this, my interactive application, I need to generate that many tokens under two seconds." And that's where-- that's really where, like, cross-stack optimizations shine, uh, whereby, uh, tuning the particular, like, latency cutoff and changing many settings, you can deliver much higher throughput, multiple times higher

[8:43](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=523s) throughput, uh, which bas- higher throughput basically means fewer GPUs and lower cost.

[8:50](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=530s) Uh, in terms, in terms of, uh, model support, we support, support best quality open source models. Uh, you know, we heard about Gemma now, obviously, uh, uh, Llama's, uh, some of the ASR and text-to-speech models, uh, pretty much from, from many providers.

[9:05](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=545s) We also work with model developers. For exampl- uh, for example, Yi-Large, uh, in US is also served on, uh, on Fireworks, launched, uh, laun-launched, uh, last week. And, uh, as a kind of pla- platform capabilities, as I mentioned, we have a lot of op- open source models to st- uh, to get you started or cus- uh,

[9:23](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=563s) customized ones. We do some of the fine-tuning of those models, uh, in-house. So I'm gonna talk a little bit about function calling specialized models, uh, later on, or we do some of the, uh, vision-language models fusion, uh, ourselves, which we re-release as well.

[9:38](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=578s) And of course, the key for open source, um, mo- open model development is that it can tune for a particular use case. So we do provide a platform for fine-tuning, uh, whether you're bringing your data set collected elsewhere or collecting it live, uh, with the feedback wh-when served on our platform.

[9:56](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=596s) Uh, specifically on customization is like one, uh, interest... One interesting feature which a lot of people expe- starting to experiment with models, uh, find interesting is if you try to fi-fine-tune and deploy the resulting model, how, uh, how to serve it efficiently.

[10:10](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=610s) Uh, it turns out if you do LoRA fine-tuning, which a lot of folks do, uh, you can do, uh, s-smart tricks and deploy multiple LoRA models on the same GPU, uh, actually thousands of them, which means that we can give you still serverless inference with paying for, per token, even if you have like thousands of m- uh,

[10:26](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=626s) model variants, uh, sitting and deployed there without having to pay any fixed cost.

[10:33](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=633s) Of, of course, single model is all great, uh, but what we see increasingly more and more in applications is model is not the product, right, uh, uh, by itself.

[10:43](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=643s) You need a kind of bigger system, uh, in order to solve target application. And the reason for that is because, uh, models by themselves tend to hallucinate, so you need some grounding, and that's where like RAG or a-access to external, uh, knowledge bases comes in.

[10:57](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=657s) Uh, also, we don't have, you know, yet an industry magical multimodal, uh, AI across all the modalities, so often you have to kind of chain multiple types of models.

[11:07](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=667s) And of course, you have all these like external tools and external actions which, uh, uh, kind of end-to-end applications might want to do in agentic form. Uh, so I think the term which I really like, which is, which is, like, popularized by Databricks, is like compound AI system.

[11:22](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=682s) But basically, re- increasingly seeing, like, transition from just the model being the product to kind of this combination of maybe like RAG and function calling and external tools, et cetera, built together as the product.

[11:32](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=692s) And that's pretty much the direction which we kind of see, uh, th-this field mo-, uh, moving along, uh, o-o-over time. So what does it mean from, uh, from kind of our perspective, what we, what we do in this case?

[11:44](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=704s) Uh, so, uh, we see kind of as a function calling like agent as a-- at the core of this, uh, emerging architecture, which might be connected to either domain specialized, uh, models, uh, uh, se-served on our platform di-directly or maybe tuned for partic- different needs, and connected to external tools, uh, maybe it's a code interpreter or maybe

[12:03](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=723s) it's like external APIs somewhere, uh, with really like this kind of central agentic, uh, view, uh, kind of central, central model kind of coordinating and trying to triage the, uh, user, user requirements if it's, for example, a chatbot or something.

[12:19](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=739s) Uh, you probably all, uh, heard about, like, function calling, you know, popularized by OpenAI initially. That's, uh, that's basically the same idea. Uh, so yeah, the function calling is really like a how to- How to connect, uh, LLM to external tools and exter- e- and external elements.

[12:36](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=756s) What does it mean in practice? So we actually, uh, uh, fo- focus on fine-tuning models specifically for function calling. So we released a series of models like that, like the latest one, Firefunction, which was released two, uh, two weeks ago.

[12:48](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=768s) And, uh, what you can do with that is, uh... Oh. Okay, if I manage to click.

[12:57](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=777s) If I manage to click on this button. [laughs]

[12:59](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=779s) Uh, w- what it means is, like, you can build, uh, applications which kind of combine freeform general chat, uh, capabilities with function calling. So in this case, uh, this is, you know, this Firefunction has some, uh, chat capabilities.

[13:13](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=793s) So you can see, you can, like, ask it what, what you can you do, and it has, like, some self-reflection to tell you what it can do. It's also connected in this demo app to a bunch of, uh, external tools, so it can, uh, query, uh, like, stock quotes.

[13:26](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=806s) It can plot some charts, all those, like, external APIs. Uh, it can, um, also gener- gen- generate images, but what it really needs to figure out is how to translate user query into, do complex reasoning, translate it into function calls.

[13:40](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=820s) So for example, if you ask it to generate a bar chart with top three, uh, like with stocks of top cloud providers, like the Big Three, it actually needs to do several steps, right?

[13:51](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=831s) It needs to understand that, like, top three cloud providers means, you know, AWS, GCP, and, uh, a- and Azure, uh, right? And Azure is o- owned by Microsoft. It needs to, uh, then go do function calls querying their stock prices, and finally, it needs to combine those information and send it to chart plotting API, which is what

[14:10](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=850s) just happened, uh, in the, in the background. Uh, another important aspect which you have to do for, like, e- efficient, uh, kind of function calling chat capabilities, you need to have contextual awareness.

[14:20](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=860s) So if I ask it to add particular, uh, if I ask it to add Oracle to this graph, it needs to understand what I'm referring to and, like, still keeps the previous context and regenerates the image.

[14:30](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=870s) And finally, you know, if I switch to a partic- into a different topic, it kind of needs to drop the previous context and understands that, like, "Hey, this is less, uh, this, uh, historical context is less important.

[14:40](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=880s) I'm gonna start from scratch." So there is no, like, Oracle in that cat photo or whatever. Uh, so you know, this particular demo is, uh, i- is actually open source.

[14:48](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=888s) You can, like, go to our GitHub and try it out. It's built with Firefunction, and it builds with, like, a few other mo- a few other models, including, like, SDXL, which are run on our platform.

[14:58](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=898s) Uh, the model itself for function calling is actually open source. Uh, it's on Hugging Face. I mean, you can of course call it on, uh, at, at Fireworks with, for optimal speeds, but you can also, uh, run it locally if you want.

[15:09](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=909s) It uses a bunch of, uh, you know, functionality on our platform, uh, for example, like structured generation, uh, with, like, uh, with JSON mode or grammar mode, which I think was similar to some of the previous talks from, like, Outlines guys, uh, which were talking here yesterday.

[15:25](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=925s) Uh, yeah, so finally, try, try it out. And generally, like, how to get started on Fireworks. So if you head, head out to fireworks.ai/models, you'll, uh, you'll find a lot of, uh, open, open source, open base models, which I mentioned about.

[15:38](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=938s) They're available in the playground. In terms of product offering, uh, we have this kind of range which can take you from early prototyping all the way to enterprise scale.

[15:47](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=947s) So you can start with serverless inference, which is, you know, not different from, uh, you know, getting to OpenAI p- uh, OpenAI playground or something, where you pay per token.

[15:56](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=956s) Uh, it's a constant price. You don't need to worry about, like, hardware settings or anything. As I mentioned, you can still do fine-tune, so you can, uh, you can do hosted fine-tune on our platform.

[16:05](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=965s) You can bring your own LoRA adapter and still serve it serverless. As you kind of graduate to, like, maybe, like, a startup, and you graduate to, uh, more production scale, uh, you might want to go to on-demand where it, where it's more, like, dedicated hardware with more settings and modifications, uh, for your use case.

[16:20](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=980s) Uh, you can bring your own custom model fine-tuned from scratch or do it on our platform. And finally, as a kind of, if you scale up, uh, to bigger volume and want to go to enterp- enterpr- enterprise level, where it's kind of discounted long-term, uh, long-term contracts, and we also will help you to kind of personalize hardware

[16:37](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=997s) setup and do some of those tuning, uh, for performance, which I, uh, which I talked about earlier. And in terms of use cases, I mean, we're running production for m- uh, for many, many companies, ranging from small startups to big enterprises.

[16:50](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=1010s) We're serving, like, last time I checked, like, more than 150 billion tokens per day. So you know, companies like Quora built chatbots like Poe. Uh, Sourcegraph and Cursor, which I think, I think Cursor had a talk here yesterday.

[17:03](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=1023s) They use us for, like, some of their code assistant functionality, and their, like, latency is really important. Uh, as you can imagine, you know, folks like Upstage and Liner are building, like, different assistants and agents, uh, on top of that.

[17:15](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=1035s) So, uh, we are definitely production ready. Go try, try it out. Uh, finally, we care a lot about developers, you, uh, you guys. Um, so actually, this is external numbers from, uh, like, last year, LangChain State of AI stuff, where turns out we are one of the, like, a- after Hugging Face, the most popular platform for where

[17:34](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=1054s) people pull models, which is great. Which is very, was very nice to, uh, nice to hear. And again, for, like, for getting started, just, uh, you know, head out to, uh, head out to our website.

[17:44](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=1064s) You can go in the, go play in the playground, uh, right away. So for example, you can run, you know, Llama or Gemma or whatever at the, at the top speeds.

[17:53](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=1073s) Um, and kind of go start building from there. I'm really excited to see what you can build with open models or Firefunction or some stuff which you can, uh, which you can fine-tune on, uh, on your own.

[18:04](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=1084s) And yeah, last point, we are, as I mentioned, OpenAI API compatible, so you can still use your, you know, your, your favorite tools, uh, the same clients, or you can use frameworks like LangChain or LlamaIndex or et cetera.

[18:16](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=1096s) So yeah, really excited to, uh, to kind of, uh, to be here and talk, tell a little bit about open source, uh, open source models and how we at Fireworks are focusing on productionizing that and scaling it up.

[18:30](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=1110s) Uh, go try it out, and you can also find us at the booth, uh, at the expo. Thank you. [upbeat music]
