Customized, production ready inference with open source models: Dmytro (Dima) Dzhulgakov

[on-hold electronic music] Hello, everyone.

So my name is Dima. Um, as mentioned, uh, unfortunately my co- uh, co-founder Lin, who was, uh, on the schedule couldn't make it today because of some personal emergency.

So you, you get me. And as you saw, we don't have yet AI to figure out video, uh, projection, but we have AI for a lot of other things.

Uh, so today I'm gonna talk about Fireworks AI and generally gonna con- continue this theme which Caitlin started about open models, uh, and how we, uh, basically focus on productionization and customization of, uh, open source models, uh, in inference at Fireworks.

But first, uh, as a introduction, uh, what's our background? Uh, so f- the founding team of Fireworks, uh, comes from PyTorch leads at Meta and some veterans from Google AI.

So we combined have, like, probably decade of experience in productionizing AI in some of the biggest companies in the world. And I myself personally has been core maintainer of PyTorch, uh, for the past five years, so topic of open source is really close to my heart.

And since we kind of led this revolution of open source toolchain for deep learning through our work through, on PyTorch and some of the Google technologies, we really believe that open source, uh, models are the future also for, for,

for GenAI application, and our focus at Fireworks is precisely on that. Um, so

I mean, uh, how many people in the audience actually, like, use GPT and deploy it in pr- production?

And how many people-- how many folks use open models, um, or deploy to production? Oh, okay. So I was about to convince you that share of open source models is gonna grow over time, but it looks like in this audience it's already [laughs] already sizable.

But nevertheless, um, so why, why basically this trade-off? Why go big o- uh, why go small? Um, currently still, like, bulk of production inference is still based on proprietary models and, uh, the catch is that it-- those are really good models and they are often fr- frontier in ma- in, in many domains.

Uh, however, the catch is that it's one model which is good in many, many sense, and it's often served, uh, in the same way, uh, regardless of the use case.

Which means that maybe if you have batch inference, uh, on some narrow domain or you have some super real-time use case, uh, where you need to, you need to do, like, voice, uh, assistant or something, uh, those are often s- served from the same infrastructure without customization.

In terms of model capabilities, it also means, yeah, like GPT-4 is great, uh, or, uh, Claude is great and can handle a lot of sense, but you are often paying a lot for additional capabilities which are not needed in particular use case.

You don't really need customer support chatbot to know about 150 Pokémons or be able to write, write you poetry, uh, but you really want it to be really good in the particular, uh, narrow domain.

So this, uh, can, uh, this can discrepancy for large models leads to several issues. One, as I mentioned, is high latency because using a big model means, uh, longer response times, uh, which is particularly important for real-time use cases like voice assistants.

It gets more and more important with agentic stuff because, uh, for stuff like, um, for example, N- Next is gonna be a Devin, right? Like you, you need to do a lot of steps for, like, something like agent-like application to do reasoning and call the model many times.

So latency is really, really important. And often you see that you can pick smaller models like Llama or Gemma [laughs] which we, which we just talk about and achieve the, for a narrow domain, uh, same or better quality while being, you know, up to 10 times faster.

Uh, for example, for some of the function calling use cases, like externally benchmark from, uh, from Berkeley, yeah, like the... You get similar performance from fine-tuned Llama 3 at 10X speed.

Cost is also, uh, is, is also an issue. If you're running a big model for, uh, on a lot of traffic, you know, even if you have perhaps, I dunno, 5K tokens prompt and 10,000 users and each of them calls LLM 20 times per day, you know, on GPT-4, even on GPT-4o, it probably adds up to, like,

10K per day or something like several million per month. Also several, several million per year, which is a sizable cost of a startup. You can easily cut that with much smaller models and that often we see as a, uh, as a kind of motivation for reaching out for smaller and more customizable models.

Uh, but really the, uh, like, where open models shine is, uh, domain adaptability and that comes, uh, in two aspects. First, um, there is so many different fine-tunes and customizations.

Uh, I think Caitlin was mentioning about, you know, Gemma built Indian languages adaptations. Like, there are models specialized for code or for medicine. If you head to Hugging Face, there are, like, tens of thousands of diff- different model variants and because the weights are open you can always customize to your particular use case and tune and, uh,

tune quality specifically for, for what you need. So open source models are great, so what are the challenges? Uh, the challenges really come from three areas. Uh, first, like, what we usually see when people try to use, you know, open model, something like Gemma or whatever, uh, or, or Llama 2D, uh, you run into com- complicated setup

and maintenance, right? You need to go and find GPU somewhere. You need to figure out which frameworks to run on those. You need to, like, download your models, maybe do some performance optimization and tuning and you kind of have to repeat this process end to end every time the model gets updated or new version is released, et

cetera. Uh, on optimization itself, uh, there is, especially for LLMs but generally for GenAI models, there are many attributes, uh, and settings which are really de- really dependent on your use case and requirements.

Somebody needs low latency, somebody needs high throughput. Prompts can be short, prompts can be long, et cetera. And choosing the optimal settings across the stack is actually not trivial and as I show later, in many cases you can get multiple X improvements from doing, from doing this efficiently.

And finally, like, just getting it production ready is actually hard. Uh, if y- as you sc- kind of go from experimentation to production, even just BBC and GPUs on public clouds is not, not as easy because GPUs are finicky and not always reliable.

Uh, but getting to enterprise scale requires, you know, sca- all the scalability technology, telemetry, observability, et cetera. So those are, uh, things which we focus on, uh, solving, uh, at Fireworks.

So starting with efficiency, we built, uh, our own custom service stack, which we believe is one of the fastest, if not the fastest. Uh, we did it, uh, did it from the ground up, meaning from writing our own, you know, CUDA kernels all the way to customizing how the s- uh, stuff gets deployed and orchestrated on the

service level, and that brings multiple optimizations. But most importantly, we really focus on customizing the service stack to your needs, uh, which basically means for your custom workload and for your custom cost and latency, uh, latency requirements, we can, we can tune it for, uh, for those settings.

What does it mean in practice? And what does customization means in practice? Uh, for example, many use cases use RAG and, uh, use very long prompts. Uh, so there are many settings you can tune actually on the runtime level and the deployment level to optimize for long prompts, which often can be repeatable, so caching is useful or

just m- tuning settings so the throughput is higher while maintaining latency. So this is i- independently benchmarkable. If you go to, uh, you know, Artificial Analysis and select Long Prompt, where Fireworks actually is the fastest, even faster than some of the other providers which are over there at Expo booth.

Uh, and, uh, we don't only focus-- we don't only focus on LLM inference. Uh, we are focused on many modalities. Uh, as an example, for image generation, we are the fastest provider serving SDXL.

We're also the only provider serving SD3, uh, uh, in, in, uh, Stability's new model because their API actually routes to our servers. Um, and finally, as I mentioned, with the, like, LL- LLMs, like customiza- especially for LLMs, customization matters a lot.

Uh, one requirement, like one problem we have to think about performance of LLMs, often it's useful for use cases, is to think about maxi- like minimizing cost under a particular latency constraint.

We often have customers come and say, like, "Hey, I need to, like, have this, my interactive application, I need to generate that many tokens under two seconds." And that's where-- that's really where, like, cross-stack optimizations shine, uh, whereby, uh, tuning the particular, like, latency cutoff and changing many settings, you can deliver much higher throughput, multiple times higher

throughput, uh, which bas- higher throughput basically means fewer GPUs and lower cost.

Uh, in terms, in terms of, uh, model support, we support, support best quality open source models. Uh, you know, we heard about Gemma now, obviously, uh, uh, Llama's, uh, some of the ASR and text-to-speech models, uh, pretty much from, from many providers.

We also work with model developers. For exampl- uh, for example, Ye Large, uh, in US is also served on, uh, on Fireworks, launched, uh, laun-launched, uh, last week. And, uh, as a kind of pla- platform capabilities, as I mentioned, we have a lot of op- open source models to st- uh, to get you started or cus- uh,

customized ones. We do some of the fine-tuning of those models, uh, in-house. So I'm gonna talk a little bit about function calling specialized models, uh, later on, or we do some of the, uh, vision-language models fusion, uh, ourselves, which we re-release as well.

And of course, the key for open source, um, mo- open model development is that it can tune for a particular use case. So we do provide a platform for fine-tuning, uh, whether you're bringing your data set collected elsewhere or collecting it live, uh, with the feedback wh-when served on our platform.

Uh, specifically on customization is like one, uh, interest... One interesting feature which a lot of people expe- starting to experiment with models, uh, find interesting is if you try to fi-fine-tune and deploy the resulting model, how, uh, how to serve it efficiently.

Uh, it turns out if you do LoRA fine-tuning, which a lot of folks do, uh, you can do, uh, s-smart tricks and deploy multiple LoRA models on the same GPU, uh, actually thousands of them, which means that we can give you still serverless inference with paying for, per token, even if you have like thousands of m- uh,

model variants, uh, sitting and deployed there without having to pay any fixed cost.

Of, of course, single model is all great, uh, but what we see increasingly more and more in applications is model is not the product, right, uh, uh, by itself.

You need a kind of bigger system, uh, in order to solve target application. And the reason for that is because, uh, models by themselves tend to hallucinate, so you need some grounding, and that's where like RAG or a-access to external, uh, knowledge bases comes in.

Uh, also, we don't have, you know, yet an industry magical multimodal, uh, AI across all the modalities, so often you have to kind of chain multiple types of models.

And of course, you have all these like external tools and external actions which, uh, uh, kind of end-to-end applications might want to do in agentic form. Uh, so I think the term which I really like, which is, which is, like, popularized by Databricks, is like compound AI system.

But basically, re- increasingly seeing, like, transition from just the model being the product to kind of this combination of maybe like RAG and function calling and external tools, et cetera, built together as the product.

And that's pretty much the direction which we kind of see, uh, th-this field mo-, uh, moving along, uh, o-o-over time. So what does it mean from, uh, from kind of our perspective, what we, what we do in this case?

Uh, so, uh, we see kind of as a function calling like agent as a-- at the core of this, uh, emerging architecture, which might be connected to either domain specialized, uh, models, uh, uh, se-served on our platform di-directly or maybe tuned for partic- different needs, and connected to external tools, uh, maybe it's a code interpreter or maybe

it's like external APIs somewhere, uh, with really like this kind of central agentic, uh, view, uh, kind of central, central model kind of coordinating and trying to triage the, uh, user, user requirements if it's, for example, a chatbot or something.

Uh, you probably all, uh, heard about, like, function calling, you know, popularized by OpenAI initially. That's, uh, that's basically the same idea. Uh, so yeah, the function calling is really like a how to- How to connect, uh, LLM to external tools and exter- e- and external elements.

What does it mean in practice? So we actually, uh, uh, fo- focus on fine-tuning models specifically for function calling. So we released a series of models like that, like the latest one, Fire Function, which was released two, uh, two weeks ago.

And, uh, what you can do with that is, uh... Oh. Okay, if I manage to click.

If I manage to click on this button. [laughs]

Uh, w- what it means is, like, you can build, uh, applications which kind of combine freeform general chat, uh, capabilities with function calling. So in this case, uh, this is, you know, this Fire Function has some, uh, chat capabilities.

So you can see, you can, like, ask it what, what you can you do, and it has, like, some self-reflection to tell you what it can do. It's also connected in this demo app to a bunch of, uh, external tools, so it can, uh, query, uh, like, stock quotes.

It can plot some charts, all those, like, external APIs. Uh, it can, um, also gener- gen- generate images, but what it really needs to figure out is how to translate user query into, do complex reasoning, translate it into function calls.

So for example, if you ask it to generate a bar chart with top three, uh, like with stocks of top cloud providers, like the Big Three, it actually needs to do several steps, right?

It needs to understand that, like, top three cloud providers means, you know, AWS, GCP, and, uh, a- and Azure, uh, right? And Azure is o- owned by Microsoft. It needs to, uh, then go do function calls querying their stock prices, and finally, it needs to combine those information and send it to chat plotting API, which is what

just happened, uh, in the, in the background. Uh, another important aspect which you have to do for, like, e- efficient, uh, kind of function calling chat capabilities, you need to have contextual awareness.

So if I ask it to add particular, uh, if I ask it to add Oracle to this graph, it needs to understand what I'm referring to and, like, still keeps the previous context and regenerates the image.

And finally, you know, if I switch to a partic- into a different topic, it kind of needs to drop the previous context and understands that, like, "Hey, this is less, uh, this, uh, historical context is less important.

I'm gonna start from scratch." So there is no, like, Oracle in that cat photo or whatever. Uh, so you know, this particular demo is, uh, i- is actually open source.

You can, like, go to our GitHub and try it out. It's built with Fire Function, and it builds with, like, a few other mo- a few other models, including, like, SDXL, which are run on our platform.

Uh, the model itself for function calling is actually open source. Uh, it's on Hugging Face. I mean, you can of course call it on, uh, at, at Fireworks with, for optimal speeds, but you can also, uh, run it locally if you want.

It uses a bunch of, uh, you know, functionality on our platform, uh, for example, like structure generation, uh, with, like, uh, with JSON model grammar mode, which I think was similar to some of the previous talks from, like, outline guys, uh, which were talking here yesterday.

Uh, yeah, so finally, try, try it out. And generally, like, how to get started on Fireworks. So if you head, head out to fireworksai.models, you'll, uh, you'll find a lot of, uh, open, open source, open base models, which I mentioned about.

They're available in the playground. In terms of product offering, uh, we have this kind of range which can take you from early prototyping all the way to enterprise scale.

So you can start with serverless inference, which is, you know, not different from, uh, you know, getting to OpenAI p- uh, OpenAI playground or something, where you pay per token.

Uh, it's a constant price. You don't need to worry about, like, hardware settings or anything. As I mentioned, you can still do fine-tune, so you can, uh, you can do hosted fine-tune on our platform.

You can bring your own LoRA adapter and still serve it serverless. As you kind of graduate to, like, maybe, like, a startup, and you graduate to, uh, more production scale, uh, you might want to go to on-demand where it, where it's more, like, dedicated hardware with more settings and modifications, uh, for your use case.

Uh, you can bring your own custom model fine-tuned from scratch or do it on our platform. And finally, as a kind of, if you scale up, uh, to bigger volume and want to go to enterp- enterpr- enterprise level, where it's kind of discounted long-term, uh, long-term contracts, and we also will help you to kind of personalize hardware

setup and do some of those tuning, uh, for performance, which I, uh, which I talked about earlier. And in terms of use cases, I mean, we're running production for m- uh, for many, many companies, ranging from small startups to big enterprises.

We're serving, like, last time I checked, like, more than 150 billion tokens per day. So you know, companies like Quora built chatbots like Poe. Uh, Sourcegraph and Courser, which I think, I think Courser had a talk here yesterday.

They use us for, like, some of their code assistant functionality, and their, like, latency is really important. Uh, as you can imagine, you know, folks like Upstage and Liner are building, like, different assistants and agents, uh, on top of that.

So, uh, we are definitely production ready. Go try, try it out. Uh, finally, we care a lot about developers, you, uh, you guys. Um, so actually, this is external numbers from, uh, like, last year, LangChain State of AI stuff, where turns out we are one of the, like, a- after Hugging Face, the most popular platform for where

people pull models, which is great. Which is very, was very nice to, uh, nice to hear. And again, for, like, for getting started, just, uh, you know, head out to, uh, head out to our website.

You can go in the, go play in the playground, uh, right away. So for example, you can run, you know, Llama or Gemma or whatever at the, at the top speeds.

Um, and kind of go start building from there. I'm really excited to see what you can build with open models of Fire Function or some stuff which you can, uh, which you can fine-tune on, uh, on your own.

And yeah, last point, we are, as I mentioned, OpenAI API compatible, so you can still use your, you know, your, your favorite tools, uh, the same clients, or you can use frameworks like LangChain or LlamaIndex or et cetera.

So yeah, really excited to, uh, to kind of, uh, to be here and talk, tell a little bit about open source, uh, open source models and how we at Fireworks are focusing on productionizing that and scaling it up.

Uh, go try it out, and you can also find us at the booth, uh, at the expo. Thank you. [upbeat music]