AI Engineer World's Fair 2024
Productionizing GenAI Models – Lessons from the world's best AI teams
Read the talk
From an impressive AI demo to an assistant worth using
Lukas Biewald’s local voice assistant shows why production AI depends on exact outputs, fast responses, recorded experiments, and evaluations that guide the next change.
From a talk by Lukas Biewald
Before you start: Familiarity with LLM prompts, API calls, and basic Python is helpful; no model-training experience is required.
In production, but is it working?
How many people already have LLM applications in production at their companies? When Lukas Biewald asks the room, he estimates that more than 70% raise their hands. Why attend a talk on productionizing AI, then? An attendee answers: “It's not going great.” Deployment and satisfactory performance are different milestones.
Even the deployment category needs unpacking. Biewald estimates roughly 30% participation when he asks about custom solutions. Purchased solutions initially draw fewer hands, until someone points out that GitHub Copilot belongs in that category. The apparent adoption picture changes with what people count as an AI application.
Looking back ten and fifteen years—with traditional ML explicitly included—Biewald recalls an earlier expectation that AI would become accessible through AutoML or graphical interfaces. Instead, chat and conversational interfaces made that access commonplace. He reports seeing AI use in every company he speaks with and suspects that probably every Fortune 500 company is investing in custom AI, often combining GenAI with traditional ML. These are observations from his customer conversations, but the practical shift is clear: production use now extends beyond the companies people instinctively consider technologically adventurous. The question is no longer only when GenAI will reach production; it is how to make the applications already there work well.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A compelling demo hides the distance to production
AI is easy to demo and hard to productionize. An astonishing example can persuade a CEO to stop listening to customers and release something those customers find terrible. Biewald sees this in demonstrations from his own team and from startups pitching him. Software already has a gap between a promising prototype and a dependable product; AI makes that gap unusually easy to underestimate.
Biewald founded Weights & Biases to address a tooling gap in AI operations. Its users now span three groups, with different starting points:
- Foundation model builders develop the underlying models and push the platform’s scale.
- AI engineers build applications combining traditional ML and GenAI.
- Software developers can now attempt applications that previously required building a custom model, but must learn how to move beyond a demo.
The last group is much larger than the first two. Its arrival expands both the number of possible AI applications and the need for tools that support the development process.
Biewald says W&B works with nearly all foundation model companies, naming OpenAI GPT, Mistral, and Llama as model families built using the platform. Yet most of its customers are application builders, not foundation model developers. Their work reaches healthcare, agricultural technology, and manufacturing—a broader set of tangible applications than the search ranking, finance, and ad targeting opportunities he remembers from the beginning of his career. That application experience is the basis for the production practices that follow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The reusable asset is what the experiments taught you
Why do AI applications need different development tools? Adding an AI label to observability, CI/CD, or code versioning does not by itself answer that question. Biewald’s distinction concerns the work those tools must support: software development is generally additive, while LLM development is experimental. A new prompt or workflow is a hypothesis to test, not a feature whose addition guarantees progress. Outputs can vary, and a change can improve one case while breaking another. Broad behavioral evaluation therefore cannot be reduced to expecting every case to pass, even though narrow must-pass tests still have a role later in the workflow.
The learning is the intellectual property. A final model or prompt is an outcome of the development process. The unsuccessful prompts, rejected workflows, and reasons an approach failed tell the next engineer where to go—and where not to spend another week. If those discoveries live only in one person’s memory, the ability to continue iterating leaves with that person. Saving the final artifact alone does not preserve the work that produced it.
Reproducibility makes that learning usable, but recording enough information is painful, especially when experiments call nondeterministic endpoints. Biewald recommends passive capture by processes outside the engineer’s manual note-taking routine. People forget details even when they intend to document everything. Background tracking gives a colleague a way to reconstruct the experiment and pick up the investigation. Its practical return is shorter iteration time: preserved learning enables collaboration, and collaboration gets the application from demo to useful product faster.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turning a weather question into a function call
Biewald encountered the production gap in a project he started during time off after his second child was born. His older daughter, Matilda, asked Alexa to play her favorite song and was surprised that it did not know which song she meant. She repeatedly requested Baby Shark; the preference seemed obvious to her father. He speculates that privacy considerations might explain the missing memory, but the immediate product request came from Matilda: could he build an Alexa?
He looked at Mycroft, which he found dated and not using LLMs at the time. Then a neighbor showed him a seven-billion-parameter model running on a ROCK Pi costing roughly $200; Biewald tentatively identifies the model as Llama 2. That suggested a feasible local assistant. He built a library of skills for music, weather, math, and news. The API wrappers were straightforward: the weather skill was the sort of short function Copilot could generate. The difficult part was recognizing speech and translating the resulting words into the correct API call.
For the input “What's the weather in Boston?”, the on-device LLM must produce a weather function call with Boston as the location. The essential Python contract can be expressed as:
python
expected_call = 'weather(location="Boston")'
def matches_weather_call(generated_call: str) -> bool:
return generated_call.strip() == expected_call
This illustrates a strict output contract, not the project’s unspecified scoring implementation. A sentence explaining that the weather function should be called is not the same thing as producing the call. The downstream weather skill needs the function name and argument in the format it accepts before it can query its API.
The local stack combines Whisper for speech recognition with Llama for interpreting the request. Biewald chose llama.cpp to run the language model on inexpensive hardware; he also mentions vLLM and hosted providers as alternatives. Running the system himself was part of the appeal, and he reports that large LLMs could run in real time on a MacBook.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fast enough to answer, exact enough to run
Biewald skips the live demonstrations because he is unsure the audio will work, then turns to a constraint that becomes obvious when using the assistant: latency is part of correctness from the user’s perspective. Speech must first be transcribed, and the transcription then passes through an LLM to generate the function call. He describes delays beyond a couple hundred milliseconds as painful in this interaction; this is a responsiveness judgment, not a measured latency result. It motivates choosing a small model and running it quickly. At the time, that led him to Llama 2.
Biewald reports 0% function-call accuracy for Llama 2 with the default prompt. The accuracy figures throughout this project are his reported results; the recording does not specify evaluation-set size, train/test separation, scoring rules, or whether speech-recognition errors were included. The first intervention was ordinary prompt engineering: describe the available functions more clearly. That made the answer look closer to the desired call, but it still included a leading instruction to call the weather function rather than cleanly producing runnable output. Semantic proximity was not enough for the interface.
Switching to Llama 2-Chat, which was trained for conversation, raised reported function-call accuracy to 11%. Inspecting its errors and incorporating feedback raised reported accuracy to 75%. This was enough for a pleasing demo when Matilda phrased her request just right, but it remained an annoying assistant to use. The distinction matters: a carefully chosen successful interaction can hide the frequency of failures across ordinary requests.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A new model, then better training examples
Mistral arrived during the project. Replacing Llama 2 with Mistral raised Biewald’s reported function-call accuracy from 75% to 79% without fine-tuning. In his setup, the replacement used the same API, so he could take advantage of the new model without rebuilding the application. This is a useful reason to keep model replacement easy: a new release can improve the existing task at little implementation cost. But the remaining failures were still too frequent to satisfy his daughter.
The next step was fine-tuning. LoRA and QLoRA had made this more tractable, with hosted services also available for teams that did not want to run training themselves. Biewald ran QLoRA in a few hours on a basement GPU he remembers, uncertainly, as a 4080. The remaining prerequisite was a dataset.
His data-building procedure was small enough to do personally:
- Write a few examples of requests paired with the desired skill calls.
- Ask ChatGPT to generate additional examples.
- Review the generated examples by hand and remove unsuitable ones.
- Specify an output schema and supply more examples to elicit a wider variety of cases.
Biewald estimates that about 95% of the generated examples were good and that he obtained a couple thousand fine-tuning examples in roughly 15 minutes. The larger model supplied training material for the smaller local model; manual review remained part of that process.
Fine-tuning Mistral raised Biewald’s reported function-call accuracy to 98%. At that point, the assistant began to feel potentially shippable to him. The change was not merely a more impressive isolated answer: it was a substantial improvement in how often the system produced the required function call.
Multilingual behavior was a welcome byproduct. Biewald tested Japanese and tentatively identifies the displayed example as French. Whisper could transcribe other languages, and the LLM could interpret those requests. For anyone following the repository setup, multilingual use requires a multilingual Whisper checkpoint: Otto’s README defaults to the English-only base.en checkpoint. The broader attraction of the project was that useful, interesting AI systems were again within reach of an individual hobbyist, even after the growth of model sizes had made that seem doubtful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Failed experiments and the problem of version two
The project is available as Otto. Its progression shows why prompting and fine-tuning are not opposing strategies: prompt engineering helped from a poor starting point, and fine-tuning supplied later gains. Biewald extends that observation to production applications more generally, where retrieval-augmented generation, prompting, and fine-tuning may all contribute. He does not describe adding RAG to this assistant; the point is that the appropriate techniques can be complementary.
The clean sequence of improvements leaves out much of the work. The project took a couple of weeks, and most experiments failed. A colleague who inherited only the final model and this success story could easily repeat those failures before catching up. Biewald says he recorded the experiments—including unsuccessful ones—in W&B and made them available for inspection. That history is the concrete version of preserving learning rather than just preserving the winning prompt.
The first broader production practice is to build an evaluation framework and take it seriously. Biewald recalls asking an unnamed CEO on Gradient Dissent how his company tested GenAI applications. The CEO said he tested by vibes and declined Biewald’s offer to remove the remark. There is some value in that approach: personally trying a product can reveal glaring failures that should have prevented a release.
The limit appears when it is time to release a second version. A model or prompt change can make some requests better and others worse. A subjective impression will not reliably distinguish the relatively small improvement Otto gained from its model swap, so it cannot confidently justify the replacement. Evaluation makes those tradeoffs visible. When Biewald checked back with the CEO a year later, the company was doing substantial evaluation work after confronting this problem. A framework for comparing versions is what turns experimentation into a release decision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the whole loop, then measure what matters
Evaluation belongs alongside three other practices: start with a lightweight prototype, incorporate end-user feedback, and iterate. Biewald sees an enterprise failure mode in which teams try to perfect the first stage before building the second, indefinitely postponing the moment anyone can use the application. A small end-to-end prototype should reach its intended audience, whether internal employees or external customers. Their feedback then gives the next experiment a purpose.
When Biewald asks how many attendees feel confident in their evaluations, participation is much lower than in the deployment poll. Teams that do mature their evaluations tend to use several sets and techniques rather than one all-purpose score:
| Evaluation layer | Intended role |
|---|---|
| Restricted must-never-fail set | Target 100% on simple, critical cases |
| Small, quick suite | Return feedback in 20–30 seconds |
| Larger nightly suite | Cover more cases outside the immediate iteration loop |
The restricted set explains where an all-pass requirement can make sense. It protects a narrow contract; it does not imply that the application’s full behavioral evaluation should be perfect. The quick suite supports interactive experimentation, while the nightly suite supplies broader feedback.
Metrics must correspond to user experience or customer value. That relationship is application-dependent and difficult to establish. W&B initially expected teams to track perhaps 10–20 metrics per model or application. Biewald reports thousands or tens of thousands of metrics in production self-driving and mission-critical banking applications. Some customers even request regex search because they cannot find the metric they need among so many. The point is not to maximize the number of metrics; it is that a consequential application can fail in many distinct ways, and a single aggregate score may hide the failures users care about.
Actually running evaluations is what makes the other decisions tractable. Should this application use RAG? Would fine-tuning help? Is another technique worth adding? With a useful evaluation system, the team can try the change and answer for its own workload instead of treating the choice as a general debate. The evaluation loop connects a proposed intervention to evidence about whether it improves the product.
Biewald closes by inviting attendees to see Weave, with the qualification that it can help with some production problems, not all of them. The current Weave evaluation guide is a follow-up resource rather than an API demonstrated in this recording. Tools can support the loop, but the application’s users and failure modes still determine what it needs to measure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source and setup instructions for a local voice assistant using llama.cpp and whisper.cpp.
Original paper on memory-efficient fine-tuning using a quantized base model and low-rank adapters.
C/C++ inference engine with quantization and support for local CPU and GPU execution.
Speech recognition models with English-only and multilingual checkpoints, installation instructions and examples.
Further reading
Instructions for generating skill-call examples, fine-tuning with QLoRA, and logging evaluations in W&B.
Updates since the talk
Current guide to evaluating application outputs with curated datasets, custom scorers and repeated trials.
Read the complete timestamped transcript
- 0:00
[upbeat music] I guess as we're getting started, this talk is about productionizing AI.
- 0:18
But I'm curious, um, I keep asking this question and it keeps changing. How many of you folks have, um, LLM applications in production in your company?
- 0:28
Wow. That's probably the highest I've ever seen, more than seventy percent.
- 0:34
So why did you come to this talk? You know, [laughs] you figured it out, right?
- 0:38
It's not going great.
- 0:39
It's not going great? [laughs]
- 0:42
Yeah.
- 0:46
All right. You wanna put it in slideshow mode?
- 0:51
It is in slideshow mode.
- 0:51
Oh, it is in slideshow mode? Okay. Hey, you're showing my IP to the whole, uh, audience. [laughs]
- 1:02
Um...
- 1:05
Yeah. You might ask if they're custom solutions or, or purchased, uh, LLM solutions.
- 1:09
Oh, good question. Yeah. How many, uh, is, uh, custom solutions?
- 1:15
Interesting. That's maybe thirty percent. And how many purchased solutions?
- 1:20
Wow. Way, way less. [laughs] What are you, what are you purchasing?
- 1:24
Uh, well, I put like Copilot, uh, for from GitHub in that category of using, uh-
- 1:28
Oh, okay. Well, if you put Copilot in that category, how many have purchased solutions?
- 1:33
It's gotta be most, right? Interesting. Interesting. Thanks for the... [laughs] Any more... Should we crowdsource some more questions? No, we can begin. Awesome. Um, all right. So I wanted to talk, um, briefly about our experience with our customers productionizing GenAI models.
- 1:51
But I want to give you a little bit of context, first of all. Um, this is clearly preaching to the choir, but it's interesting to see that democratization of AI is here.
- 2:00
I don't know, how many people were working in AI ten years ago?
- 2:04
Nice. Fifteen years ago?
- 2:06
ML counts.
- 2:06
So yeah, ML totally counts. Hundred percent ML counts. Um, ML definitely counts as a type of AI, absolutely. Um, you know, I think people talked about the democratization of AI, you know, back when AI was kind of a lame, um, term to use.
- 2:20
But, uh, I think we thought that it might take the form of, you know, something like AutoML or more, you know, graphical user interfaces. But clearly democratization of AI has happened in a way that we didn't expect, clearly through these, these kind of chat and conversational interfaces and LLMs.
- 2:34
We now see AI in literally every company that we talk to in some way. And not just purchasing solutions, although there's, I think, probably every company in this room purchases what I would call an AI solution.
- 2:45
We also see quite a lot of custom solutions, and probably every Fortune five hundred right now is investing in custom AI solutions. More LLMs, more GenAI, but I would say most also some traditional ML.
- 2:59
And it's moving into production, right? You see it in this audience, you see seventy percent of this audience. But everywhere I go to give a talk, even with companies that you wouldn't necessarily think of as the most forward-thinking, they are starting to have GenAI applications in production.
- 3:11
So people keep asking me, "When is GenAI gonna move into production?" It really has moved into production, right? There's a lot more that could happen, but GenAI is in production right now in a lot of applications that you use today.
- 3:24
But I think the challenge everybody has, and people keep saying it, important to say, AI is so easy to demo and so hard to productionize. You see this with a lot of AI apps getting released.
- 3:33
There's something about AI that makes CEOs stupid. It makes them stop listening to their customers. It makes them willing to put things into production that are absolutely terrible. And I think the fundamental reason is that it's so easy to make an incredibly compelling demo.
- 3:49
I get these all the time from my own team. I get them from startups, you know, pitching me on potentially angel investing. You can make just an absolutely astonishing demo, and it's so much further from production than we're used to.
- 4:00
Even with software, which has this characteristic, AI is so much more easy to demo and so much harder to productionize. So let's talk about how to productionize it. I'll, I'll state I come from Weights & Biases.
- 4:12
This is a company I started that I feel very proud of. We've built an AI developer platform that helps develop AI engineers with ML, uh, applications or with GenAI applications.
- 4:24
And we started this company because we saw this massive tools gap for AI ops.
- 4:31
We now see that we have three types of customers. We see foundation model builders, which we've long served since the beginning. We see a bigger group of AI engineers that are doing a blend of ML applications and GenAI applications.
- 4:44
And we see a new type of audience for our software in software developers that are now getting used to the fact that AI applications are so easy to demo and so hard to productionize, and trying to figure out how to productionize those applications.
- 4:58
What's exciting for us, and exciting for everyone, I think, is that the software developers are so much more numerous than ML engineers and foundation model builders, right? Now that companies can use software developers to do what traditionally would've taken a custom model, it opens us up to do so many more AI applications. [clears throat]
- 5:18
And from our perspective, this is a really exciting development because the market's so much bigger, and that's why we see a whole LLMOps tools track at this conference and so many companies popping up to help with this problem.
- 5:29
I feel really proud, and I always wanna say, we work with nearly all the foundation model companies. They have special needs, but they do push our scale. So a- almost all the LLMs that you would've used, from OpenAI's GPT to Mistral to Llama, these, these were built using the Weights & Biases platform.
- 5:44
So we have years of experience dealing with, with large scale. But it's also important to say almost all of our customers, and we, we have thousands of customers, are not foundation model builders, right?
- 5:53
They're, they're companies in, in nearly every industry. And the really fun part about being in the AI space right now- Is that there's applications to every industry. So we see healthcare, we see ag tech, we see it in manufacturing.
- 6:04
I remember when I, when I came out of school and wanted to get into the AI industry, it was really kind of ranking search results or, or going to Wall Street or targeting ads, right?
- 6:12
Which are, aren't the most exciting applications. Now, it's things that you can touch and feel. It's things that you can explain to your parents. This is, this is incredibly exciting development.
- 6:21
But what makes this hard, and I, I've thought a lot about this, what makes it hard, and, and, and what makes kind of AI engineering a different type of process with a different set of tools needed, because every software company has now launched an AI tool version of itself, right?
- 6:35
They, they call me up every day and they say, "You know, what do, what do you think? We're, we're launching, you know, AI version of observability. We're launching AI version of CI/CD.
- 6:43
We're launching AI version of, um, you know, code versioning."
- 6:48
Why don't those take, right? Why do we need a new set of tools? Why is Weights & Biases able to compete against companies that are, you know, much bigger and have lots of experience?
- 6:55
And I think it's fundamentally that the software development process, which I love, is a linear process.
- 7:01
You, you, you add features, you add code, and generally things improve over time. And the AI process is experimental. Almost all the things that you do when you're using an LLM is, is trying it, seeing what happens, right?
- 7:15
And it's, it's fundamental non, fundamentally nondeterministic. You don't know what's gonna happen. You're never gonna have a CI/CD test that's meaningful where 100% passes. So instead of kind of adding things all day long and, like what you do when you write code, you're doing lots of experiments.
- 7:28
I think this is a fundamentally different workflow, and what that means is that, you know, when you develop software, your code is your IP. Companies recognize this, and they take a lot of, uh, pain to protect their IP.
- 7:39
But when you're developing AI, it's the learning that's your IP. It's not the model that you build, right? The model's kind of an outcome, right? The prompt that you discover is kind of an outcome.
- 7:48
But the things that you learned along the way, all the prompts that you tried, all the workflows that you tried that didn't work, that's really the learning. That's really the IP that you have.
- 7:56
And if you're not saving that, when the person that figured out walks out the door, your IP walks out the door with them because you're not gonna be able to iterate further from there.
- 8:04
And so what that means is reproducibility matters, right? You don't really save your IP, you don't save your learnings if you can't reproduce your learnings, right? And this is a obvious thing.
- 8:12
I think everybody asks for reproducibility. We've always talked about a reproducibility crisis in AI. But reproducibility is incredibly hard. It's incredibly painful to keep track of everything so you can reproduce your stuff, especially with these nondeterministic endpoints.
- 8:25
And you see a reproducibility crisis in many fields, not just software.
- 8:30
And so my perspective is what you should be doing is tracking everything, and the tracking needs to happen with outside processes. You can't rely on humans to write everything down.
- 8:39
No matter how much they want to, they're gonna forget. You need to track everything in the background passively to have real reproducibility. And the real reason that you wanna do this is not just protecting your IP, but because now you can collaborate, right?
- 8:50
Now someone else can pick up the learnings that some engineer had.
- 8:55
And when you can collaborate well, you can iterate faster. And the iteration time is really the key to bringing things to market, right? If you wanna go from demo to production quickly, you need to bring down that iteration time, you need to have good collaboration, and you need speed.
- 9:09
So that's the real ROI, I think, of all these tools that we might be building. Now, we work with lots of enterprises and I, I, I wanted to give you an enterprise example, but more than that, I wanted to give you a real world example that, that mattered to me.
- 9:22
So this is very similar to what many of our enterprise customers are doing, but this is actually a project that I did, um, in a, in a, a few weeks that, that I took off after my second child was born.
- 9:34
Um, and actually, this is my first kid, um, Matilda, and, um, she was really demanding my attention after my, um, baby was born, and we were talking a lot, and she was actually, um, asking Alexa to play her favorite song.
- 9:46
And it was really interesting. I mean, Alexa's an interesting product. I, I really love it. My daughter is kind of connected to it, um, talks to, talks to her a lot.
- 9:53
But she was really kind of shocked when she asked Alexa to play her favorite song, and Alexa didn't know what her favorite song is because she asks Alexa to play "Baby Shark," like literally like five times a day.
- 10:01
So it's like there's no data science needed to know what the favorite song is. [laughing] But Alexa, um, couldn't f- couldn't, didn't know this. And I think there's kind of data privacy reasons that Alexa doesn't keep this, but it's kind of remarkable with all these amazing applications out there to see Alexa not able to do it.
- 10:15
And so, uh, my daughter was asking me, "Hey, could you build an Alexa?" And I was like, "I think maybe I could, actually, with, with LLMs today." So I kind of looked around with the open source Alexa alternatives where there's something called Mycroft that's, um, that's out there, but, but, um, kind of out of date.
- 10:29
Doesn't really use LLMs. And then my neighbor actually was showing me that he got, um, a seven billion, um, model, uh, I think Llama 2, running on a Rock Pi, which is like a $200 device, similar price to Alexa.
- 10:40
So I thought, "Okay, if you have that, maybe we could build an architecture that would, um, kind of do the things that Alexa does." So built a library of skills, um, things that we ask Alexa to do, like play music, um, ask what the weather is, do math problems, um, check the news, things like that.
- 10:56
And these skills are really simple, right? So you know, like a- getting what the weather is is like a five line thing that actually, um, Copilot can generate instantly and works phenomenally well.
- 11:05
There's APIs here. Uh, the challenge is s- is actually understanding the speech and then getting the natural language into an API call. So kind of a simple architecture here, right?
- 11:14
So the weather comes in. You ask what's the weather in Boston. It goes through an on-device LLM to translate it to kind of Python looking code that says, "Weather location equals Boston."
- 11:23
Has to look like that because it's gonna call that function on the weather skill that I wrote that calls a, a weather API.
- 11:30
Um, yeah, I think Llama, I think, and Whisper is a fantastic open source stack here. Llama.cpp is one of the many options to, to run Llama on, um, cheap hardware.
- 11:39
There's vLLM and others, um, and lots of companies that'll do it for you. I kind of wanted to do it myself, so Llama.cpp was a great choice. Um, and you can run large LLMs on a MacBook in, in real time.
- 11:50
Um, maybe I'll skip the demos here. Not sure the audio is working. Um, the... What you will, if you, if you actually do try to do this yourselves, and a lot of our customers do try to do this themselves and they discover the same thing, latency really matters a lot, right?
- 12:04
Because you're actually transcribing audio and then you're running the aud- you're running the transcription through an LLM- To, um, make this function call. And so if that takes longer than a couple hundred milliseconds, it's incredibly painful.
- 12:16
So you need to go kind of small model here, and you need to, to run it fast on, on hardware. So that's why we went with, um, Llama 2 at the time, and put it in a default prompt, and it didn't work.
- 12:26
0% accuracy. And so every kind of customer goes through this journey to try to improve the accuracy or whatever the real kind of user-facing metric is. I'll tell you briefly about my journey to improve this.
- 12:37
Um, you know, first thing I did was I did some prompt engineering and just kind of common sense to make the prompt better, kinda lay out what functions are available.
- 12:45
Um, the model kind of gets closer, in a sense. Now it's saying, "Call colon weather location equals Boston," but still actually not in any kind of format that I can run.
- 12:52
Here, I need really exact accuracy because I'm generating, um, code, function calls. Switched to Llama 2 Chat. Um, there's so many models to try these days. Um, Llama 2 Chat was trained on conversations, performed better in this case, and got the accuracy up to 11%.
- 13:09
Still not possible to ship this, um, but I kind of looked at the errors it was making. Looking at the errors, incorporating feedback, got the accuracy up to 75%.
- 13:17
So pretty satisfying, pretty iterative process, probably typical to what you're doing or what your engineers are doing. Now it's kind of a cool demo. My daughter is, like, mildly impressed if she says the, the thing just right.
- 13:29
But actually, 75% accuracy is incredibly annoying. It-it-it's, it's still not-
- 13:34
Okay, so-
- 13:35
Um, it's still not gonna, gonna work so well. Switched to Mistral. Um, Mistral came out mid-project here. Um, and Mistral's model for free, exact same API to Llama 2, got the accuracy up to 79%, right?
- 13:46
So this is sort of happening in real time, probably happening to you also. If you're ready to switch the models quickly, you can get free accuracy improvements as, as new models come out.
- 13:56
77% accuracy is still really annoying, trust me. Um, [laughs] so I thought, "Okay, I need to, I need to improve this." Still not impressing my daughter. Um, and so I thought, "Okay, let's try some fine-tuning."
- 14:07
And now, um, you know, fine-tuning used to be a hard thing to do, especially on these really large models. Um, LoRA, QLoRA has made this a lot more tractable.
- 14:16
Now there's probably 20 companies here that will actually do this, um, for you for, for super cheap. So, you know, no excuse for not fine-tuning once you get to a point where, um, you wanna make that kind of last improvement of accuracy.
- 14:27
Um, I was able to run QLoRA on my GPU in my [REDACTED:location]. This runs on kind of standard hardware, I think it's a 4080, um, and, and ran a dataset in, in a few hours.
- 14:38
But you need the dataset. So I manually created a couple examples here, you know, using a larger model, in this case ChatGPT, to create more examples. Worked phenomenally well.
- 14:46
I think 95% of them were good. You know, I kinda went through by hand and, and filtered some out. But that, you know, in a, in, you know, 15 minutes, I have, um, a couple thousand examples to fine-tune on.
- 14:56
Um, I gave it a schema to format the answer in, and once it saw more examples, it was actually ge- able to generate more interesting examples kind of from that, that dataset.
- 15:07
Um, fine-tuning plus Mistral actually led to 98% accuracy. So this is super satisfying project. 98% accuracy actually starts to feel like, "Hey, maybe we could, we could ship this thing."
- 15:17
Um, and as a surprising byproduct, it actually worked for other languages. So I was using Whisper model that, that works really well in other languages. Um, Japanese is the one that I tested on.
- 15:27
I think this is, uh, French here. Um, but, but the, the LLM's also work in other languages. So we've kind of built this amazing multilingual Alexa as kind of a fun side project.
- 15:37
Which by the way, I think is exciting because I think there was a time when these models got really big, and there's always been kind of a vibrant hacker culture, I think, in AI, but started to wonder if, if you could only do really expensive projects to get anything interesting done.
- 15:50
I think now there's just an incredible amount of, um, things that, that people can do.
- 15:56
Um, so you can find this project at, uh, github[REDACTED:url]. Um, and the, and the lessons learned, stepping back, are exactly the things that our customers are discovering. It's exactly the things that you're probably discovering.
- 16:07
Prompt engineering works really well, especially when you sto- start with a terrible prompt. And then fine-tuning improves the performance at the end. People often talk about, "Hey, should I be using RAG?
- 16:16
Should I be using fine-tuning? Should I be using prompt engineering?" I think almost always in the end, to get something shipped into production and working well, you end up using a whole bunch of these different, um, methods.
- 16:26
Everything is making iterative improvement. Gradually, we get to the accuracy that we need. Um, the other thing that I kind of hid from you in this project is this did take me a couple weeks to get done, and most of the experiments didn't work, right?
- 16:38
So, you know, in, in the real world, you know, if I gave this to a colleague and they tried to iterate o- on top of it,
- 16:45
they would-- if they just saw this talk here, they wouldn't realize all the things that I tried that failed, and they'd probably start to do them over and kind of have to experience the same failures to catch up to where I was.
- 16:56
And so, um, we actually kept track in WandB of, of all the, um, the failures and experiments, and you can go in there and, and look at them yourself.
- 17:03
But I think this is generally best practice.
- 17:06
Um, so what, what are the lessons? You know, people come to me now e-every day asking, "Hey, you know, how do I get LLM apps from demo into production?
- 17:16
What are the kind of broader lessons besides using, um, Weights & Biases?" which you obviously are already doing. Um, and I think the, um... I think that there's kind of four things that we see our customers do where they actually get things into production, they actually make them successful.
- 17:32
Um, the, the first is that they build an evaluation framework, and they take this really seriously. We've had people-- I wrote a-- I do a podcast called Gradient Dissent, where I talk to a lot of people.
- 17:41
Um, and, and one prominent CEO of a well-known company, um, came on the podcast, and I asked him, you know, "How do you do testing? How do you test your GenAI applications in production?"
- 17:51
And he said, "I, I, I test these applications by vibes." [laughs] [laughs]
- 17:56
And, um, I asked him, you know, "Hey, do you want me to remove that, you know, from the, the pod?" Like, we're not trying to, like, embarrass people on, on our podcast, you know?
- 18:03
And he's like, "No, no, it's great. I, I love it," you know? Um, and, and I think that there actually is a kind of a value to testing by vibes.
- 18:10
Like, I think, um, if more CEOs tested by vibes, you might actually not have some of the failures that, that you see. I mean, some of these things that get released are, are so embarrassing that I think a vibes check would've actually, um, caused better decision-making.
- 18:23
But When you're only testing by vibes, the problem is you can't release a V2, right? Because what happens is, unlike in software development where you're typically adding features and mostly moving forward,
- 18:34
any of these new things that you try are gonna make some things better and some things worse, and the vibes check is not gonna tell you the difference between a 75% accuracy and a 79% accuracy.
- 18:43
So he never would've been able to make that switch from Llama 2 to Mistral because it just wouldn't have been clear if it's better or worse. And so taking time to build an evaluation framework is really the, the foundation of getting these things into production and working well.
- 18:56
I actually checked back with that CEO a year later, and he was doing quite a lot of evaluation because he kind of realized that thing where you wanna get that second model out, and you, you don't know how to do it.
- 19:05
Um, the other thing is starting with a lightweight prototype. We see typically, this is kind of enterprise anti-pattern, but we see it a lot at Weights & Biases, where people kinda wanna build the first step, and kinda nail that, and then build the second step, and they kinda never get something into the hands of the end user,
- 19:19
which might be an internal audience or it might be a customer. I think this is kind of basic agile product development, but somehow people forget this with GenAI, so it's worth mentioning because we see this failure mode so frequently.
- 19:30
And then, of course, incorporating end user feedback in lots of different ways is, is critical.
- 19:38
Um, and then iterate. So I'll say, you know, evaluation best practices because I think it's such a cornerstone, and I think... Well, actually, let me ask you this. How many, how many people in the audience feel like they're doing solid evaluations of their GenAI, um, models?
- 19:53
GenAI applications? Interesting. So, so a lot less. It seems like you feel people are, you all are not feeling great about your, um, [laughs] evaluation frameworks. Um, I think, I think what we see, where people kind of end up, um, is lots of different evaluation sets and techniques.
- 20:09
Um, so we see a lot of our customers will do some things like, "Okay, these are the things we can never get wrong, and we wanna have, like, 100% on this simple test."
- 20:15
And there's lots of tricks to getting that right. Then people wanna have, you know, something that's sort of very quick to run, so they can, they can kind of run it in real time.
- 20:23
They can test something new in 20 or 30 seconds. And then our customers typically will have much bigger evaluation sets that run nightly and give feedback. That's where people end up, and I think that's really important to do all the different kinds of evaluations that might matter for your application.
- 20:37
And then, you know, the other thing that's very hard to do, but I mean, MLOps has been doing this for decades, is trying to make sure that the metrics correlate with the user experience or the value that you actually provide for your customer.
- 20:48
Totally application dependent, um, very hard to do. I mean, when we started Weights & Biases, we kind of thought people might wanna track, I don't know, like 10 or 20 metrics per m- model that they build or per application, and we see with, with people that are really in production, like a typical, you know, self-driving car company
- 21:04
or, like, a typical bank with a mission critical application, we see thousands of metrics. We see tens of thousands of metrics. We get requests like, "Hey, can I do regex search across my metrics?
- 21:13
Because I have, like, so many metrics, I can't even [laughs] I can't find, you know, the, the, the metrics." And, and I think that's because, um, you really wanna get these metrics working and, and connected to your user experience, and there's so many different ways that something could fail.
- 21:25
But then the most important thing is to actually do it, right? It seems like a lot of people in this room maybe haven't started to do it yet, but doing evaluation is so critical because you can't, you can't improve any of these other things.
- 21:36
And when people come to me and they ask me questions like, you know, "Should I be doing fine-tuning? Should I be doing RAG?" You know, "Should I be doing XYZ?"
- 21:45
What that tells me is they don't have a good evaluation system in place, right? Because it shouldn't take you very long to have an answer for yourself, for your own application, um, if one of these techniques works or not.
- 21:56
Um, so look, you know, we're here at this conference, and this is a lot of fun. Um, if you wanna come by our booth, the Weights & Biases booth, we can show you, um, Weave, which we can help you with some of your problems getting, um, getting AI applications into production, but, but not all of them today.
- 22:10
But we'd love to talk to you about what we're doing, and we'd love to hear from you about how your experience has been. Thank you. [audience applauding] [upbeat music]