# Productionizing GenAI Models – Lessons from the world's best AI teams

Lukas Biewald, Weights & Biases

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/uiq95JYpBGY-productionizing-genai-models-lessons-from-worlds) · [Video](https://www.youtube.com/watch?v=uiq95JYpBGY)

## Productionizing GenAI Models – Lessons from the world's best AI teams

Lukas Biewald’s local voice assistant shows why production AI depends on exact outputs, fast responses, recorded experiments, and evaluations that guide the next change.

Familiarity with LLM prompts, API calls, and basic Python is helpful; no model-training experience is required.

### In production, but is it working?

[0:18](https://www.youtube.com/watch?v=uiq95JYpBGY&t=18s)

How many people already have LLM applications in production at their companies? When Lukas Biewald asks the room, he estimates that more than 70% raise their hands. Why attend a talk on productionizing AI, then? An attendee answers: “It's not going great.” Deployment and satisfactory performance are different milestones.

Even the deployment category needs unpacking. Biewald estimates roughly 30% participation when he asks about custom solutions. Purchased solutions initially draw fewer hands, until someone points out that GitHub Copilot belongs in that category. The apparent adoption picture changes with what people count as an AI application.

Looking back ten and fifteen years—with traditional ML explicitly included—Biewald recalls an earlier expectation that AI would become accessible through AutoML or graphical interfaces. Instead, chat and conversational interfaces made that access commonplace. He reports seeing AI use in every company he speaks with and suspects that probably every Fortune 500 company is investing in custom AI, often combining GenAI with traditional ML. These are observations from his customer conversations, but the practical shift is clear: production use now extends beyond the companies people instinctively consider technologically adventurous. The question is no longer only when GenAI will reach production; it is how to make the applications already there work well.

![Slide stating “Democratization of AI is here” beside a chart comparing AI maturity across industries in 2021 and 2024.](https://ai.engineer/api/talk-essay-media/bc279c3b2b868c26e11c0ca5943a0ab5b0f0430214cd0e60fa4b537683c7340d)

Democratization of AI, illustrated by industry maturity in 2021 and 2024.

### A compelling demo hides the distance to production

[3:24](https://www.youtube.com/watch?v=uiq95JYpBGY&t=204s)

**AI is easy to demo and hard to productionize.** An astonishing example can persuade a CEO to stop listening to customers and release something those customers find terrible. Biewald sees this in demonstrations from his own team and from startups pitching him. Software already has a gap between a promising prototype and a dependable product; AI makes that gap unusually easy to underestimate.

![Dark slide displaying “AI: Easy to Demo, Hard to Productionize,” with the speaker inset at lower left.](https://ai.engineer/api/talk-essay-media/4169bcafa7e63255967cf5f7428fb34a97de918b8f5fab6baf6b883291d5519c)

AI: Easy to Demo, Hard to Productionize.

Biewald founded Weights & Biases to address a tooling gap in AI operations. Its users now span three groups, with different starting points:

- **Foundation model builders** develop the underlying models and push the platform’s scale.
- **AI engineers** build applications combining traditional ML and GenAI.
- **Software developers** can now attempt applications that previously required building a custom model, but must learn how to move beyond a demo.

The last group is much larger than the first two. Its arrival expands both the number of possible AI applications and the need for tools that support the development process.

Biewald says W&B works with nearly all foundation model companies, naming **OpenAI GPT**, **Mistral**, and **Llama** as model families built using the platform. Yet most of its customers are application builders, not foundation model developers. Their work reaches healthcare, agricultural technology, and manufacturing—a broader set of tangible applications than the search ranking, finance, and ad targeting opportunities he remembers from the beginning of his career. That application experience is the basis for the production practices that follow.

### The reusable asset is what the experiments taught you

[6:21](https://www.youtube.com/watch?v=uiq95JYpBGY&t=381s)

Why do AI applications need different development tools? Adding an AI label to observability, CI/CD, or code versioning does not by itself answer that question. Biewald’s distinction concerns the work those tools must support: software development is generally additive, while LLM development is experimental. A new prompt or workflow is a hypothesis to test, not a feature whose addition guarantees progress. Outputs can vary, and a change can improve one case while breaking another. Broad behavioral evaluation therefore cannot be reduced to expecting every case to pass, even though narrow must-pass tests still have a role later in the workflow.

**The learning is the intellectual property.** A final model or prompt is an outcome of the development process. The unsuccessful prompts, rejected workflows, and reasons an approach failed tell the next engineer where to go—and where not to spend another week. If those discoveries live only in one person’s memory, the ability to continue iterating leaves with that person. Saving the final artifact alone does not preserve the work that produced it.

Reproducibility makes that learning usable, but recording enough information is painful, especially when experiments call nondeterministic endpoints. Biewald recommends passive capture by processes outside the engineer’s manual note-taking routine. People forget details even when they intend to document everything. Background tracking gives a colleague a way to reconstruct the experiment and pick up the investigation. Its practical return is shorter iteration time: preserved learning enables collaboration, and collaboration gets the application from demo to useful product faster.

### Turning a weather question into a function call

[9:22](https://www.youtube.com/watch?v=uiq95JYpBGY&t=562s)

Biewald encountered the production gap in a project he started during time off after his second child was born. His older daughter, Matilda, asked Alexa to play her favorite song and was surprised that it did not know which song she meant. She repeatedly requested Baby Shark; the preference seemed obvious to her father. He speculates that privacy considerations might explain the missing memory, but the immediate product request came from Matilda: could he build an Alexa?

He looked at Mycroft, which he found dated and not using LLMs at the time. Then a neighbor showed him a seven-billion-parameter model running on a ROCK Pi costing roughly $200; Biewald tentatively identifies the model as **Llama 2**. That suggested a feasible local assistant. He built a library of skills for music, weather, math, and news. The API wrappers were straightforward: the weather skill was the sort of short function Copilot could generate. The difficult part was recognizing speech and translating the resulting words into the correct API call.

For the input “What's the weather in Boston?”, the on-device LLM must produce a weather function call with Boston as the location. The essential Python contract can be expressed as:

```python
expected_call = 'weather(location="Boston")'

def matches_weather_call(generated_call: str) -> bool:
    return generated_call.strip() == expected_call
```

This illustrates a strict output contract, not the project’s unspecified scoring implementation. A sentence explaining that the weather function should be called is not the same thing as producing the call. The downstream weather skill needs the function name and argument in the format it accepts before it can query its API.

The local stack combines **[Whisper](https://github.com/openai/whisper)** for speech recognition with Llama for interpreting the request. Biewald chose [`llama.cpp`](https://github.com/ggml-org/llama.cpp) to run the language model on inexpensive hardware; he also mentions vLLM and hosted providers as alternatives. Running the system himself was part of the appeal, and he reports that large LLMs could run in real time on a MacBook.

![“How does it work” slide showing Whisper, a Boston weather question, an LLM block, a weather function call, and a Weather Skill block with an audio icon.](https://ai.engineer/api/talk-essay-media/c57d24a48f107e3d874b47c04aea82324d3b8bed3e5650106df56ca0d66b8e12)

A weather question passes through Whisper, an LLM, and a weather skill.

### Fast enough to answer, exact enough to run

[11:50](https://www.youtube.com/watch?v=uiq95JYpBGY&t=710s)

Biewald skips the live demonstrations because he is unsure the audio will work, then turns to a constraint that becomes obvious when using the assistant: **latency is part of correctness from the user’s perspective**. Speech must first be transcribed, and the transcription then passes through an LLM to generate the function call. He describes delays beyond a couple hundred milliseconds as painful in this interaction; this is a responsiveness judgment, not a measured latency result. It motivates choosing a small model and running it quickly. At the time, that led him to Llama 2.

Biewald reports 0% function-call accuracy for Llama 2 with the default prompt. The accuracy figures throughout this project are his reported results; the recording does not specify evaluation-set size, train/test separation, scoring rules, or whether speech-recognition errors were included. The first intervention was ordinary prompt engineering: describe the available functions more clearly. That made the answer look closer to the desired call, but it still included a leading instruction to call the weather function rather than cleanly producing runnable output. Semantic proximity was not enough for the interface.

Switching to **Llama 2-Chat**, which was trained for conversation, raised reported function-call accuracy to 11%. Inspecting its errors and incorporating feedback raised reported accuracy to 75%. This was enough for a pleasing demo when Matilda phrased her request just right, but it remained an annoying assistant to use. The distinction matters: a carefully chosen successful interaction can hide the frequency of failures across ordinary requests.

### A new model, then better training examples

[13:35](https://www.youtube.com/watch?v=uiq95JYpBGY&t=815s)

Mistral arrived during the project. Replacing Llama 2 with Mistral raised Biewald’s reported function-call accuracy from 75% to 79% without fine-tuning. In his setup, the replacement used the same API, so he could take advantage of the new model without rebuilding the application. This is a useful reason to keep model replacement easy: a new release can improve the existing task at little implementation cost. But the remaining failures were still too frequent to satisfy his daughter.

The next step was fine-tuning. LoRA and [QLoRA](https://arxiv.org/abs/2305.14314) had made this more tractable, with hosted services also available for teams that did not want to run training themselves. Biewald ran QLoRA in a few hours on a GPU he remembers, uncertainly, as a 4080. The remaining prerequisite was a dataset.

His data-building procedure was small enough to do personally:

1. Write a few examples of requests paired with the desired skill calls.
2. Ask ChatGPT to generate additional examples.
3. Review the generated examples by hand and remove unsuitable ones.
4. Specify an output schema and supply more examples to elicit a wider variety of cases.

Biewald estimates that about 95% of the generated examples were good and that he obtained a couple thousand fine-tuning examples in roughly 15 minutes. The larger model supplied training material for the smaller local model; manual review remained part of that process.

Fine-tuning Mistral raised Biewald’s reported function-call accuracy to 98%. At that point, the assistant began to feel potentially shippable to him. The change was not merely a more impressive isolated answer: it was a substantial improvement in how often the system produced the required function call.

Multilingual behavior was a welcome byproduct. Biewald tested Japanese and tentatively identifies the displayed example as French. Whisper could transcribe other languages, and the LLM could interpret those requests. For anyone following the repository setup, multilingual use requires a multilingual Whisper checkpoint: Otto’s README defaults to the English-only `base.en` checkpoint. The broader attraction of the project was that useful, interesting AI systems were again within reach of an individual hobbyist, even after the growth of model sizes had made that seem doubtful.

### Failed experiments and the problem of version two

[15:56](https://www.youtube.com/watch?v=uiq95JYpBGY&t=956s)

The project is available as [Otto](https://github.com/lukas/otto). Its progression shows why prompting and fine-tuning are not opposing strategies: prompt engineering helped from a poor starting point, and fine-tuning supplied later gains. Biewald extends that observation to production applications more generally, where retrieval-augmented generation, prompting, and fine-tuning may all contribute. He does not describe adding RAG to this assistant; the point is that the appropriate techniques can be complementary.

The clean sequence of improvements leaves out much of the work. The project took a couple of weeks, and most experiments failed. A colleague who inherited only the final model and this success story could easily repeat those failures before catching up. Biewald says he recorded the experiments—including unsuccessful ones—in W&B and made them available for inspection. That history is the concrete version of preserving learning rather than just preserving the winning prompt.

The first broader production practice is to build an evaluation framework and take it seriously. Biewald recalls asking an unnamed CEO on Gradient Dissent how his company tested GenAI applications. The CEO said he tested by vibes and declined Biewald’s offer to remove the remark. There is some value in that approach: personally trying a product can reveal glaring failures that should have prevented a release.

The limit appears when it is time to release a second version. A model or prompt change can make some requests better and others worse. A subjective impression will not reliably distinguish the relatively small improvement Otto gained from its model swap, so it cannot confidently justify the replacement. Evaluation makes those tradeoffs visible. When Biewald checked back with the CEO a year later, the company was doing substantial evaluation work after confronting this problem. **A framework for comparing versions is what turns experimentation into a release decision.**

### Build the whole loop, then measure what matters

[19:05](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1145s)

Evaluation belongs alongside three other practices: start with a lightweight prototype, incorporate end-user feedback, and iterate. Biewald sees an enterprise failure mode in which teams try to perfect the first stage before building the second, indefinitely postponing the moment anyone can use the application. A small end-to-end prototype should reach its intended audience, whether internal employees or external customers. Their feedback then gives the next experiment a purpose.

When Biewald asks how many attendees feel confident in their evaluations, participation is much lower than in the deployment poll. Teams that do mature their evaluations tend to use several sets and techniques rather than one all-purpose score:

| Evaluation layer | Intended role |
| --- | --- |
| Restricted must-never-fail set | Target 100% on simple, critical cases |
| Small, quick suite | Return feedback in 20–30 seconds |
| Larger nightly suite | Cover more cases outside the immediate iteration loop |

The restricted set explains where an all-pass requirement can make sense. It protects a narrow contract; it does not imply that the application’s full behavioral evaluation should be perfect. The quick suite supports interactive experimentation, while the nightly suite supplies broader feedback.

**Metrics must correspond to user experience or customer value.** That relationship is application-dependent and difficult to establish. W&B initially expected teams to track perhaps 10–20 metrics per model or application. Biewald reports thousands or tens of thousands of metrics in production self-driving and mission-critical banking applications. Some customers even request regex search because they cannot find the metric they need among so many. The point is not to maximize the number of metrics; it is that a consequential application can fail in many distinct ways, and a single aggregate score may hide the failures users care about.

Actually running evaluations is what makes the other decisions tractable. Should this application use RAG? Would fine-tuning help? Is another technique worth adding? With a useful evaluation system, the team can try the change and answer for its own workload instead of treating the choice as a general debate. The evaluation loop connects a proposed intervention to evidence about whether it improves the product.

![“Evaluation Best Practices” slide listing multiple evaluation sets or techniques, metrics correlated with user experience, and “Actually do it.”](https://ai.engineer/api/talk-essay-media/00b04106b1ff02b83f5f23a7c7ea967479c1075b1da86c7409dce4107ce57882)

Evaluation best practices: use multiple evaluation sets, align metrics with user experience, and actually evaluate.

Biewald closes by inviting attendees to see Weave, with the qualification that it can help with some production problems, not all of them. The [current Weave evaluation guide](https://docs.wandb.ai/weave/guides/core-types/evaluations) is a follow-up resource rather than an API demonstrated in this recording. Tools can support the loop, but the application’s users and failure modes still determine what it needs to measure.

## Resources

- [Otto voice assistant](https://github.com/lukas/otto) — Source and setup instructions for a local voice assistant using llama.cpp and whisper.cpp.
- [Otto fine-tuning workflow](https://github.com/lukas/otto/tree/main/finetune) — Instructions for generating skill-call examples, fine-tuning with QLoRA, and logging evaluations in W&B.
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314) — Original paper on memory-efficient fine-tuning using a quantized base model and low-rank adapters.
- [llama.cpp](https://github.com/ggml-org/llama.cpp) — C/C++ inference engine with quantization and support for local CPU and GPU execution.
- [Whisper](https://github.com/openai/whisper) — Speech recognition models with English-only and multilingual checkpoints, installation instructions and examples.
- [Build evaluations with Weave](https://docs.wandb.ai/weave/guides/core-types/evaluations) — Current guide to evaluating application outputs with curated datasets, custom scorers and repeated trials.

## Slides

### Presenting Sponsor

Microsoft is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=uiq95JYpBGY&t=5s) [Microsoft presenting sponsor slate.](https://assets.ai.engineer/sites/aiecode2025/sha256/b454bc46ecdb19263637c5a4050f98d54d326639bdc28e447e35026e624cee73.webp)

PRESENTING SPONSOR

Microsoft

### Innovation and Platinum Sponsors

AWS is the innovation sponsor; MongoDB, Google Cloud, and Neo4j are platinum sponsors.

[0:10](https://www.youtube.com/watch?v=uiq95JYpBGY&t=10s) [AWS is the innovation sponsor; MongoDB, Google Cloud, and Neo4j are platinum sponsors.](https://assets.ai.engineer/sites/aiecode2025/sha256/977ee36701a751383c48d7ef8004c16a4798b79a10a81629bd0947921e28b07f.webp)

INNOVATION SPONSOR
aws

PLATINUM SPONSORS
MongoDB®
Google Cloud
neo4j

### Democratization of AI Is Here

Accenture’s chart shows higher median AI maturity in every listed industry in 2024 than in 2021. The speaker credits chat interfaces and LLMs with widening AI’s reach and predicts every software engineering team will add AI capabilities.

[1:56](https://www.youtube.com/watch?v=uiq95JYpBGY&t=116s) [A prediction that software engineering teams will add AI capabilities, alongside a chart comparing AI maturity by industry in 2021 and 2024.](https://assets.ai.engineer/sites/aiecode2025/sha256/68f042a485686e6e3a9ee28109f7e43bfbcad46e30630a4d3845e8009d39195f.webp)

Prediction

Democratization of AI is here.

Every software engineering team will add AI capabilities

Levels of AI maturity by industry, 2021 and 2024
Accenture

Legend: 2021, 2024
The median AI Maturity Index in 2021 and 2024 by industry

Industry:
Tech
Automotive
Aerospace & Defence
Life Sciences
Natural Resources
Retail
Utilities
Industrial
Average
Energy
Communications & Media
Public Services
Chemicals
Insurance
Travel
Consumer Goods & Services
Healthcare
Banking & Capital Markets

Median AI Maturity (0-100)
X-axis ticks: 30, 40, 50, 60
Arthmetric average of Foundation index and Differentiation index

2

Interface text: docs.google.com

[2:50](https://www.youtube.com/watch?v=uiq95JYpBGY&t=170s) [Prediction that software engineering teams will add AI capabilities, alongside a chart comparing AI maturity by industry in 2021 and 2024.](https://assets.ai.engineer/sites/aiecode2025/sha256/cc87e74802d03c76107245e9d674cca3fc8977f295efa2d4984096d3ae73becc.webp)

Prediction

Democratization of AI is here.

Every software engineering team will add AI capabilities

Levels of AI maturity by industry, 2021 and 2024
Accenture

Legend: 2021, 2024
The median AI Maturity Index in 2021 and 2024 by industry

Industry:
Tech
Automotive
Aerospace & Defence
Life Sciences
Natural Resources
Retail
Utilities
Industrial
Average
Energy
Communications & Media
Public Services
Chemicals
Insurance
Travel
Consumer Goods & Services
Healthcare
Banking & Capital Markets

Median AI Maturity (0-100)
X-axis ticks: 30, 40, 50, 60
Arthmetric average of Foundation index and Differentiation index

2

Broadcast text: AIE; World's Fair; Microsoft; smol ai; aws

Interface text: docs.google.com

### AI: Easy to Demo, Hard to Productionize

Compelling AI demos can be much further from production than teams expect. The speaker warns that their appeal can lead CEOs to stop listening to customers and ship poor applications.

[3:22](https://www.youtube.com/watch?v=uiq95JYpBGY&t=202s) [A dark slide states that AI is easy to demo and hard to productionize.](https://assets.ai.engineer/sites/aiecode2025/sha256/dbcac015c454cef612dd8dac90c25177de9ca4306857a1e3da1455ff9ae8e948.webp)

AI: Easy to Demo,
Hard to Productionize

4

AIE
AI Engineer World’s Fair
Microsoft
smol ai
aws

Interface text:
docs.google.com

### Weights & Biases – The AI Developer Platform

Weights & Biases presents two workflows: Models for building and fine-tuning ML models, and Weave for developing GenAI applications. Both sit on a shared foundation of artifacts, tables, and reports.

[4:07](https://www.youtube.com/watch?v=uiq95JYpBGY&t=247s) [Diagram of model development and Weave workflows, supported by artifacts, tables and reports.](https://assets.ai.engineer/sites/aiecode2025/sha256/1ad171420cf6bfe7554c7956398ca8f3c8394c2c560fb07de012de507901ac9e.webp)

Weights & Biases – The AI Developer Platform

Models
Build & Fine-tune models

Experiments
Track and visualize your ML experiments

Sweeps
Optimize your hyperparameters

Model Registry
Register and manage your ML models

Automations
Trigger workflows automatically

Launch
Package and run your ML workflow jobs

Weave
Develop GenAI applications

Traces
Monitor and debug LLMs and prompts

Evaluations
Rigorous evaluations of GenAI applications

W&B Core: A foundational framework supporting AI developers

Artifacts
Version and manage your ML pipelines

Tables
Visualize and explore your ML data

Reports
Document and share your ML insights

Snowflake [illegible]
5

Interface text:
docs.google.com
AIE
AI Engineer World's Fair
Microsoft
smol.ai
aws

[4:11](https://www.youtube.com/watch?v=uiq95JYpBGY&t=251s) [Diagram of Models and Weave workflows above shared artifacts, tables, and reports.](https://assets.ai.engineer/sites/aiecode2025/sha256/856c48c769583f21e65ded22f015008c104512b1b814b5286fe250161560db6e.webp)

Weights & Biases – The AI Developer Platform

Models
Build & Fine-tune models

Experiments
Track and visualize your ML experiments

Sweeps
Optimize your hyperparameters

Model Registry
Register and manage your ML models

Automations
Trigger workflows automatically

Launch
Package and run your ML workflow jobs

Weave
Develop GenAI applications

Traces
Monitor and debug LLMs and prompts

Evaluations
Rigorous evaluations of GenAI applications

W&B Core: A foundational framework supporting AI developers

Artifacts
Version and manage your ML pipelines

Tables
Visualize and explore your ML data

Reports
Document and share your ML insights

Snowflake [illegible]
5

Interface text:
docs.google.com
AIE
AI Engineer World's Fair
Microsoft
smol.ai
aws

### Three developer audiences

Weights & Biases serves foundation model builders and AI engineers, and is reaching software developers working to productionize AI applications. The slide illustrates each audience with portraits.

[4:31](https://www.youtube.com/watch?v=uiq95JYpBGY&t=271s) [Three columns show foundation model builders, AI engineers, and software developers with example portraits.](https://assets.ai.engineer/sites/aiecode2025/sha256/0b1489f23e210e583e706ee88413c933a51ce298f450f81e75c01a7c1d9397c8.webp)

Foundation Model Builder
Lilian Weng
OpenAI

AI Engineer
Dr Aengus Tran
Harrison
Ramitha Sundar
Blue River
Audrey Beard
Disney
Vitor Guizilini
TRI

Software Developer

AIE
AI Engineer World’s Fair
Microsoft
smol ai
aws

Interface text:
docs.google.com

### Developer audience sizes

The slide places software developers in the 10,000,000s, compared with 100,000s of ML engineers and 1000s of foundation model builders. The speaker sees this larger audience expanding AI applications.

[4:59](https://www.youtube.com/watch?v=uiq95JYpBGY&t=299s) [Three circles represent thousands of foundation model builders, hundreds of thousands of ML engineers, and tens of millions of software developers.](https://assets.ai.engineer/sites/aiecode2025/sha256/afbcc7b9d26d37b35be8c8f7972c272048a996b9395f8a530ee4f5b2beea4867.webp)

1000s
Foundation Model Builder

100,000s
ML Engineer

10,000,000s
Software Developer

AIE
World’s Fair
Microsoft
smol ai
aws

Interface text:
docs.google.com

### Foundation model companies

Weights & Biases works with nearly all foundation model companies, the speaker says. Their special needs push the platform’s scale; he cites GPT, Mistral and Llama as models built using it.

[5:17](https://www.youtube.com/watch?v=uiq95JYpBGY&t=317s) [A slide claims the company works with nearly all foundation model companies and displays a grid of logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/29da0a38699ce5d383b4d427698669b006bdc6503f04c1715afd237f64579801.webp)

We work with nearly all the foundation model companies

OpenAI
LLAMA 2 — Meta AI
co:here
MISTRAL AI_
Alphabet
Midjourney
character.ai
amazon
NVIDIA NEMO
runway
salesforce
IBM
Inflection
Ai2
Ideogram
Hippocratic AI — Do No Harm
contextual.ai
mosaicML
ALEPH ALPHA
stability.ai
TII — Technology Innovation Institute
NTT
scale
Liquid
CyberAgent.
Adept
together.ai
Hugging Face
imbue
DeepMind
SoftBank
poolside
sarvam.ai
AI21labs
mistral
01.AI
Preferred Networks
NEC
KRUTRIM — India’s own AI
Canva
deci. — Break the AI Barrier

9

Interface text:
docs.google.com
AIE
AI Engineer World’s Fair
Microsoft
smol ai
aws

[5:27](https://www.youtube.com/watch?v=uiq95JYpBGY&t=327s) [Slide showing a grid of foundation model company logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/d3392ffeb7a5ba9f67c938564e7b10b5d45ea495a3570fef83d1d44ff8384ae2.webp)

We work with nearly all the foundation model companies

OpenAI
LLAMA 2 — Meta AI
co:here
MISTRAL AI_
Alphabet
Midjourney
character.ai
amazon
NVIDIA NEMO
runway
salesforce
IBM
Inflection
Ai2
Ideogram
Hippocratic AI — Do No Harm
contextual.ai
mosaicML
ALEPH ALPHA
stability.ai
TII — Technology Innovation Institute
NTT
scale
Liquid
CyberAgent.
Adept
together.ai
Hugging Face
imbue
DeepMind
SoftBank
poolside
sarvam.ai
AI21labs
mistral
01.AI
Preferred Networks
NEC
KRUTRIM — India’s own AI
Canva
deci. — Break the AI Barrier

9

Interface text:
docs.google.com
AIE
AI Engineer World’s Fair
Microsoft
smol ai
aws

### Customers across industries

The speaker says almost all of Weights & Biases’ thousands of customers are not foundation model builders. The slide groups customer examples across industries, from healthcare and finance to media and robotics.

[5:46](https://www.youtube.com/watch?v=uiq95JYpBGY&t=346s) [Customer logos grouped by industry, including computing, healthcare, media and autonomous vehicles.](https://assets.ai.engineer/sites/aiecode2025/sha256/57c3cca3ebf65a94d723e2b3d6327fe9585259e971370eee260abc7a2204162c.webp)

Customers
Our customers span real world use cases in every industry

Innovators / Influencers / AI Companies
OpenAI
Meta
Midjourney
mosaicML
cohere
stability.ai
AppLovin
Canva

High performance computing
SAMSUNG
GRAPHCORE
NVIDIA

Gaming
magic leap
EPIC GAMES

Healthcare / Life sciences
Johnson & Johnson
absci
ORACLE
Genentech

Autonomous vehicle / Robots
GM
TOYOTA RESEARCH INSTITUTE
Volkswagen
Ford
Lyft
BMW
RIVIAN
nuro
ZOOX
AImotive
JOHN DEERE
WAYVE
iRobot
covariant

Financial Services
Fidelity
RBC
Square

Media
Spotify
sky
pandora
SONY

Retail / Consumer
nanit
zalando
PRECISIONHAWK

[illegible] / Software
[illegible]rosoft
salesforce
GONG
IBM
UiPath
qualtrics XM

10

Interface text:
docs.google.com
AIE
AI Engineer World's Fair
Microsoft
smol ai
aws

### Software Development: Linear

The pipeline shows successive workflows running or succeeding. The speaker describes software development as a linear process where adding features and code generally improves things over time, while AI development is experimental.

[6:19](https://www.youtube.com/watch?v=uiq95JYpBGY&t=379s) [Slide showing a software pipeline with running and successful workflows and a magnified detail.](https://assets.ai.engineer/sites/aiecode2025/sha256/6a625e57136fc4787238a0feea212cefc8b5a592925c7118a32e7c000f4b860b.webp)

Software Development: Linear

Pipeline  Status  Workflow  Trigger Event  Start  Duration  Actions
[illegible]

Magnified workflow detail:
Running
23m 17s remain
build_accept_deploy
Success
setup-workflow  SETUP
Success
build_accept_deploy
Success
setup-workflow  SETUP
Success
build_accept_deploy
Success
setup-workflow  SETUP
Success
build_accept_deploy
Success
setup-workflow

11

Interface text:
docs.google.com
AIE
AI Engineer World's Fair
Microsoft
smol ai
aws

### Experiment parameter importance and accuracy

The dashboard compares parameter importance and accuracy across runs: Runtime leads for acc, while num_train leads for car_acc. The speaker describes AI development as a series of experiments.

[7:05](https://www.youtube.com/watch?v=uiq95JYpBGY&t=425s) [Dashboard comparing parameter importance and correlations above accuracy charts for multiple runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/44a316095a4a2c7ec9d94b8bbc1f56c464ac875577cf3458881e02a0aa637618.webp)

Parameter importance with respect to acc
Config parameter | Importance | Correlation
Runtime
num_train
num_valid
learning_rate
model_type.value_r...
model_type.value_...
batch_size

Parameter importance with respect to car_acc
Config parameter | Importance | Correlation
num_train
num_valid
batch_size
model_type.value_...
weight_decay
learning_rate
model_type.value_r...

road_acc, car_acc
Legend: second-best-run road_acc; snowy-totem-184 road_acc; second-best-run car_acc; best 20% data car_acc; efficient-shape-156 car_acc; solar-sweep-2 car_acc; wise-sun-171 car_acc; snowy-totem-184 car_acc
Y-axis ticks: 0.9, 0.8, 0.7

human_acc, iou, car_acc
Legend: [illegible] human_acc; [illegible] human_acc; [illegible] human_acc; [illegible] human_acc; [illegible] human_acc; [illegible] human_acc; [illegible] iou; [illegible] car_acc; [illegible] car_acc; [illegible] car_acc; [illegible] car_acc; [illegible] car_acc; [illegible] car_acc
Y-axis ticks: 0.95, 0.9, 0.85, 0.8, 0.75, 0.7, 0.65

Interface text:
Charts 10
Add panel
Search
Parameters
1-10 of 15
Search
Parameters
1-10 of 14

### Weights & Biases GitHub repository

Against the wandb code repository, the speaker contrasts software’s code as IP with AI’s learning from experiments; models and discovered prompts are outcomes.

[7:29](https://www.youtube.com/watch?v=uiq95JYpBGY&t=449s) [The Weights & Biases repository page lists project files and describes its machine learning tools.](https://assets.ai.engineer/sites/aiecode2025/sha256/8ce9309f9e1cf281ee8654e2224ecef496b1905497c33b99d7477269303f4d8d.webp)

wandb
Public

main
1,000 Branches
147 Tags
Go to file
Add file
Code

moredatarequired  chore(artifacts): use a retrying client to download manifest...  3 days ago
6,337 Commits

.circleci  chore(sdk): adding back the required flag for wandb-core ...  last week
.github  chore(sdk): Use ruff as the formatter (#7338)  last week
.vscode  test(core): add tests for watcher (#6763)  3 months ago
apple_stats  chore(sdk): replace the gpu apple binary with the new on...  last week
client  chore(sdk): Ignore wandb_internal.rs (#7192)  last month
core  chore(artifacts): use a retrying client to download manifes...  3 days ago
docs  feat(nexus): add sync file counts in the footer (#6371)  6 months ago
tests  feat(artifacts): allow any os.PathLike as the download root...  3 days ago
tools  chore(sdk): Use ruff as the formatter (#7338)  last week
wandb  feat(artifacts): allow any os.PathLike as the download root...  3 days ago
.bumpversion.cargo.cfg  chore(sdk): Bump core version to 0.17.0b9 (#6991)  2 months ago
.bumpversion.core.cfg  chore(sdk): bump wandb-core version to [illegible]  4 days ago
.bumpversion.wandb.cfg  chore(sdk): bump version to 0.16.7.dev1 (#7296)  2 weeks ago
.clang-format  chore(sdk): format protobuf files (#6517)  6 months ago
.codecov.yml  chore(core): finish renaming nexus dir to core (#6687)  4 months ago
.git-blame-ignore-revs  Add blame ignore (#3901)  2 years ago

About
🔥 A tool for visualizing and tracking your machine learning experiments. This repo contains the CLI and Python API.
wandb.ai

data-science  machine-learning
reinforcement-learning  deep-learning
tensorflow  keras  collaboration
pytorch  hyperparameter-optimization
reproducibility  hyperparameter-tuning
data-versioning  hyperparameter-search
model-versioning  jax  mlops
experiment-track  ml-platform

Readme
MIT license
Code of conduct
Security policy
Activity
Custom properties
8.2k stars
54 watching
603 forks
Report repository
Releases 133

Interface text:
wandb / wandb
Type / to search
Code  Issues 856  Pull requests 215  Actions  Projects 1  Wiki  Security 539  Insights
Edit Pins  Watch 54  Fork 603  Starred 8.2k

### Reproducibility Is Hard

Across every field shown, more scientists reported failing to reproduce someone else’s experiment than their own. The speaker says tracking everything needed for reproduction is especially hard with nondeterministic endpoints.

[8:16](https://www.youtube.com/watch?v=uiq95JYpBGY&t=496s) [A bar chart compares reported failures to reproduce experiments across scientific fields.](https://assets.ai.engineer/sites/aiecode2025/sha256/ab18f1814f5017e816c425db68188f7996c1399558c75ed038de6229a1d1b7bd.webp)

Reproducibility Is Hard

HAVE YOU FAILED TO REPRODUCE AN EXPERIMENT?
Most scientists have experienced failure to reproduce results.

Legend: Someone else’s; My own
Fields: Chemistry; Biology; Physics and engineering; Medicine; Earth and environment; Other
X-axis ticks: 0, 20, 40, 60, 80, 100%

16

Interface text:
docs.google.com

### Track Everything

The speaker argues for tracking experiment settings and metadata in the background: people forget to record details, and shared records let others build on an engineer’s learnings.

[8:32](https://www.youtube.com/watch?v=uiq95JYpBGY&t=512s) [A configuration panel displays model parameters beside magnified experiment metadata.](https://assets.ai.engineer/sites/aiecode2025/sha256/26696e10a95bb14fd3c7de68353ffceaab9ec511b0e28b3cc0c870787c370b73.webp)

Track Everything

Config
View raw data
Config parameters are your model's inputs. Learn more
Search keys with regex
Config parameters: {} 13 keys
batch_size: 8
bn_weight_decay: false
encoder: "resnet18"
epochs: 20
framework: "fast.ai"
img_size: [] 2 items
  0: 360
  1: 640
learning_rate: 0.001
num_train: 1,421
num_valid: 183
one_cycle: true

[illegible] examples
humans
stacey
Finished
January 16th, 2020 at 10:20:22 am
37m 54s
stacey/deep-drive/oqo4iodf
wbrave
Linux-5.0.0-37-generic-x86_64-with-debian-buster-sid
3.7.2
/home/stacey/.pyenv/versions/sd/bin/python
git clone https://github.com/borisdayma/semantic-segme[illegible]
git checkout -b "best-20%-data" [illegible]

17

Interface text:
docs.google.com
AIE
AI Engineer World's Fair
Microsoft
smol ai
aws

### Collaboration = Speed

The W&B team page brings reports and members together. The speaker argues that tracking an engineer’s work lets others build on those learnings and iterate faster.

[8:47](https://www.youtube.com/watch?v=uiq95JYpBGY&t=527s) [A collaboration slide showing a Weights & Biases team page.](https://assets.ai.engineer/sites/aiecode2025/sha256/24c3b722a234450fea500e1d77850896c56245806604092aee0908b1141338ea.webp)

Collaboration = Speed

wandb
Overview  Reports  Projects  Members  Settings

Intro
Hello, we're W&B
Our mission is to build the best tools for machine learning practitioners
We're interested in helping the world's ML engineers and researchers do their best work
Let's collaborate on reproducible, explainable, safe, and effective AI in production
You can reach us at wandb.com or follow us on Twitter
Follow @weights_biases

Reports
Select reports to showcase
How Many Discoveries Were Lost Because They Weren't...
Collaboration in ML made easy with W&B Teams
Exploring Bounding Boxes for Object Detection With Weight...
An Introduction to Weights & Biases Artifacts
Reproducible spaCy NLP Experiments with Weights &...
Hyperparameter Search With spaCy and Weights & Biases
[illegible]

Interface text:
docs.google.com
AIE
World's Fair
Microsoft
smol ai
aws
18

### Mycroft

The speaker explored Mycroft as an open-source Alexa alternative while considering building a voice assistant with LLMs. He describes Mycroft as dated and says it doesn't really use LLMs.

[10:20](https://www.youtube.com/watch?v=uiq95JYpBGY&t=620s) [A slide showing the Mycroft Core GitHub repository.](https://assets.ai.engineer/sites/aiecode2025/sha256/76f89cfe64825cb599ee3f9eff1b7fc3b062ae3fcffd9a8d63622e3805bf62c7.webp)

Mycroft

MycroftAI / mycroft-core
Code  Issues 164  Pull requests 38  Actions  Projects 1  Security  Insights

mycroft-core  Public
Watch 294  Fork 1.3k  Star 6.4k
dev  69 branches  134 tags
Go to file  Add file  Code

forslund  Handle multiple intents with the same name (#2921)
ab242a2 on Nov 22, 2022  4,921 commits
.github  Shellcheck scripts in root folder (#3090)  last year
bin  Silence file does not exist error when clearing VK files (#3093)  last year
doc  Merge pull request #2898 from MycroftAI/bugfix/docstring-cleanup  2 years ago
mycroft  Handle multiple intents with the same name (#2921)  last year
requirements  Relax pyyaml version requirement  last year
scripts  Shellcheck scripts folder (#3063)  last year
test  Handle multiple intents with the same name (#2921)  last year
.gitignore  Add .theia to .gitignore  3 years ago
.gitmessage  Add suggestion and examples of imperative text  6 years ago
.pep8speaks.yml  Set pep8-speaks line lenght limit to 79 (#1953)  5 years ago
.readthedocs.yml  Fix issues with .readthedocs file  3 years ago
.shellcheckrc  Disable source lookup errors  2 years ago

About
Mycroft Core, the Mycroft Artificial Intelligence platform.
mycroft.ai
python  nlp  linux  open-source
raspberry-pi  natural-language-processing
oss  ai  skills  arch  fedora
natural-language  artificial-intelligence
personal-assistant  hacktoberfest
voice-assistant  voice-as-an-interface
Readme
Apache-2.0 license
Code of conduct
Activity
6.4k stars
294 watching
1.3k forks
Report repository

Interface text:
docs.google.com
Type / to search

### Useful Sensors/RockPI

The slide shows an AI in a Box case concept. A neighbor’s roughly $200 ROCK Pi running a 7-billion model inspired the speaker to consider an Alexa-like architecture.

[10:35](https://www.youtube.com/watch?v=uiq95JYpBGY&t=635s) [A slide showing the AI in a Box product and its integrated display.](https://assets.ai.engineer/sites/aiecode2025/sha256/ec54f8caf6dbf452f5991cae9277d26542e5397c31063bce14eead63274ad91c.webp)

Useful Sensors/RockPI

PRODUCTS
See for yourself.

AI IN A BOX
PERSON SENSOR
TINY CODE READER

Will be delivered in a case with an integrated display, speaker, microphone and connectors

Hi! I am AI
in a Box!

Rendered for illustration purposes only, final design may vary

Interface text:
docs.google.com

### Extensible Library of Skills

The weather skill calls an API; the speaker describes using an on-device LLM to turn a request into a function call. The library also shows math, OpenAI, news, and an option to add skills.

[10:44](https://www.youtube.com/watch?v=uiq95JYpBGY&t=644s) [A weather function example above tiles for custom, weather, math, OpenAI, and news skills.](https://assets.ai.engineer/sites/aiecode2025/sha256/c32386406c60b3b338541b2051c260510decafb89d3215c52f992d8ad9b55331.webp)

Extensible Library of Skills

def get_weather(self, city_name):
    print("Getting weather for " + city_name)
    params = {"q": city_name,
              "appid": self.api_key, "units": "imperial"}
    base_url = "http://api.openweathermap.org/data/2.5/weather"

    response = requests.get(base_url, params=params)

Add your own
weather
math
openai
news

AIE
World's Fair
Microsoft
smol ai
aws

Interface text:
docs.google.com

### LLaMA and Whisper: The Open Source Stack

Whisper handles audio transcription, while LLaMA supplies the LLM. The slide lists their checkpoints and inference options and says both require GPUs, with LLaMA requiring one for serving.

[11:28](https://www.youtube.com/watch?v=uiq95JYpBGY&t=688s) [Slide compares Whisper transcription and LLaMA model checkpoints, inference code, and GPU requirements.](https://assets.ai.engineer/sites/aiecode2025/sha256/a0e3cb847a84d90b65249a12345cc4d52f012ab96f65a0b35bc90ec3d8339211.webp)

LLaMA and Whisper: The Open Source stack

Whisper is an open source model for audio transcription by OpenAI:
• Model checkpoints in PyTorch
• Inference code in PyTorch
• Requires GPU

LLaMA is an open source LLM by MetaAI:
• Model checkpoints in PyTorch and HF formats
• Inference code in PyTorch or dedicated server engines (TGI, vLLM, etc)
• Requires GPU to serve

Interface text:
docs.google.com
AIE
World’s Fair
Microsoft
smol ai
aws

### Llama.cpp: Run LLaMA on a MacBook

Georgi Gerganov’s C++ ports make Whisper and LLaMA runnable on a laptop. The slide highlights a built-in server, quantization and speculative decoding; the speaker says llama.cpp can run large LLMs on a MacBook in real time.

[11:33](https://www.youtube.com/watch?v=uiq95JYpBGY&t=693s) [Slide describes running Whisper and LLaMA locally with C++ ports, quantization, and a built-in server.](https://assets.ai.engineer/sites/aiecode2025/sha256/1133212c9e8253dae5c96c9f512888a56e31f16a3efe001662bb1cd880e899eb.webp)

Llama.cpp: Run LLaMA on a MacBook

Georgi Gerganov ported Whisper and LLama to C++, with no dependencies!

• Now you can run it on your laptop
• It has a built-in server
• Quantization and other useful techniques like speculative decoding
• It’s fast!

whisper C++
LLaMA C++

AIE
Microsoft
smol ai
aws
World’s Fair

Interface text:
docs.google.com

### Latency Matters a Lot

Whisper transcribes a spoken weather request; an LLM generates a call to the weather skill, and the flow ends in audio. The speaker says delays beyond a couple hundred milliseconds make the interaction painful.

[11:54](https://www.youtube.com/watch?v=uiq95JYpBGY&t=714s) [Diagram shows speech transcription feeding an LLM, which generates a weather function call for a weather skill.](https://assets.ai.engineer/sites/aiecode2025/sha256/da5b598ab8fbbd633a42cce26f1e769ac28ffc6ed27afcf13ad33100d608787d.webp)

Latency Matters a Lot

Whisper
“What’s the weather in boston?”
LLM
weather(location="Boston")
Weather Skill

Interface text:
docs.google.com
AIE
World’s Fair
Microsoft
smol ai
aws

### Llama Doesn't Work With the Default Prompt

With the default prompt, Llama 2 answered the Boston weather question in prose instead of making a function call. The speaker reported 0% accuracy.

[12:19](https://www.youtube.com/watch?v=uiq95JYpBGY&t=739s) [A weather question receives a prose answer instead of a function call; the slide reports zero percent accuracy.](https://assets.ai.engineer/sites/aiecode2025/sha256/8461523205d8bb689fce1e079a57eb4120fa7d3c7c181f9f7dfd5e8ea821adf4.webp)

Llama doesn’t work with the default prompt

USER
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### User: What’s the weather in boston?
### Answer:

AI
I’m happy to help you with that! Unfortunately, I’m a large language model, I don’t have access to real-time weather information or current conditions in specific locations. However, I can suggest some ways for you to find out the current weather in Boston.
1. Check online weather websites: There are many websites that provide up-to-date weather information, such as AccuWeather, Weather.com, or the National Weather Service (NWS). You can enter "Boston" in the search bar and get the current weather conditions and forecast.
2. Use a mobile app: There are many mobile apps available that provide real-time weather information, such as Dark Sky (iOS, Android) or Weather Underground (Android, iOS). These apps often have features like radar and satellite imagery to help you track the weather.
3. Tune into local news: You can watch local news broadcasts or check their website for weather updates. Local news stations often provide real-time coverage of weather events in their area. Remember, it’s important to stay informed about the weather and take necessary precautions to ensure your safety.

0% acc

AIE
Microsoft
smol ai
aws
World’s Fair

Interface text:
docs.google.com

## Full transcript

[0:00](https://www.youtube.com/watch?v=uiq95JYpBGY&t=0s) [upbeat music] I guess as we're getting started, this talk is about productionizing AI.

[0:18](https://www.youtube.com/watch?v=uiq95JYpBGY&t=18s) But I'm curious, um, I keep asking this question and it keeps changing. How many of you folks have, um, LLM applications in production in your company?

[0:28](https://www.youtube.com/watch?v=uiq95JYpBGY&t=28s) Wow. That's probably the highest I've ever seen, more than seventy percent.

[0:34](https://www.youtube.com/watch?v=uiq95JYpBGY&t=34s) So why did you come to this talk? You know, [laughs] you figured it out, right?

[0:38](https://www.youtube.com/watch?v=uiq95JYpBGY&t=38s) It's not going great.

[0:39](https://www.youtube.com/watch?v=uiq95JYpBGY&t=39s) It's not going great? [laughs]

[0:42](https://www.youtube.com/watch?v=uiq95JYpBGY&t=42s) Yeah.

[0:46](https://www.youtube.com/watch?v=uiq95JYpBGY&t=46s) All right. You wanna put it in slideshow mode?

[0:51](https://www.youtube.com/watch?v=uiq95JYpBGY&t=51s) It is in slideshow mode.

[0:51](https://www.youtube.com/watch?v=uiq95JYpBGY&t=51s) Oh, it is in slideshow mode? Okay. Hey, you're showing my IP to the whole, uh, audience. [laughs]

[1:02](https://www.youtube.com/watch?v=uiq95JYpBGY&t=62s) Um...

[1:05](https://www.youtube.com/watch?v=uiq95JYpBGY&t=65s) Yeah. You might ask if they're custom solutions or, or purchased, uh, LLM solutions.

[1:09](https://www.youtube.com/watch?v=uiq95JYpBGY&t=69s) Oh, good question. Yeah. How many, uh, is, uh, custom solutions?

[1:15](https://www.youtube.com/watch?v=uiq95JYpBGY&t=75s) Interesting. That's maybe thirty percent. And how many purchased solutions?

[1:20](https://www.youtube.com/watch?v=uiq95JYpBGY&t=80s) Wow. Way, way less. [laughs] What are you, what are you purchasing?

[1:24](https://www.youtube.com/watch?v=uiq95JYpBGY&t=84s) Uh, well, I put like Copilot, uh, for from GitHub in that category of using, uh-

[1:28](https://www.youtube.com/watch?v=uiq95JYpBGY&t=88s) Oh, okay. Well, if you put Copilot in that category, how many have purchased solutions?

[1:33](https://www.youtube.com/watch?v=uiq95JYpBGY&t=93s) It's gotta be most, right? Interesting. Interesting. Thanks for the... [laughs] Any more... Should we crowdsource some more questions? No, we can begin. Awesome. Um, all right. So I wanted to talk, um, briefly about our experience with our customers productionizing GenAI models.

[1:51](https://www.youtube.com/watch?v=uiq95JYpBGY&t=111s) But I want to give you a little bit of context, first of all. Um, this is clearly preaching to the choir, but it's interesting to see that democratization of AI is here.

[2:00](https://www.youtube.com/watch?v=uiq95JYpBGY&t=120s) I don't know, how many people were working in AI ten years ago?

[2:04](https://www.youtube.com/watch?v=uiq95JYpBGY&t=124s) Nice. Fifteen years ago?

[2:06](https://www.youtube.com/watch?v=uiq95JYpBGY&t=126s) ML counts.

[2:06](https://www.youtube.com/watch?v=uiq95JYpBGY&t=126s) So yeah, ML totally counts. Hundred percent ML counts. Um, ML definitely counts as a type of AI, absolutely. Um, you know, I think people talked about the democratization of AI, you know, back when AI was kind of a lame, um, term to use.

[2:20](https://www.youtube.com/watch?v=uiq95JYpBGY&t=140s) But, uh, I think we thought that it might take the form of, you know, something like AutoML or more, you know, graphical user interfaces. But clearly democratization of AI has happened in a way that we didn't expect, clearly through these, these kind of chat and conversational interfaces and LLMs.

[2:34](https://www.youtube.com/watch?v=uiq95JYpBGY&t=154s) We now see AI in literally every company that we talk to in some way. And not just purchasing solutions, although there's, I think, probably every company in this room purchases what I would call an AI solution.

[2:45](https://www.youtube.com/watch?v=uiq95JYpBGY&t=165s) We also see quite a lot of custom solutions, and probably every Fortune 500 right now is investing in custom AI solutions. More LLMs, more GenAI, but I would say most also some traditional ML.

[2:59](https://www.youtube.com/watch?v=uiq95JYpBGY&t=179s) And it's moving into production, right? You see it in this audience, you see seventy percent of this audience. But everywhere I go to give a talk, even with companies that you wouldn't necessarily think of as the most forward-thinking, they are starting to have GenAI applications in production.

[3:11](https://www.youtube.com/watch?v=uiq95JYpBGY&t=191s) So people keep asking me, "When is GenAI gonna move into production?" It really has moved into production, right? There's a lot more that could happen, but GenAI is in production right now in a lot of applications that you use today.

[3:24](https://www.youtube.com/watch?v=uiq95JYpBGY&t=204s) But I think the challenge everybody has, and people keep saying it, important to say, AI is so easy to demo and so hard to productionize. You see this with a lot of AI apps getting released.

[3:33](https://www.youtube.com/watch?v=uiq95JYpBGY&t=213s) There's something about AI that makes CEOs stupid. It makes them stop listening to their customers. It makes them willing to put things into production that are absolutely terrible. And I think the fundamental reason is that it's so easy to make an incredibly compelling demo.

[3:49](https://www.youtube.com/watch?v=uiq95JYpBGY&t=229s) I get these all the time from my own team. I get them from startups, you know, pitching me on potentially angel investing. You can make just an absolutely astonishing demo, and it's so much further from production than we're used to.

[4:00](https://www.youtube.com/watch?v=uiq95JYpBGY&t=240s) Even with software, which has this characteristic, AI is so much more easy to demo and so much harder to productionize. So let's talk about how to productionize it. I'll, I'll state I come from Weights & Biases.

[4:12](https://www.youtube.com/watch?v=uiq95JYpBGY&t=252s) This is a company I started that I feel very proud of. We've built an AI developer platform that helps develop AI engineers with ML, uh, applications or with GenAI applications.

[4:24](https://www.youtube.com/watch?v=uiq95JYpBGY&t=264s) And we started this company because we saw this massive tools gap for AI ops.

[4:31](https://www.youtube.com/watch?v=uiq95JYpBGY&t=271s) We now see that we have three types of customers. We see foundation model builders, which we've long served since the beginning. We see a bigger group of AI engineers that are doing a blend of ML applications and GenAI applications.

[4:44](https://www.youtube.com/watch?v=uiq95JYpBGY&t=284s) And we see a new type of audience for our software in software developers that are now getting used to the fact that AI applications are so easy to demo and so hard to productionize, and trying to figure out how to productionize those applications.

[4:58](https://www.youtube.com/watch?v=uiq95JYpBGY&t=298s) What's exciting for us, and exciting for everyone, I think, is that the software developers are so much more numerous than ML engineers and foundation model builders, right? Now that companies can use software developers to do what traditionally would've taken a custom model, it opens us up to do so many more AI applications. [clears throat]

[5:18](https://www.youtube.com/watch?v=uiq95JYpBGY&t=318s) And from our perspective, this is a really exciting development because the market's so much bigger, and that's why we see a whole LLMOps tools track at this conference and so many companies popping up to help with this problem.

[5:29](https://www.youtube.com/watch?v=uiq95JYpBGY&t=329s) I feel really proud, and I always wanna say, we work with nearly all the foundation model companies. They have special needs, but they do push our scale. So a- almost all the LLMs that you would've used, from OpenAI's GPT to Mistral to Llama, these, these were built using the Weights & Biases platform.

[5:44](https://www.youtube.com/watch?v=uiq95JYpBGY&t=344s) So we have years of experience dealing with, with large scale. But it's also important to say almost all of our customers, and we, we have thousands of customers, are not foundation model builders, right?

[5:53](https://www.youtube.com/watch?v=uiq95JYpBGY&t=353s) They're, they're companies in, in nearly every industry. And the really fun part about being in the AI space right now- Is that there's applications to every industry. So we see healthcare, we see ag tech, we see it in manufacturing.

[6:04](https://www.youtube.com/watch?v=uiq95JYpBGY&t=364s) I remember when I, when I came out of school and wanted to get into the AI industry, it was really kind of ranking search results or, or going to Wall Street or targeting ads, right?

[6:12](https://www.youtube.com/watch?v=uiq95JYpBGY&t=372s) Which are, aren't the most exciting applications. Now, it's things that you can touch and feel. It's things that you can explain to your parents. This is, this is incredibly exciting development.

[6:21](https://www.youtube.com/watch?v=uiq95JYpBGY&t=381s) But what makes this hard, and I, I've thought a lot about this, what makes it hard, and, and, and what makes kind of AI engineering a different type of process with a different set of tools needed, because every software company has now launched an AI tool version of itself, right?

[6:35](https://www.youtube.com/watch?v=uiq95JYpBGY&t=395s) They, they call me up every day and they say, "You know, what do, what do you think? We're, we're launching, you know, AI version of observability. We're launching AI version of CI/CD.

[6:43](https://www.youtube.com/watch?v=uiq95JYpBGY&t=403s) We're launching AI version of, um, you know, code versioning."

[6:48](https://www.youtube.com/watch?v=uiq95JYpBGY&t=408s) Why don't those take, right? Why do we need a new set of tools? Why is Weights & Biases able to compete against companies that are, you know, much bigger and have lots of experience?

[6:55](https://www.youtube.com/watch?v=uiq95JYpBGY&t=415s) And I think it's fundamentally that the software development process, which I love, is a linear process.

[7:01](https://www.youtube.com/watch?v=uiq95JYpBGY&t=421s) You, you, you add features, you add code, and generally things improve over time. And the AI process is experimental. Almost all the things that you do when you're using an LLM is, is trying it, seeing what happens, right?

[7:15](https://www.youtube.com/watch?v=uiq95JYpBGY&t=435s) And it's, it's fundamental non, fundamentally nondeterministic. You don't know what's gonna happen. You're never gonna have a CI/CD test that's meaningful where 100% passes. So instead of kind of adding things all day long and, like what you do when you write code, you're doing lots of experiments.

[7:28](https://www.youtube.com/watch?v=uiq95JYpBGY&t=448s) I think this is a fundamentally different workflow, and what that means is that, you know, when you develop software, your code is your IP. Companies recognize this, and they take a lot of, uh, pain to protect their IP.

[7:39](https://www.youtube.com/watch?v=uiq95JYpBGY&t=459s) But when you're developing AI, it's the learning that's your IP. It's not the model that you build, right? The model's kind of an outcome, right? The prompt that you discover is kind of an outcome.

[7:48](https://www.youtube.com/watch?v=uiq95JYpBGY&t=468s) But the things that you learned along the way, all the prompts that you tried, all the workflows that you tried that didn't work, that's really the learning. That's really the IP that you have.

[7:56](https://www.youtube.com/watch?v=uiq95JYpBGY&t=476s) And if you're not saving that, when the person that figured out walks out the door, your IP walks out the door with them because you're not gonna be able to iterate further from there.

[8:04](https://www.youtube.com/watch?v=uiq95JYpBGY&t=484s) And so what that means is reproducibility matters, right? You don't really save your IP, you don't save your learnings if you can't reproduce your learnings, right? And this is a obvious thing.

[8:12](https://www.youtube.com/watch?v=uiq95JYpBGY&t=492s) I think everybody asks for reproducibility. We've always talked about a reproducibility crisis in AI. But reproducibility is incredibly hard. It's incredibly painful to keep track of everything so you can reproduce your stuff, especially with these nondeterministic endpoints.

[8:25](https://www.youtube.com/watch?v=uiq95JYpBGY&t=505s) And you see a reproducibility crisis in many fields, not just software.

[8:30](https://www.youtube.com/watch?v=uiq95JYpBGY&t=510s) And so my perspective is what you should be doing is tracking everything, and the tracking needs to happen with outside processes. You can't rely on humans to write everything down.

[8:39](https://www.youtube.com/watch?v=uiq95JYpBGY&t=519s) No matter how much they want to, they're gonna forget. You need to track everything in the background passively to have real reproducibility. And the real reason that you wanna do this is not just protecting your IP, but because now you can collaborate, right?

[8:50](https://www.youtube.com/watch?v=uiq95JYpBGY&t=530s) Now someone else can pick up the learnings that some engineer had.

[8:55](https://www.youtube.com/watch?v=uiq95JYpBGY&t=535s) And when you can collaborate well, you can iterate faster. And the iteration time is really the key to bringing things to market, right? If you wanna go from demo to production quickly, you need to bring down that iteration time, you need to have good collaboration, and you need speed.

[9:09](https://www.youtube.com/watch?v=uiq95JYpBGY&t=549s) So that's the real ROI, I think, of all these tools that we might be building. Now, we work with lots of enterprises and I, I, I wanted to give you an enterprise example, but more than that, I wanted to give you a real world example that, that mattered to me.

[9:22](https://www.youtube.com/watch?v=uiq95JYpBGY&t=562s) So this is very similar to what many of our enterprise customers are doing, but this is actually a project that I did, um, in a, in a, a few weeks that, that I took off after my second child was born.

[9:34](https://www.youtube.com/watch?v=uiq95JYpBGY&t=574s) Um, and actually, this is my first kid, um, Matilda, and, um, she was really demanding my attention after my, um, baby was born, and we were talking a lot, and she was actually, um, asking Alexa to play her favorite song.

[9:46](https://www.youtube.com/watch?v=uiq95JYpBGY&t=586s) And it was really interesting. I mean, Alexa's an interesting product. I, I really love it. My daughter is kind of connected to it, um, talks to, talks to her a lot.

[9:53](https://www.youtube.com/watch?v=uiq95JYpBGY&t=593s) But she was really kind of shocked when she asked Alexa to play her favorite song, and Alexa didn't know what her favorite song is because she asks Alexa to play "Baby Shark," like literally like five times a day.

[10:01](https://www.youtube.com/watch?v=uiq95JYpBGY&t=601s) So it's like there's no data science needed to know what the favorite song is. [laughing] But Alexa, um, couldn't f- couldn't, didn't know this. And I think there's kind of data privacy reasons that Alexa doesn't keep this, but it's kind of remarkable with all these amazing applications out there to see Alexa not able to do it.

[10:15](https://www.youtube.com/watch?v=uiq95JYpBGY&t=615s) And so, uh, my daughter was asking me, "Hey, could you build an Alexa?" And I was like, "I think maybe I could, actually, with, with LLMs today." So I kind of looked around with the open source Alexa alternatives where there's something called Mycroft that's, um, that's out there, but, but, um, kind of out of date.

[10:29](https://www.youtube.com/watch?v=uiq95JYpBGY&t=629s) Doesn't really use LLMs. And then my neighbor actually was showing me that he got, um, a seven billion, um, model, uh, I think Llama 2, running on a ROCK Pi, which is like a $200 device, similar price to Alexa.

[10:40](https://www.youtube.com/watch?v=uiq95JYpBGY&t=640s) So I thought, "Okay, if you have that, maybe we could build an architecture that would, um, kind of do the things that Alexa does." So built a library of skills, um, things that we ask Alexa to do, like play music, um, ask what the weather is, do math problems, um, check the news, things like that.

[10:56](https://www.youtube.com/watch?v=uiq95JYpBGY&t=656s) And these skills are really simple, right? So you know, like a- getting what the weather is is like a five line thing that actually, um, Copilot can generate instantly and works phenomenally well.

[11:05](https://www.youtube.com/watch?v=uiq95JYpBGY&t=665s) There's APIs here. Uh, the challenge is s- is actually understanding the speech and then getting the natural language into an API call. So kind of a simple architecture here, right?

[11:14](https://www.youtube.com/watch?v=uiq95JYpBGY&t=674s) So the weather comes in. You ask what's the weather in Boston. It goes through an on-device LLM to translate it to kind of Python looking code that says, "Weather location equals Boston."

[11:23](https://www.youtube.com/watch?v=uiq95JYpBGY&t=683s) Has to look like that because it's gonna call that function on the weather skill that I wrote that calls a, a weather API.

[11:30](https://www.youtube.com/watch?v=uiq95JYpBGY&t=690s) Um, yeah, I think Llama, I think, and Whisper is a fantastic open source stack here. llama.cpp is one of the many options to, to run Llama on, um, cheap hardware.

[11:39](https://www.youtube.com/watch?v=uiq95JYpBGY&t=699s) There's vLLM and others, um, and lots of companies that'll do it for you. I kind of wanted to do it myself, so llama.cpp was a great choice. Um, and you can run large LLMs on a MacBook in, in real time.

[11:50](https://www.youtube.com/watch?v=uiq95JYpBGY&t=710s) Um, maybe I'll skip the demos here. Not sure the audio is working. Um, the... What you will, if you, if you actually do try to do this yourselves, and a lot of our customers do try to do this themselves and they discover the same thing, latency really matters a lot, right?

[12:04](https://www.youtube.com/watch?v=uiq95JYpBGY&t=724s) Because you're actually transcribing audio and then you're running the aud- you're running the transcription through an LLM- To, um, make this function call. And so if that takes longer than a couple hundred milliseconds, it's incredibly painful.

[12:16](https://www.youtube.com/watch?v=uiq95JYpBGY&t=736s) So you need to go kind of small model here, and you need to, to run it fast on, on hardware. So that's why we went with, um, Llama 2 at the time, and put it in a default prompt, and it didn't work.

[12:26](https://www.youtube.com/watch?v=uiq95JYpBGY&t=746s) 0% accuracy. And so every kind of customer goes through this journey to try to improve the accuracy or whatever the real kind of user-facing metric is. I'll tell you briefly about my journey to improve this.

[12:37](https://www.youtube.com/watch?v=uiq95JYpBGY&t=757s) Um, you know, first thing I did was I did some prompt engineering and just kind of common sense to make the prompt better, kinda lay out what functions are available.

[12:45](https://www.youtube.com/watch?v=uiq95JYpBGY&t=765s) Um, the model kind of gets closer, in a sense. Now it's saying, "Call colon weather location equals Boston," but still actually not in any kind of format that I can run.

[12:52](https://www.youtube.com/watch?v=uiq95JYpBGY&t=772s) Here, I need really exact accuracy because I'm generating, um, code, function calls. Switched to Llama 2-Chat. Um, there's so many models to try these days. Um, Llama 2-Chat was trained on conversations, performed better in this case, and got the accuracy up to 11%.

[13:09](https://www.youtube.com/watch?v=uiq95JYpBGY&t=789s) Still not possible to ship this, um, but I kind of looked at the errors it was making. Looking at the errors, incorporating feedback, got the accuracy up to 75%.

[13:17](https://www.youtube.com/watch?v=uiq95JYpBGY&t=797s) So pretty satisfying, pretty iterative process, probably typical to what you're doing or what your engineers are doing. Now it's kind of a cool demo. My daughter is, like, mildly impressed if she says the, the thing just right.

[13:29](https://www.youtube.com/watch?v=uiq95JYpBGY&t=809s) But actually, 75% accuracy is incredibly annoying. It-it-it's, it's still not-

[13:34](https://www.youtube.com/watch?v=uiq95JYpBGY&t=814s) Okay, so-

[13:35](https://www.youtube.com/watch?v=uiq95JYpBGY&t=815s) Um, it's still not gonna, gonna work so well. Switched to Mistral. Um, Mistral came out mid-project here. Um, and Mistral's model for free, exact same API to Llama 2, got the accuracy up to 79%, right?

[13:46](https://www.youtube.com/watch?v=uiq95JYpBGY&t=826s) So this is sort of happening in real time, probably happening to you also. If you're ready to switch the models quickly, you can get free accuracy improvements as, as new models come out.

[13:56](https://www.youtube.com/watch?v=uiq95JYpBGY&t=836s) 79% accuracy is still really annoying, trust me. Um, [laughs] so I thought, "Okay, I need to, I need to improve this." Still not impressing my daughter. Um, and so I thought, "Okay, let's try some fine-tuning."

[14:07](https://www.youtube.com/watch?v=uiq95JYpBGY&t=847s) And now, um, you know, fine-tuning used to be a hard thing to do, especially on these really large models. Um, LoRA, QLoRA has made this a lot more tractable.

[14:16](https://www.youtube.com/watch?v=uiq95JYpBGY&t=856s) Now there's probably 20 companies here that will actually do this, um, for you for, for super cheap. So, you know, no excuse for not fine-tuning once you get to a point where, um, you wanna make that kind of last improvement of accuracy.

[14:27](https://www.youtube.com/watch?v=uiq95JYpBGY&t=867s) Um, I was able to run QLoRA on my GPU in my [REDACTED:location]. This runs on kind of standard hardware, I think it's a 4080, um, and, and ran a dataset in, in a few hours.

[14:38](https://www.youtube.com/watch?v=uiq95JYpBGY&t=878s) But you need the dataset. So I manually created a couple examples here, you know, using a larger model, in this case ChatGPT, to create more examples. Worked phenomenally well.

[14:46](https://www.youtube.com/watch?v=uiq95JYpBGY&t=886s) I think 95% of them were good. You know, I kinda went through by hand and, and filtered some out. But that, you know, in a, in, you know, 15 minutes, I have, um, a couple thousand examples to fine-tune on.

[14:56](https://www.youtube.com/watch?v=uiq95JYpBGY&t=896s) Um, I gave it a schema to format the answer in, and once it saw more examples, it was actually ge- able to generate more interesting examples kind of from that, that dataset.

[15:07](https://www.youtube.com/watch?v=uiq95JYpBGY&t=907s) Um, fine-tuning plus Mistral actually led to 98% accuracy. So this is super satisfying project. 98% accuracy actually starts to feel like, "Hey, maybe we could, we could ship this thing."

[15:17](https://www.youtube.com/watch?v=uiq95JYpBGY&t=917s) Um, and as a surprising byproduct, it actually worked for other languages. So I was using Whisper model that, that works really well in other languages. Um, Japanese is the one that I tested on.

[15:27](https://www.youtube.com/watch?v=uiq95JYpBGY&t=927s) I think this is, uh, French here. Um, but, but the, the LLM's also work in other languages. So we've kind of built this amazing multilingual Alexa as kind of a fun side project.

[15:37](https://www.youtube.com/watch?v=uiq95JYpBGY&t=937s) Which by the way, I think is exciting because I think there was a time when these models got really big, and there's always been kind of a vibrant hacker culture, I think, in AI, but started to wonder if, if you could only do really expensive projects to get anything interesting done.

[15:50](https://www.youtube.com/watch?v=uiq95JYpBGY&t=950s) I think now there's just an incredible amount of, um, things that, that people can do.

[15:56](https://www.youtube.com/watch?v=uiq95JYpBGY&t=956s) Um, so you can find this project at, uh, github[REDACTED:url]. Um, and the, and the lessons learned, stepping back, are exactly the things that our customers are discovering. It's exactly the things that you're probably discovering.

[16:07](https://www.youtube.com/watch?v=uiq95JYpBGY&t=967s) Prompt engineering works really well, especially when you sto- start with a terrible prompt. And then fine-tuning improves the performance at the end. People often talk about, "Hey, should I be using RAG?

[16:16](https://www.youtube.com/watch?v=uiq95JYpBGY&t=976s) Should I be using fine-tuning? Should I be using prompt engineering?" I think almost always in the end, to get something shipped into production and working well, you end up using a whole bunch of these different, um, methods.

[16:26](https://www.youtube.com/watch?v=uiq95JYpBGY&t=986s) Everything is making iterative improvement. Gradually, we get to the accuracy that we need. Um, the other thing that I kind of hid from you in this project is this did take me a couple weeks to get done, and most of the experiments didn't work, right?

[16:38](https://www.youtube.com/watch?v=uiq95JYpBGY&t=998s) So, you know, in, in the real world, you know, if I gave this to a colleague and they tried to iterate o- on top of it,

[16:45](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1005s) they would-- if they just saw this talk here, they wouldn't realize all the things that I tried that failed, and they'd probably start to do them over and kind of have to experience the same failures to catch up to where I was.

[16:56](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1016s) And so, um, we actually kept track in W&B of, of all the, um, the failures and experiments, and you can go in there and, and look at them yourself.

[17:03](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1023s) But I think this is generally best practice.

[17:06](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1026s) Um, so what, what are the lessons? You know, people come to me now e-every day asking, "Hey, you know, how do I get LLM apps from demo into production?

[17:16](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1036s) What are the kind of broader lessons besides using, um, Weights & Biases?" which you obviously are already doing. Um, and I think the, um... I think that there's kind of four things that we see our customers do where they actually get things into production, they actually make them successful.

[17:32](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1052s) Um, the, the first is that they build an evaluation framework, and they take this really seriously. We've had people-- I wrote a-- I do a podcast called Gradient Dissent, where I talk to a lot of people.

[17:41](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1061s) Um, and, and one prominent CEO of a well-known company, um, came on the podcast, and I asked him, you know, "How do you do testing? How do you test your GenAI applications in production?"

[17:51](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1071s) And he said, "I, I, I test these applications by vibes." [laughs] [laughs]

[17:56](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1076s) And, um, I asked him, you know, "Hey, do you want me to remove that, you know, from the, the pod?" Like, we're not trying to, like, embarrass people on, on our podcast, you know?

[18:03](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1083s) And he's like, "No, no, it's great. I, I love it," you know? Um, and, and I think that there actually is a kind of a value to testing by vibes.

[18:10](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1090s) Like, I think, um, if more CEOs tested by vibes, you might actually not have some of the failures that, that you see. I mean, some of these things that get released are, are so embarrassing that I think a vibes check would've actually, um, caused better decision-making.

[18:23](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1103s) But When you're only testing by vibes, the problem is you can't release a V2, right? Because what happens is, unlike in software development where you're typically adding features and mostly moving forward,

[18:34](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1114s) any of these new things that you try are gonna make some things better and some things worse, and the vibes check is not gonna tell you the difference between a 75% accuracy and a 79% accuracy.

[18:43](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1123s) So he never would've been able to make that switch from Llama 2 to Mistral because it just wouldn't have been clear if it's better or worse. And so taking time to build an evaluation framework is really the, the foundation of getting these things into production and working well.

[18:56](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1136s) I actually checked back with that CEO a year later, and he was doing quite a lot of evaluation because he kind of realized that thing where you wanna get that second model out, and you, you don't know how to do it.

[19:05](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1145s) Um, the other thing is starting with a lightweight prototype. We see typically, this is kind of enterprise anti-pattern, but we see it a lot at Weights & Biases, where people kinda wanna build the first step, and kinda nail that, and then build the second step, and they kinda never get something into the hands of the end user,

[19:19](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1159s) which might be an internal audience or it might be a customer. I think this is kind of basic agile product development, but somehow people forget this with GenAI, so it's worth mentioning because we see this failure mode so frequently.

[19:30](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1170s) And then, of course, incorporating end user feedback in lots of different ways is, is critical.

[19:38](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1178s) Um, and then iterate. So I'll say, you know, evaluation best practices because I think it's such a cornerstone, and I think... Well, actually, let me ask you this. How many, how many people in the audience feel like they're doing solid evaluations of their GenAI, um, models?

[19:53](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1193s) GenAI applications? Interesting. So, so a lot less. It seems like you feel people are, you all are not feeling great about your, um, [laughs] evaluation frameworks. Um, I think, I think what we see, where people kind of end up, um, is lots of different evaluation sets and techniques.

[20:09](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1209s) Um, so we see a lot of our customers will do some things like, "Okay, these are the things we can never get wrong, and we wanna have, like, 100% on this simple test."

[20:15](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1215s) And there's lots of tricks to getting that right. Then people wanna have, you know, something that's sort of very quick to run, so they can, they can kind of run it in real time.

[20:23](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1223s) They can test something new in 20 or 30 seconds. And then our customers typically will have much bigger evaluation sets that run nightly and give feedback. That's where people end up, and I think that's really important to do all the different kinds of evaluations that might matter for your application.

[20:37](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1237s) And then, you know, the other thing that's very hard to do, but I mean, MLOps has been doing this for decades, is trying to make sure that the metrics correlate with the user experience or the value that you actually provide for your customer.

[20:48](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1248s) Totally application dependent, um, very hard to do. I mean, when we started Weights & Biases, we kind of thought people might wanna track, I don't know, like 10 or 20 metrics per m- model that they build or per application, and we see with, with people that are really in production, like a typical, you know, self-driving car company

[21:04](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1264s) or, like, a typical bank with a mission critical application, we see thousands of metrics. We see tens of thousands of metrics. We get requests like, "Hey, can I do regex search across my metrics?

[21:13](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1273s) Because I have, like, so many metrics, I can't even [laughs] I can't find, you know, the, the, the metrics." And, and I think that's because, um, you really wanna get these metrics working and, and connected to your user experience, and there's so many different ways that something could fail.

[21:25](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1285s) But then the most important thing is to actually do it, right? It seems like a lot of people in this room maybe haven't started to do it yet, but doing evaluation is so critical because you can't, you can't improve any of these other things.

[21:36](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1296s) And when people come to me and they ask me questions like, you know, "Should I be doing fine-tuning? Should I be doing RAG?" You know, "Should I be doing XYZ?"

[21:45](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1305s) What that tells me is they don't have a good evaluation system in place, right? Because it shouldn't take you very long to have an answer for yourself, for your own application, um, if one of these techniques works or not.

[21:56](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1316s) Um, so look, you know, we're here at this conference, and this is a lot of fun. Um, if you wanna come by our booth, the Weights & Biases booth, we can show you, um, Weave, which we can help you with some of your problems getting, um, getting AI applications into production, but, but not all of them today.

[22:10](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1330s) But we'd love to talk to you about what we're doing, and we'd love to hear from you about how your experience has been. Thank you. [audience applauding] [upbeat music]
