AI Engineer World's Fair 2024
What We Learned From A Year of Building With LLMs
Read the talk
What a Year of Building with LLMs Teaches About Production
Useful LLM products depend on domain expertise, deliberate hiring, and a feedback loop that turns real user interactions into evaluations, better decisions, and production guardrails.
From a talk by Eugene Yan, Hamel Husain, Jason Liu, Dr Bryan Bischof, Charles Frye and Shreya Shankar
Before you start: Familiarity with LLM APIs and retrieval-augmented generation will help; no model-training background is required.
Six practitioners encounter the same problems
What happens when six practitioners discover that they are all about to write the same lessons-learned article? After a year of posting, blogging, and comparing frustrations in a group chat, Eugene Yan, Hamel Husain, Jason Liu, Bryan Bischof, Charles Frye, and Shreya Shankar combined their experiences into What We’ve Learned From a Year of Building with LLMs, published through O’Reilly. Their recurring problems crossed three levels: choosing what to build, organizing the work, and making the system reliable.
The collaboration became a 30-page article and then an unusual six-person keynote, with room for sharper opinions that had missed the written version. The Avengers comparison fit the format, though the authors pushed back on being mistaken for opportunists arriving from crypto: their experience extended back to writing neural-network gradients by hand. Charles and Bryan begin with strategy, Jason and Hamel move into operations, and Shreya and Eugene finish with production techniques.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a product advantage that survives better models
How do you build an LLM application without getting outmaneuvered or wasting everyone’s money? Training a custom model can look like the obvious route: pretraining, fine-tuning, perhaps some reinforcement learning from human feedback. Bryan rejects that as the default. For almost everyone in the audience, the model is not the moat. Product expertise, an existing customer relationship, or a well-understood niche offers a more useful starting point.
Build where model providers are unlikely to concentrate. Providers must serve requirements shared across many customers; reproducing those common capabilities creates exposure to the next platform release—the steamrolling risk Bryan invokes. Treat the underlying model like another SaaS dependency and replace it when a competitor becomes clearly better. His contemporary comparison is GPT-4o versus Claude 3.5 Sonnet, illustrating the willingness to switch rather than prescribing a permanent winner.
Benchmark performance does not define a customer outcome. Bryan uses an illustrative 87% SpiderSQL score to explain why benchmark accuracy does not imply automating 87% of a company’s data requests. High MMLU performance and a HumanEval score are similarly difficult to sell as products in themselves. A product still needs good design, a job to be done, and a way to make its user more capable.
That suggests three durable places to invest:
- Adaptability: capabilities that remain useful as models become smarter and faster.
- Quality control: mechanisms that maintain the product’s quality bar despite uncertain outputs.
- Continuous improvement: ways to learn from use and make the next version better.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluation belongs inside an improvement loop
The reason to collect data and create evaluations is to move an improvement loop forward. Charles starts with Hamel’s diagram of that loop, then connects it to Shreya’s work on operationalizing machine learning: collect data, experiment, evaluate and deploy, monitor the result, and feed what you learn into the next iteration. Evals occupy the center because they help determine whether a change worked. Collecting them is not the endpoint.
The pattern predates LLMs. MLOps inherited iterative improvement and production monitoring from DevOps. DevOps, in turn, drew on the Lean Startup movement’s use of measurement to improve an entire business. Charles traces the lineage further back to the Toyota Production System and Kaizen, or continuous improvement. Each version addresses uncertainty by observing the real system and using that observation to decide what to change.
Toyota’s Genchi Genbutsu principle brings this down to a concrete practice: go to the real place and inspect the real thing. For executives, that meant the factory floor. For LLM engineers, it means reading actual application data—the requests, responses, and interactions that show whether users receive value.
Tools can make the loop easier to operate, but they can also become a distraction. Charles invokes a line he attributes to Shigeo Shingo: “Value is only created when metal gets bent.” Building an evaluation framework or calculating concept drift matters when it leads to a better experience for users. The machinery of improvement must eventually produce an improvement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Get real interactions into the loop
Evals express objectives: what should the system do? Starting with a convenient bespoke metric reverses that relationship. Bryan recommends beginning with any adequate system for capturing the behavior that matters, rather than waiting for the perfect evaluation framework.
A beta can start that process. Bryan points to Apple Intelligence, Photoshop, and Hex as examples of products using that route to put capabilities in front of people. Once users interact with the system, inspect the responses with human eyes. AI review can help, but it does not remove the need for direct human inspection.
Start with binary human feedback, then add richer explanations where they help. Those interactions expose more than quality problems: users ask chatbots to do things the product does not yet support. Repeated unmet requests can reveal where product-market fit might lie. The same stream of data can therefore guide both reliability work and the product roadmap.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prototype for economics that may become possible
Immediate user feedback does not eliminate the need to plan for future capabilities. Alan Kay and colleagues at PARC projected Moore’s Law forward, then built expensive, slow, buggy systems to experience the computing environments that would eventually become affordable. A prototype could be economically impractical and still teach its builders what the future product should feel like.
Charles applies that approach to inference costs, while acknowledging that the scaling evidence is less established than the industrial evidence behind Moore’s Law. Charles reports roughly tenfold cost reductions every 12–18 months at three capability levels: GPT-3 davinci, text-davinci-002, and GPT-4/Sonnet. His shorthand is about fifteen months per tenfold reduction, faster than Moore’s Law. The companion analysis uses five-shot MMLU as a capability proxy; matching a benchmark score does not establish equivalent behavior in a particular product.
The planning question is which currently uneconomical applications might become viable by the next fundraising round. Charles estimates a chatbot-NPC game at about $625 per hour in 2023, compared with roughly $6 per hour for inflation-adjusted 1980 Pac-Man play. If the assumed cost trend continued, two tenfold reductions over about thirty months from mid-2023 would bring the chatbot game to roughly $6 per hour. Those are approximate economics and a conditional forecast, including the expectation that people would pay. But they motivate useful work before commercialization: experience the game, design its interactions, and learn what makes it compelling.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Buying another shovel does not tell you where to dig
Jason turns the operational discussion into deliberately bad advice. Drawing on consulting work with Hamel across roughly twenty companies, from pre-seed startups to public businesses, he offers a recipe for ruining a business: keep buying shovels. Where should you dig? Buy another shovel. When should you stop? Surely the shovel will tell you. Should you dig one deep hole or many shallow ones? More shovels.
The software equivalents are familiar: a failing RAG application prompts a vector-database swap; an ineffective method prompts implementation of another paper; weak product-market fit prompts an embedding-model upgrade. Any of those changes might solve an identified technical problem. None supplies the missing decision process by itself.
The satire targets substitution of tools for expertise and process. Teams still need to decide where to experiment, when to stop, and when to exploit a mechanism that already works. A better tool cannot make those choices on their behalf.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Premature model ownership creates the wrong work
Jason’s next satirical recommendation is to replace a $2,000 monthly OpenAI bill with a $250,000 hire plus 1% equity. The new engineer can fight CUDA build errors and server cold starts. Anxiety about being a wrapper then becomes another reason to fine-tune, even when the harder business problem is building something worth charging for. The comparison exposes the costs that disappear from view when API spend is considered in isolation.
Hiring a machine-learning specialist before a data-generating product also creates a mismatch between the hire and the work. The urgent task may be fixing a Vercel TypeScript build, not training a model. Jason extends the joke to claims that Python is dead, that its ecosystem can be rebuilt over a few weekends, and that tooling, product sense, and data literacy transfer effortlessly between communities. The sarcasm depends on the opposite being true: these are substantial bodies of expertise.
A catch-all title does not resolve that mismatch. Ninja, wizard, data scientist, prompt engineer, or AI engineer can all conceal an unspecified set of outcomes. Jason hands the problem to Hamel with a familiar prompting joke: “take a deep breath, think out loud, step by step.” The operational task is to become more explicit about the skills and results the business actually needs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluation is a core engineering skill
Hamel values the AI engineer label popularized by Swyx because it brings a community together. The problem is inflated expectations about what the title guarantees. Talent is a major lever for founders and engineering leaders, so misunderstanding the required skills directly affects what the team can deliver.
A familiar skills diagram places AI engineering on the application side of an API dividing line, emphasizing chains, agents, tooling, and infrastructure. Evals and data are conspicuously absent. Read too literally, that diagram suggests that application builders can ignore evaluation. Tools and vibe checks can get a team from zero to one quickly, but without measurement the team cannot reliably identify improvements. The initial speed gives way to stagnation.
Hamel reports that four to six weeks of deliberate practice can make engineers effective at writing evals and working with data. This is his and Jason’s consulting experience, not a guaranteed training schedule. The practical implication is that evaluation and data literacy belong in the core of AI engineering and can be deliberately developed within the team.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Hire for the next stage of the product
An ambiguous AI role can become the destination for every AI-related failure, regardless of whether the person has the skills to fix it. Data science provides a precedent: the original unicorn combined software engineering, statistics, mathematics, and domain expertise. Over time, organizations separated that work into roles such as decision scientist, machine-learning engineer, and data engineer. AI engineering may undergo a similar differentiation.
Specify both the skills and the product context. Hamel uses GitHub Copilot job postings to illustrate that specificity, then describes a hiring sequence tied to maturity:
- Build the application. Application development, software engineering, and AI engineering get the product from zero to one.
- Capture its data. Platform and data engineering make interactions available for analysis and improvement.
- Add machine-learning specialization. Hire ML engineers once there is data for them to work with.
The warning is straightforward: do not hire a machine-learning engineer into a product that has no data. Developing eval and data skills in the existing team can extend its effectiveness before that specialization is needed. The recurring failure mode is an AI engineer role that is over-scoped but under-specified.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn broad quality goals into testable criteria
Eugene begins the tactical discussion with a Hacker News observation: the importance a team gives evaluation separates poor outputs from real products. The useful question is what to evaluate for this product. He points to Apple’s foundation-model evaluation, which uses summaries representative of actual features. Apple’s published evaluation sampled 750 responses for each summarization use case: email, messages, and notifications. The unit of evaluation is the feature people use, rather than a generic model score.
Start by decomposing the task. Product-attribute extraction becomes checks for title, price, and rating. Summarization becomes dimensions such as factual inconsistency, relevance, and informational density. Natural-language-to-SQL generation can include checks for expected tables and columns. Narrow criteria make failures easier to identify and give simple assertions useful work to do.
For an illustrative product record, the extraction checks can be ordinary Python assertions. This fixture represents a description of a Trail Mug priced at $18.50 with a 4.6 rating; each field has a separate failure message.
python
from decimal import Decimal
def check_product(actual: dict, expected: dict) -> None:
assert actual["title"] == expected["title"], "incorrect title"
assert Decimal(str(actual["price"])) == expected["price"], "incorrect price"
assert actual["rating"] == expected["rating"], "incorrect rating"
expected = {
"title": "Trail Mug",
"price": Decimal("18.50"),
"rating": 4.6,
}
extracted = {"title": "Trail Mug", "price": "18.50", "rating": 4.6}
check_product(extracted, expected)
These checks isolate field correctness; they do not require a model to judge the entire response as one undifferentiated object.
Assertions eventually reach their limits. For criteria requiring semantic judgment, Eugene suggests a factual-inconsistency classifier or a relevance reward model. Formulating the criterion as classification or regression gives an evaluator a more precise job than deciding whether an output is broadly good.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose an evaluator you can align and maintain
Shreya makes the case for LLM-as-a-judge as an easy starting point. Write a prompt describing the criterion, then provide a few examples of good and bad outputs to express your preferences. A trained evaluator demands more upfront work: collecting data and building a training pipeline before it can help.
Eugene’s concern is alignment with business-specific criteria. A prompted judge may be easy to create but difficult to make precise. Eugene suggests that a few hundred to a few thousand examples may suffice to fine-tune a simpler, more precise evaluator. He also estimates five to eight seconds per request for a judge using chain-of-thought, versus roughly ten milliseconds for a classifier or reward model. These are estimates without a specified model, hardware, or load; their practical significance is the possible effect on throughput and response latency.
Both approaches create maintenance obligations, whether the validator runs asynchronously or in the request’s critical path. A prompted judge needs examples and instructions that continue to match the current definition of good and bad. A trained validator needs new production data and continued fine-tuning to resist drift. An evaluator can become stale even when the code calling it has not changed.
| Decision | Prompted LLM judge | Trained evaluator |
|---|---|---|
| Starting investment | Criteria prompt and examples | Labeled data and training pipeline |
| Alignment work | Revise prompts and examples | Train on task-specific labels |
| Ongoing maintenance | Refresh the definition of good | Update with production data |
| Best initial fit | Low-volume, quick prototypes | Greater resources and product confidence |
Shreya frames the choice around resources and application maturity. For a prototype needing quick evals with little development effort, start with a judge and invest in keeping it aligned. As confidence in the product and available resources grow, a trained evaluator becomes more attractive.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspect recognizable failures with their execution context
Where do the evaluation criteria come from? Regular inspection of production data. Shreya describes a startup that creates a dedicated Slack channel whenever it ships a new agent and streams the agent’s outputs there in real time. After a couple of weeks, it transitions to daily batch inspection. The cadence changes, but observation continues beyond the initial deployment.
Look for characterizable slices: outputs from a particular source, containing a keyword or phrase, or concerning a topic. A collection labeled bad is hard to act on if the failures have no recognizable common property. A slice gives the team something it can locate again, investigate, and use to assess a pipeline change.
Preserve the execution context alongside those outputs. Record the GitHub commit, model version, and prompt version so an inspected trace can be connected to the system that produced it. Shreya points to MLflow as an example of the traceability that traditional MLOps tools made useful, with replay as a possible next step.
Finally, pin model versions where the provider supports it. Shreya’s historical examples are GPT-4 1106 and GPT-4o; in an implementation, a model-family name alone does not establish that a fixed snapshot is selected. The purpose is to limit behavior changes that are difficult to quantify and to preserve a useful explanation of what ran. Version records aid diagnosis, but do not by themselves guarantee exact reproduction of a hosted model’s output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn reference-free evaluations into guardrails
Human inspection cannot examine every output continuously. Automated guardrails extend the checks across production traffic. Eugene adapts Brandolini’s law to emphasize the asymmetry: generating an output is easy; detecting and fixing its defects takes much more work. Basic checks include toxicity, personally identifiable information, copyright, and the expected language.
Even apparently straightforward requirements can fail because the application does not control all of its context. An advertisement submitted in another language to an English website can cause attribute extraction or summarization to return that language. A movie summary generated from a trailer description can introduce spoilers or other facts absent from the supplied text. The model’s attempt to be helpful can violate the actual product requirement.
The useful distinction is whether evaluation requires an ideal answer for each input.
| Evaluation style | Comparison target | Production implication |
|---|---|---|
| Reference-based | A gold output | Requires an ideal sample for comparison |
| Reference-free | Source document and criteria | Can check newly generated outputs |
Reference-free does not mean evidence-free or label-free. Labels can still train and evaluate the checking model. What disappears is the need to write a gold response for every new input.
For summaries, inspect whether the source supports the generated claims and whether they contradict it. That turns factual consistency into a reusable hallucination check: the incoming source document supplies the evidence for judging each new summary. An evaluation that works without a gold response can also become a production guardrail. The investment in defining and training the check can therefore serve both offline evaluation and new outputs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The surrounding system still needs maintenance
The closing production-system diagram puts the model inside a much larger body of engineering work. Data verification, feature engineering, and infrastructure monitoring all require maintenance. Replacing a traditional model with an LLM does not remove those surrounding responsibilities or the technical debt they can create.
The responsibilities often return in a different form:
| Traditional ML responsibility | LLM application counterpart |
|---|---|
| Feature engineering | Retrieve and shape relevant context through RAG |
| Data validation and verification | Evals and deployed guardrails |
| Infrastructure monitoring | Monitor the production system around the model |
RAG requires decisions about which context to retrieve and how to present it without distracting the model. Those decisions involve experiments and ongoing maintenance, especially as underlying models change. Evals and guardrails likewise have to remain useful as the application evolves. A model wrapped in software is only one component of the investment needed to operate the product.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A compelling demo can precede a product by decades
Eugene ends with an observation attributed to Andrej Karpathy: a large class of problems is easy to imagine and demonstrate, yet extremely difficult to turn into products. Autonomous driving makes the gap tangible. The ALVINN paper from 1988 demonstrated neural-network road following under particular conditions. It was an early concrete result, not a finished unrestricted driving service.
The keynote then jumps twenty-five years to a demo ride Eugene says Karpathy took in 2013, labeled Waymo on the slide, and another decade to San Francisco driverless passenger service. The precise 2023 milestone was CPUC approval expanding paid driverless service to all times of day; earlier non-fared authorization already existed. The timeline shows how much development can separate an impressive demonstration from a service operating under real-world requirements.
Eugene’s suggestion that the service might reach all of California a few years later remains a forecast within the talk. His closing point is more durable: getting from demo to production takes time. The work around the model—learning from users, defining quality, maintaining data and context, and catching failures—is how a team spends that time productively.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The six authors’ practical guide to tactical, operational, and strategic decisions when building LLM products.
Apple’s original account of its foundation models and product-specific evaluations for email, message, and notification summaries.
Pomerleau’s NIPS 1988 paper describes neural-network road following using camera and laser-range inputs.
Further reading
Shreya Shankar and colleagues study how 18 ML engineers experiment, deploy, monitor, and maintain production systems.
The 2015 paper explains maintenance costs and dependencies surrounding production machine learning models.
Updates since the talk
Current instructions for recording and inspecting experiment runs, parameters, metrics, and artifacts.
Read the complete timestamped transcript
- 0:00
[upbeat music] You're about to experience something of a strange talk, and not just because Bryan and I are strange, but because something kind of strange happened.
- 0:24
Over the last year, a bunch of us were posting things on Twitter. We were, uh, writing blog posts complaining about LLMs. And we formed a little group chat, and we were, you know, continuing to complain about LLMs to each other, uh, and sharing what we were working on when we realized we were all about to write the
- 0:42
exact same blog post: what we learned in the last year. So we, we got together, and we, uh, we turned what was initially a couple of short blog posts into a long whitepaper on O'Reilly, uh, combining our lessons across strategic, operational, and tactical levels of building LLM applications.
- 1:04
And the response to that whitepaper was overwhelmingly positive. We got, uh... We heard from everybody from people who contribute to Postgres to venture capitalists to tool builders saying, "We loved what you wrote in that article."
- 1:21
Um, uh, like, "I've felt that pain, too." And we were invited on the strength of that to give this keynote address. And so we faced a kind of funny challenge, which is part of the appeal of this blog post, uh, of this article, was that the six of us all came together to write it.
- 1:38
As, uh, Scott Conran put it, it was like the, an Avengers team-up. Uh, so we had to figure out a way to deliver, uh, one keynote talk from six people.
- 1:49
Uh, so we, we pulled the Avengers together [laughs] for, uh, one night only, uh, to sort of ex- like, deliver some of the most important insights from that 30-page article, uh, to add some of our spicy extra takes that ended up on the cutting room floor, and to respond to the allegations.
- 2:11
I'd like to state unequivocally that we are not, in fact, crypto bros who just found out that GPT-4 was the new Web3. Um, uh, we all trained our first neural networks back when you had to write the gradients by hand.
- 2:27
So, uh, we split the article up into three pieces, so we split the talk into three pieces. First, you're gonna hear from me and Bryan talking about the strategic considerations for building LLM applications.
- 2:38
How do you look to the future? How do you see around corners? How do you make big decisions?
- 2:44
Then we're gonna hand the clickers and the stage over to Hamel Husain and Jason Liu, who are gonna share the operational considerations. How do you put together processes? How do you put together teams?
- 2:57
How do you think about workflows around delivering LLM applications?
- 3:02
And then they will hand over the clickers and the stage to, uh, Shreya Shankar and Eugene Yan, who will talk about the tactical considerations for building LLM applications. What are the specific techniques, tactics, and moves that have stood the test of one year's time for building LLM applications?
- 3:23
All right. So Bryan, how do you build an LLM application without getting outmaneuvered and wasting everybody's time and money?
- 3:31
Ah, yes. Yes. Well, many of you may be thinking that there's really only one way to win in this new, exciting, dynamic, and very scary industry. And that, of course, is to train your own custom model.
- 3:46
Pre-training, fine-tuning, a little RLHF here and there. You better start from scratch, buddy. Eh, not quite. The model is actually not your moat.
- 3:58
For almost no one in this audience, the model is the moat.
- 4:04
You all, as AI engineering devotees, should be building in your zone of genius. You should be leveraging your product expertise or your existing product, maybe you've got one, and you should be finding your niche and digging into that niche, exploiting it.
- 4:25
You should be building what the model providers are not.
- 4:29
There's a high likelihood that the model providers have to build a lot of things for all of their customers. Don't waste your calories on building these things. The Sam Altman phrase of steamrolling is appropriate here.
- 4:44
And you should be treating the models like any other SaaS product. You should be quickly dropping them when there's a competitor that's clearly better. No offense to GPT-4.0, but Sonnet 3.5 looking, looking pretty sharp. [laughs]
- 5:02
It's important to keep in mind that a model with high MMLU scores, that's not a product. 87% on SpiderSQL, that doesn't automate all data requests or even 87% of them.
- 5:16
You can't sell HumanEval pass at 67. At least my GTM team doesn't know how.
- 5:24
An excellent LLM-powered application is an excellent product.
- 5:30
It's well-designed. It solves a job to be done.
- 5:36
And it enhances your user. Why are we so excited about AI? Human enhancement.
- 5:45
So what should you build, if not all these things?
- 5:50
Things that generalize to smarter and faster models.
- 5:55
Things that help you maintain your product's quality bar under uncertainty.
- 6:02
And things that help you continuously improve.
- 6:06
Whoa, Bryan. Continuous improvement. That's, uh, that's my trigger phrase. [laughs]
- 6:14
The idea of continuous improvement has been brought to the world of LLM applications by, like, this shift in focus that we've all felt since the previous AI engineer summit to focus on evaluation and data.
- 6:32
It's nicely synecdochilized by this diagram from our co-author Hamel Husain showing this virtuous cycle of improvement. It has evals and data at the center, but the core reason to create those evals, the core reason to collect that data, is to drive forward this loop of continuous improvement.
- 6:52
And despite what your expensive consultants or, uh, your... the, many of the, uh, LinkedInfluencers posting about LLM apps might say, this is not actually the first time that engineers have tried to tame a complex system and make it useful and valuable.
- 7:11
This same loop of iterative improvement was also at the core of MLOps at the operationalization of machine learning models before LLMs. This figure from our co-author Shreya Shankar's paper, uh, had that same loop of iterative improvement centered also on evaluation and on data collection.
- 7:34
MLOps was also not the first time that engineers faced this problem, the problem of complexity, the problem of nondeterminism and uncertainty. This-- The DevOps movement that gave MLOps its name also focused on this kind of iterative improvement and on monitoring, uh, information in production to turn into improvements to
- 7:59
products. But dear reader, DevOps was not the first time that engineers tackled this problem of uncertainty
- 8:09
and solved it with iterative improvement. DevOps built on the ideas of the lean startup movement, uh, from Eric Ries that s- that was focusing not just on building an application, not just on building an, a machine learning model or an LLM agent, but on building the entire business.
- 8:28
And it used this same loop centered on measurement and data, uh, to, to drive the improvement of and building of a business.
- 8:38
This idea itself was not invented in Northern California, despite what, uh, some people might say. It has its roots in the Toyota Production System and in the idea of Kaizen, or continuous improvement.
- 8:53
Genchi Genbutsu is one of the core principles from that movement that we can take forward into the development of LLM applications. It means real things, real places, and at Toyota, that meant sending executives out to factory floors, getting their khakis a bit dirty.
- 9:08
For LLM applications, the equivalent is looking at your data.
- 9:13
Look at-- The, that data is the real information about how your LLM application is delivering value to users. There's nothing that is more valuable than that.
- 9:25
Finally, it's-- there's lots of people selling tools at this conference, including myself. It's easy to get overly excited about the tools and the construction of this iterative loop of improvement and to forget where value actually comes from.
- 9:38
And there's a great pithy, earthy statement from the Toyota Production System from Shigeo Shingo that I really like: "Value is only created when metal gets bent." So we have to make sure that we don't get lost just building our evals and calculating concept drift, and we instead make sure that we continue to get out there and bend
- 9:58
metal and create value for our users.
- 10:01
Not gonna lie, I might have misunderstood earlier when you said, "Let's get bent." Okay. [laughs] So right off the bat, we need to spin that data flywheel, Bob. Oh, wait, sorry.
- 10:11
Wrong, wrong game show. Point is, we need to get this moving. We need to get this in front of users and human beings. We need to express the goals for our system.
- 10:20
And how do we do that? With evals. Remember, evals are not convenient, weird, bespoke, uh, metrics. Evals are objectives. They're what we want our system to do.
- 10:32
Any system for capturing this behavior is good enough. I don't have an evals framework to sell you, but what I do have to sell you is this idea that you should be getting out there.
- 10:44
You should be getting started. But wait, Bryan, I'm really nervous. What if this isn't good enough for my customers? Fear is the mind killer. [laughs]
- 10:56
Put it out there in beta. If it's good enough for these incredible companies like Apple Intelligence, Photoshop, and Hex, that's me, it's good enough for you. You need to collect this data.
- 11:07
You need to put something in the wild. You need to start looking at your user interactions, the real user interactions. LLMs responses deserve human eyes. You can give it some AI eyes too, but definitely look at it with your human eyes.
- 11:25
Binary human feedback is valuable. It's nice to add some rich feedback too. That can be interesting. But start with binaries. And finally, user requests will reveal the PMF opportunities that lie below your product substrate.
- 11:41
Where is your PMF? Everybody wants to know. It's in your user interactions. What are they asking your chatbot that you haven't yet implemented? That's a really nice direction to skate if that's where the puck's going.
- 11:57
And despite the focus on the user interactions that you can have today, the things that you can ship right now, it's important to also think about the future. The best way to predict the future is to look at the past, find people predicting the present, and copy what they did.
- 12:17
In designing the, many of the components of the personal computing revolution, Alan Kay and others at PARC adopted as a core technique projecting Moore's Law out into the future.
- 12:29
They built expensive, unmarketable, slow, and buggy systems themselves so they could experience what it was like and build for that future and it, and create it.
- 12:41
We don't have quite the industrial scaling, uh, information that, uh, that Moore had when he wrote down his predictions, but we do have the beginnings of those same laws.
- 12:54
There's been an order-of-magnitude decrease every twelve to eighteen months at three distinct levels of capability. At the capability of davinci, the original GPT-3 API that brought-- that excited a lot of us about the idea of building on foundation models.
- 13:10
The capabilities of text-davinci-002, the model lineage underlying ChatGPT that brought the rest of the world t-to excitement about this technology, and the latest and greatest level of capabilities with GPT-4 and Sonnet.
- 13:25
In each case, around fifteen months is enough time to drop the cost by an entire order of magnitude. This is faster than Moore's Law.
- 13:35
And so the appropriate way to plan for the future is to think what this implies for what applications that are not economical today will be economical at the time that you need to raise your next round.
- 13:49
Uh, so in twenty twenty-three, it cost about six hundred and twenty-five dollars an hour to run a video game where all the NPCs were powered by a chatbot. That's pretty expensive.
- 14:00
In nineteen eighty, it cost about six dollars an hour to play Pac-Man, inflation adjusted. That suggests that if we just wait for two orders of magnitude reduction or about thirty months from mid twenty twenty-three, it should be possible to deliver a compelling video game experience with chat-chatbot NPCs at about six dollars an hour, and people will probably
- 14:21
pay for it. So you can't sell it now, but you could live it and you can design it, and you can be ready when the time comes.
- 14:30
So that's how to think about the future and how to think strategically when building LLM applications. I'd like to call to the stage my co-authors, Jason Liu and Hamel Husain, to talk about the operational aspects.
- 14:42
Let's give them a hand. [audience applauding]
- 14:48
All right. Thank you, man.
- 14:52
Cool.
- 14:52
All right. So Hamel and I have basically been doing a lot of AI consulting in the past year, right? We've worked with about twenty companies so far, and, you know, we've done something from pre-seed all the way to public companies, and I'm pretty bored of giving generic good advice, especially because there's such a range of operators here.
- 15:10
And so instead, I'm gonna invert. My goal today is to tell you how to ruin your business. [laughing]
- 15:17
First of all, everyone knows that in the gold rush, you sell shovels. And so if you wanna get gold, you gotta buy shovels too, right? You know, if you wanna find more gold, keep buying shovels.
- 15:29
Where do I dig? Keep buying shovels. How do I know when to stop digging? The shovel will tell you. And how do I dig one deep hole versus making investments in a plenty of shallow holes?
- 15:39
Again, the answer is more shovels, clearly, right? And this might be generic, so I'll give you some more specific advice. If your RAG app doesn't work, try a vector database, a different vector database.
- 15:51
If the methodology doesn't work, implement a new paper. And maybe if you update the embedding model, you'll finally find product market fit. [laughing]
- 16:02
Because truth be told, success does not lie in developing expertise or processes. Try more tools. There's no need to balance between exploring and exploiting the mechanisms that work for you.
- 16:14
Change the tools. And the processes and the decision-making frameworks don't matter. The right tool will solve everything.
- 16:23
Number two, find a machine learning engineer who can fine-tune as quickly as possible. A two thousand dollar per month OpenAI bill is very expensive. And instead, hire someone for a quarter of a million dollars, give them one percent of their company to fight CUDA build errors and figure out server cold starts, right?
- 16:41
Because what's the point of growing your company if you're just a wrapper? And if your margins are too low, try fine-tuning. It's much easier than figuring out how to build something worth charging for.
- 16:53
It's really-- I cannot, I cannot reiterate this enough. It's very important to hire a machine learning engineer as quickly as possible, right? Even if you have no data, generating products.
- 17:04
They love fixing Vercel TypeScript build errors. And generally,
- 17:10
if you hire a full stack engineer who's really caught the LLM bug, they, they're gonna lack real experience.
- 17:17
And this is because Python is a dead language, right? Machine learning engineers, research engineers can easily pick up TypeScript, and the ecosystem that exists in Python could be quickly re-implemented in a couple of weekends, right?
- 17:30
The people who wrote Python code over the past ten years doing data analysis, they're gonna easily be able to transition their tools. And if anything, it's really easy to teach things like product sense and data literacy to the JavaScript community.
- 17:45
And most important of all, in order to find this kind of magic talent, we need to create a very catch-all job title. Thus use words like ninja and wizard or data scientist or prompt engineer, or even the AI engineer.
- 18:00
In the past ten years, we've known that this works really well, right? Every time we know exactly who we want, as long as we catch a very wide net of skills, it doesn't really un- matter whether or not we don't know what outcomes we're looking for.
- 18:15
Anyways, to dig me out of this hole, I'll, uh, have Hamel explain and, uh, you know, take a deep breath, think out loud, step by step.
- 18:24
Thank you, Jason. [audience applauding] So that was really good. I mean, let's just step back from the cliff a little bit, and let's kind of linger on the topic of AI engineer.
- 18:40
I heard, heard some booing in the audience. Um, and so I love the term AI engineer. Like much props to Swyx for kind of popularizing this term. Allows us all to get together and have conversations like this.
- 18:53
But I think that there's a misunderstanding of the skills of AI engineer. What is-- what skills you need to be successful, and there's a lot of inflated expectations.
- 19:05
As a founder or engineering leader, the talent is the most important lever that you have.
- 19:11
And so what I'm gonna do is I'm gonna talk about some of the problems and perhaps some solutions when it comes to this talent, uh, misunderstanding.
- 19:21
So just to review, what is an AI engineer? So this is a diagram that everyone has probably seen. Uh, there's a spectrum of skills in the AI space, and there's this API dividing line in the middle.
- 19:33
And kind of to the right of the API dividing line, we have AI engineer. The AI engineer skills are focused on things like chains, agents, tooling, and infra. And auspiciously missing from the AI engineer are tools like evals and data.
- 19:52
And I think a lot of people have taken this diagram too literally and taken it to heart and say, "Hey, we don't really need to know about evals," for example.
- 20:02
The problem is, is that you can go from zero to one really fast. In fact, you can go to zero one faster than ever before with all the great tools out there just by using vibe checks and implementing the tools that we talked about.
- 20:16
However, without evals, you can't make progress. It quickly leads to stagnation because if you can't measure what you're doing, you can't make your system better, and you can't go beyond zero to one.
- 20:29
So what can we do about this? About this evals skill set and data literacy? So Jason and I have found that you can actually get really good at writing evals and data literacy with just four to six weeks of deliberate practice.
- 20:46
In fact, like very effective. And we think that these skills, evals and data, should be brought more into the core of AI engineer. And it really, it like helps solve this problem, and it's something that we see over and over again.
- 21:00
So the next thing I want to talk about is the AI engineer job title itself. And so vague job titles can be problematic. What we see over and over again in our consulting is that this kind of catch-all role have very inspla- inflated expectations.
- 21:19
Um, this-- anytime anything goes wrong with the AI, people look towards that role to fix it, and sometimes that role doesn't have all the skills they need to move forward.
- 21:28
And we've seen this before with the role of data scientists. Titles and names really matter.
- 21:36
Um, and what I wanna emphasize is I think AI engi-engineer is very aspirational, and you should keep learning, and it's a good thing to strive towards. But you need to have reasonable expectations.
- 21:48
And just to kind of bring it back to data science, we've seen this before in data science as well, where we had kind of a decade ago when this role was coined.
- 22:00
It was a unicorn that had all these skills, software engineering skills, statistics, math, domain expertise. And we found out as an industry that we had to unroll that into many other different roles, such as decision scientists, machine learning engineer, data engineer, so on and so forth.
- 22:17
And I think similar things may be happening with the role of AI engineer, and it's good to keep that in mind. And what I see, or what we both see in consulting, is that it's helpful to be more specific, to be more deliberate about what skills you need and at what time.
- 22:33
And depending on your maturity, it's very helpful to not only specify what the skills are, but what kinds of products you'll be working on. So these are some job titles from GitHub Copilot, um, that kind of are very specific about the skills you need at that time.
- 22:49
And really, it's important to hire the right talent at the right time on the maturity curve. So when you're first starting out, you only need application development, software engineering, and/or AI engineering to go from zero to one.
- 23:02
Then you need platform and data engineering to ca-capture that data. And then only after that, you should hire a machine learning engineer. Do not hire a machine learning engineer without having any data.
- 23:13
But again, you can get a lot more mileage out of your AI engineer with deliberate practice on evals and data. We usually find four to six weeks practice does the job.
- 23:24
So in recap, one of the biggest failure modes is talent. We think the AI engineer is often over-scoped but under-specified, but we can fix that by learning evals. Next, I wanna give it over to Shreya Shankar and Eugene Yan to talk about-- to dive into this evals and data literacy. [audience applauding]
- 23:52
Thanks. Question.
- 23:54
Thank you, Jason. Thank you, Hamel. Next up, Shreya and I are gonna share with you about the tactical aspects of building with LLMs in production, specifically evals, monitoring, and guardrails.
- 24:05
So here's a Hacker News quote: "How important evals are to the team is a differentiator between teams shipping out hot garbage and those building real products." I would agree.
- 24:16
I think here's an example of LLMs-- uh, of Apple's recent LLM, where they shared about how they actually collected seven hundred and fifty summaries of push notification and email sum-summarizations because these are datasets, they are representative of their actual use case.
- 24:32
So how do we build evals for our own products? Well, I think the same thing-- the simple thing is to just make it simpler. For example, if you're trying to extract product attributes from a product description, break it down into title, price, rating, and then you can just simple do-- simply do assertions.
- 24:50
Sim-similarly, for summarization, instead of trying to eval that amorphous blob of a summary, break it down into dimensions such as factual inconsistency, relevance, and informational density. And once you've done that, assertion-based tests can go a long way.
- 25:05
Are we extracting the correct price? Are we extracting the correct title? Or if you're doing natural language to SQL generation, is it using the expected table? Is it using the expected columns?
- 25:15
These are very simple to eval and reiterates what Hamel has mentioned about keeping it simple. Lastly,
- 25:23
assertions can do everything, but they can only go so far. So therefore, consider evaluator models. May, may be training a classifier for factual inconsistency or reward model for relevance.
- 25:34
This is easier if your evals are classification and regression based. But that said, I don't know how I feel about LLM-as-a-judge.
- 25:41
What do you mean you don't like LLM-as-a-judge? I, I personally am super bullish on LLM-as-a-judge, and I'm curious how many of you are exploring LLM-as-judge or have implemented it.
- 25:56
No.
- 25:56
Yeah. There's a judge right here. You wanna stand up?
- 25:59
No.
- 26:00
Actual ju- uh, LLM judge here. Yeah. Anyways, we're gonna go through some points on what to consider when deploying LLM-as-judge. First of all, there-- it's no brainer. LLM-as-judge is the most easy to prototype.
- 26:14
You just have to write a prompt to check for the criteria or metric that you want, and you can even align this towards your own preferences by providing few shot examples of good and bad for that criteria.
- 26:26
On the other hand, fine-tune models or LLMs that, you know, you have to collect a lot of data and set up a pipeline to train as your evaluator are not super easy to prototype and have a lot of upfront investment.
- 26:40
Yeah. But that said, LLM-as-a-judge is pretty difficult to align it to your specific criteria in the business. Who here has not had e-- Who here has not had any difficulty aligning the LLM-as-a-judge to your criteria?
- 26:53
No.
- 26:53
Anyone? Okay, we gotta talk later, Shreya. [laughs] Um, I think that if you just have a few hundred to a few thousand samples, it's very easy to fine-tune a simple model who can do it more precisely.
- 27:04
Second, if you wanna do LLM-as-a-judge and have it fairly precise, you sort of need to use chain of thought, and chain of thought is gonna be, I don't know, five to eight seconds long.
- 27:14
On the other hand, if you have a simple classifier or reward model, every request is maybe ten milliseconds long. That's two orders of magnitude lower and would improve throughput.
- 27:26
Next, we wanna think about technical debt. Okay? When we're implementing our validators in production, even if they run asynchronously or they run in the critical path, how much effort do we need to put in to keep these up to date?
- 27:39
With LLM-as-judge, if you don't make sure your few shot examples are dynamic or some way of making sure your judge kind of prompt aligns with your definition of good and bad, then you're toast.
- 27:52
And s- kind of-- The effect is not as pronounced for fine-tune models, but if you don't continually fine-tune your validators on new data, on new production data, then they will also be susceptible to drift.
- 28:04
So overall, when do you wanna use LLM-as-judge? It's honestly a resources question and where you are in your application development. If you're starting to prototype it, um, you need quick evals with minimal dev effort and need something...
- 28:19
You have low-ish volume of evals, start with LLM-as-a-judge and kind of invest in the infrastructure to align that over time. If you have more resources or you know that your product is gonna be sticky, go for a fine-tune model.
- 28:34
Next, I'm gonna talk about looking at the data. Eugene mentioned, you know, you should create evals on your custom or bespoke criteria, but how do you know what criteria you want?
- 28:44
Simple answer, look at your data. Great AI researchers, but we changed that to engineers. Great AI engineers look at their data. So how do we do this? The first question actually before how is when do you look at this?
- 28:58
I know people who never look at their data at all or people who look at it initially after deployment. Wrong answer. You wanna look at it regularly. I work with a startup that, you know, whenever they ship a new LLM agent, they create a new Slack channel with all of the agent's outputs that come in real time.
- 29:15
After a couple of weeks, they transition this to kind of daily batch jobs, um, and make sure that, you know, they're not running into errors that they didn't anticipate.
- 29:24
Second thing is what specifically are you looking for? You wanna find slices of the data that are pretty simple or easy to characterize in some way. For example, data that comes from a particular source or data that has a certain keyword or phrase or is about a certain topic, right?
- 29:39
Simply just saying all of these are bad, but having no way of characterizing them and then improving your pipeline based on that is not gonna help. Finally, some things to keep in mind throughout this whole kind of looking at your data experience is that your code base is very rapidly changing over time probably, your prompts, components of
- 29:58
the pipeline, and et cetera. So when you're inspecting traces, it's super helpful to be able to know, you know, what GitHub commit or what model version or prompt version did this correspond to.
- 30:08
I think this is one of the very successful things that traditional MLOps tools did, like MLflow, for example. They made it very easy to trace back, and then hopefully you could replay something.
- 30:18
Well-[laughs] I, I see the judge shaking his head, but [laughs]
- 30:22
great. Um, and finally, when using LLMs as APIs, pin model versions. Uh, LLM APIs are known to, you know, exhibit different behavior that is very hard to quantify for certain tasks.
- 30:35
So pin, you know, GPT-4 1106. Pin GPT-4.0, whatever it is that you're using.
- 30:43
So Shreya mentioned that we need to look at our data, but how do we look at our data all the time? I think the way to do this is via an automated guardrail.
- 30:50
Here's Brandolini's law adapted. The amount of energy to catch and fix defects is an order of magnitude larger than needed to produce it. And that's true. It's really easy to call an M- LLM API and just get something.
- 31:02
But how do we know if it's actually bad? I think it's really important that we do have some basic form of guardrails, and some of them are just table stakes.
- 31:10
Toxicity, personally identified in- information, copyright, and expected language. Now, you may imagine that this is pretty straightforward, but sometimes you don't actually have control over the context. For example, if someone's posting an ad on your English website that's in a different language, and you're asking your LLM to extract the attributes or to summarize it, you may be
- 31:29
surprised that for some non-zero proportion of the time, it actually in a different language. Similarly, hallucinations happen more often than we would like. Um, so imagine you're trying to summarize a movie based on the description.
- 31:42
You just have a description for the trailer. It may actually include spoilers because it's trying so hard to be helpful, but that's actually a bad user experience. So sometimes it will include information that's not in there.
- 31:53
Here's a tip. If we spend a little bit more time building reference-free evals, we can use them as guardrails. So reference-based evals are when we generate some kind of output, and we compare it to some ideal sample.
- 32:06
This is pretty expensive, and you actually have to collect all these goal samples. On the other hand, if we have these labels, we can train and evaluate the model and just compare it to the source document.
- 32:15
So for example, if you're comparing summarizations, we can just check if the summary entails or contradicts the source document, and now we have a summarization-- I mean, a hallucination eval.
- 32:26
So therefore, if we spend some time building reference-free evals once, we can use it to guardrail all new output.
- 32:32
Cool. Thanks, Eugene. So we're gonna wrap up the next minute or so on some high level, bird's-eye view, 2,000-foot view, whatever you wanna call it, takeaways. First off, how many of you remember this figure from this pretty seminal paper in MLOps that came out maybe 10 years ago?
- 32:52
2015, so nine years ago. Yeah. So I, I think this paper really communicated the idea that the model is a small part in when you're productionizing ML systems, right?
- 33:04
There's so much more around the model that you have to maintain over time. Data verification, uh, feature engineering, monitoring your infrastructure, et cetera. So you might be wondering, you know, we have LLMs.
- 33:17
Does any of this matter? Yeah? That was-- Yeah, I'm seeing s- few nods here. Absolutely. Um, when we have LLMs, it-- all of these, you know, tech debt principles still apply.
- 33:33
And you can even think of the exact mapping for every single component in here to the LLM equivalent. For example, maybe we don't have feature engineering pipelines, but, you know, cast in a new light, they-- it's RAG, right?
- 33:46
We're looking at context. We're trying to retrieve what's relevant, engineer that to, you know, not distract the LLM too much. We have a ton of experimentation around that. All of this is something that needs to be maintained over time, especially as models change under the hood.
- 33:59
Similarly, for data validation and verification, right, we have evals. We have guardrails that need to be deployed, right? It's not just simply wrap your, uh, model or GPT, um, in some software and ship it.
- 34:12
No. There's, like, a lot of investment that needs to happen around the model.
- 34:17
All right. So I'd like to end with this quote from Karpathy senpai. There's a large class of problems that are really easy to imagine and build demos for, but it's extremely hard to build products out of.
- 34:28
For example, Charles dug up this paper of the first car driven by a neural network. That was 1988.
- 34:37
Twenty-five years later, Andre Karpathy took his first demo drive of Waymo, 2013.
- 34:44
Ten years later, I hope all of you had a chance to try the Waymo. We got the first driverless-- We got a driverless permit for Waymo in San Francisco.
- 34:53
Maybe in a couple more years, we'll have it for the whole of California. The point is, going from demo to production takes time. That's all we had. Thank you.
- 35:03
Let's build. [upbeat music]