AI Engineer Summit 2023
Building Blocks for LLM Systems & Products
Read the talk
Building Blocks for LLM Systems & Products
Task-specific evaluations, carefully ranked context, factuality checks, and useful feedback turn model capabilities into products that can improve reliably.
From a talk by Eugene Yan
Before you start: Familiarity with LLM prompts and basic machine-learning evaluation will help; retrieval augmentation and factuality checks are explained as they appear.
Did that change actually improve the product?
You changed a prompt, added retrieval, or fine-tuned a model. How do you know whether the product improved? That question connects the four building blocks in Eugene Yan’s talk: evaluations, retrieval-augmented generation, guardrails, and collecting feedback. Evaluations are the foundation because they make the effects of the other changes visible.
In eval-driven development, task-specific tests guide what to build next. The same tests then become checks to run before deployment, helping catch regressions when a prompt or another part of the system changes. Yan points to OpenAI managers spending time writing and reviewing evaluations as an indication that this work deserves direct attention, not treatment as an afterthought.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What does a benchmark score measure?
Conventional machine learning offers relatively settled metrics: root mean square error for regression, precision and recall for classification, and nDCG for ranking. LLM evaluation introduces another variable between the task and the score: how the task is presented. MMLU, for example, tests knowledge and reasoning through multiple-choice questions spanning subjects such as computer science, mathematics, and US history. But a shared question set does not ensure a consistent evaluation procedure.
In Evaluating LLMs is a minefield, Arvind Narayanan and Sayash Kapoor ask whether an evaluation measures the model or the prompt’s ability to elicit the desired response. Anthropic’s Challenges in evaluating AI systems supplies a particularly small example: changing parentheses in multiple-choice formatting can change accuracy. Without a consistent procedure, apparently comparable benchmark scores can reflect differences in prompting as well as differences between models.
Reference quality creates a separate problem. In the summarization results Yan presents, generated summaries receive higher human ratings than the dataset’s reference summaries. His XSum example includes InstructGPT, rather than GPT-4. This comparison concerns the quality of existing references; it does not establish that models outperform human summarizers generally. A benchmark can become less informative when its reference answers no longer represent the quality the application needs.
Finally, even a carefully run benchmark may measure the wrong task. MMLU might be relevant to a college-level chatbot, but its availability does not make it a useful measure for every application. Evaluate the application on the work it is supposed to do. That is a different question from asking which model has the highest academic benchmark score.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start small and make the task measurable
There is no settled recipe for every LLM evaluation, but that need not prevent a useful first version. Yan cites Teknium starting a domain-expert evaluation with 40 questions. The example makes starting small concrete; it does not establish a sufficient sample size for every application. A narrow set of relevant cases can begin guiding development while the evaluation grows.
The next move is to simplify the task until success can be checked directly.
| Task | What to check |
|---|---|
| Content moderation | Precision and recall for the target category |
| SQL generation | Whether execution returns the expected result |
| JSON extraction | Whether keys and values match the expected output |
Moderation can be framed around specific targets such as toxicity or bias. SQL evaluation can inspect the returned result rather than merely whether the query looks plausible. JSON extraction can compare the actual structure and values with an expected answer. These tasks remain tractable because the evaluator has something concrete to check.
Open-ended dialogue is harder to reduce to an expected answer. A strong LLM may be needed to judge the output, but the evaluator then has its own cost. Yan cites Jerry’s report that running 60 evaluations with GPT-4 was expensive, without supplying a dollar amount. The choice of judge therefore affects how frequently the team can afford to evaluate changes.
Automated checks also leave room for direct inspection. Jonathan at Mosaic used a prompt that generated games for a three-year-old and a seven-year-old, then inspected the outputs across training epochs. That made qualities visible that the available evaluations did not capture. Manual inspection complements the test suite by revealing what the team has not yet learned to measure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An answer in the context can still be missed
Retrieval-augmented generation, or RAG, supplies knowledge as input context instead of relying entirely on what the model learned during training. Yan describes this as a cheaper and more precise way to supply new knowledge than continuously fine-tuning. Retrieving the right documents is difficult, but another problem remains after retrieval succeeds: the generator still has to use those documents correctly.
The question-answering experiment discussed in Lost in the Middle makes that distinction concrete. It uses historical Google queries and hand-annotated answers from Wikipedia. The presented setup supplies 20 documents of at most 100 tokens each, with one answer-bearing document and 19 distractors. That is at most 2,000 tokens of document content, not a limit on the complete prompt. The experiment changes the position of the answer-bearing document and measures how that affects answering.
In the result Yan presents, accuracy is highest when the answer-bearing document comes first and remains reasonably good when it comes last. In the middle, performance can fall below the baseline without retrieval augmentation. A larger context window therefore does not remove the need for good ranking: fitting the answer into the prompt is different from putting it where the model will use it effectively.
Yan reports roughly 75% accuracy with the answer-bearing document first in this 20-document question-answering experiment. That is a result for the presented setup, not a universal ceiling for RAG. It nevertheless exposes an important boundary: even ideal placement of the retrieved answer does not guarantee correct generation. Retrieval quality and the model’s use of evidence need to be evaluated separately.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When the context does not support a recommendation
A second RAG failure appears when the retrieved context does not support the requested answer. Yan supplies a list of 20 science-fiction movies as examples of movies he likes, then asks whether he would like Twilight. Twilight is a romantic fantasy involving a girl, a vampire, and a werewolf, so the preference evidence is a questionable fit. The prompt includes an explicit escape route: respond with not applicable when the supplied movie preferences do not support the recommendation.
The response first recognizes that Twilight belongs to a different genre and is not really science fiction. Then it draws a connection to E.T. through interspecies relationships. Recognizing a mismatch did not produce abstention; the model found a way to answer anyway. Yan connects this behavior to the pressure to be helpful: with fuzzy notions such as taste and relevance, a capable model can manufacture a plausible connection instead of acknowledging that the evidence is insufficient.
Search and recommendation systems already contain useful defenses. Rank the most relevant items highly, and apply a relevance threshold before passing context to the generator. For the Twilight example, Yan proposes measuring item distance and stopping before generation if the candidate is too far from the liked movies. This is a proposed filter, not a demonstrated threshold or a validated fix. Its purpose is to keep unsuitable evidence out of the prompt rather than depend entirely on the model to reject it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Check factual consistency one claim at a time
Production guardrails include harmful-content checks, such as the hate, harassment, and self-harm categories Yan points to in OpenAI’s moderation API. They also include factual consistency: whether an output is supported by its source. Unsupported claims can destroy user trust even when the output passes a content-safety check. A factuality guardrail can therefore also be understood as an evaluation for hallucination.
Summarization research offers a useful approach through natural language inference, or NLI. Given a premise and a hypothesis, the task distinguishes three relationships. For the premise “John likes all fruits,” Yan’s examples are:
| Hypothesis | Relationship | Reason |
|---|---|---|
| John likes apples. | Entailment | Supported by the premise |
| John eats apples daily. | Neutral | Not enough information |
| John dislikes apples. | Contradiction | Conflicts with the premise |
The neutral category matters: a statement can lack support without being contradicted by the source.
For summarization, use the source document as the premise and the summary’s claims as hypotheses. The granularity of this comparison matters. In Yan’s example, the final sentence of a summary is incorrect, but a single judgment over the whole document and summary misses the error. Checking at sentence level exposes the incorrect final sentence. The presented ablation reports better factual-inconsistency detection as document granularity becomes finer, moving from document to paragraph to sentence. The point is to prevent an otherwise plausible summary from hiding one unsupported claim.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Agreement and strong-model judging
Another approach uses repeated generation as a consistency signal:
- Generate several summaries from the same input document.
- Compare them with measures such as n-gram overlap or BERTScore.
- Treat substantial disagreement as a possible hallucination signal.
The intuition is that summaries grounded in the same document should share information, while unstable details may indicate unsupported generation. This remains a heuristic: agreement alone does not establish that the shared claims are supported by the source.
A strong LLM can also receive the source document and summary directly and return a score. That is conceptually simple, but running a capable model as a judge adds expense. Yan reports seeing simpler factual-consistency methods outperform LLM-based approaches at substantially lower cost. This is a reason to test simpler methods against the application’s needs, not a universal ordering of factuality evaluators.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A copied answer is not a correct answer
Collecting feedback closes the loop between product use and evaluation. It reveals what customers like and dislike, and it can supply examples for both evaluation and fine-tuning. Those datasets are transferable assets: models may change, while the accumulated record of useful and unsuccessful behavior remains valuable.
The challenge depends on how the feedback is collected. Explicit ratings are sparse. Yan asks how many audience members use ChatGPT, then how many click its thumbs-up or thumbs-down buttons. The much smaller response to the second question illustrates why adding rating buttons does not guarantee a useful volume of feedback. It is an audience demonstration, not a measured response rate.
Implicit feedback comes from ordinary product use and can be more plentiful, but its meaning is noisy. Clicking a copy-code button shows that someone wanted to take the snippet elsewhere; it does not show that the code works. Yan’s example supplies nrows to pandas.read_parquet. In the versioned pandas API relevant to this example, nrows is not a pandas-level row-limit parameter. Treating every copied snippet as positive feedback would therefore turn this erroneous answer into a positive training example.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make preference visible through normal use
GitHub Copilot offers an example of feedback embedded in the task itself. A developer writes a function signature or comments, receives a code suggestion, and can accept it, reject it, or move to another suggestion. These choices recur throughout the working day. They expose user decisions without requiring a separate trip to a feedback form, although acceptance still should not be mistaken for proof of correctness.
In the Midjourney interface shown in the talk, a prompt produces four candidate images. The user can rerun generation, request a variation using a V control, or upscale an image using a U control. Yan reads these actions as different strengths of preference:
| Action | Yan’s reward interpretation | Inferred preference |
|---|---|---|
| Rerun | Negative | None of these candidates works |
| Vary | Small positive | This candidate has potential |
| Upscale | Larger positive | This is an image to use |
These are interpretations of observable choices, not verified Midjourney training labels or reward weights. The design lesson is to make progress through the product also reveal what users prefer, while remembering that any implicit signal needs interpretation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Automate the checks and reuse what works
The operational starting point is modest: annotate 30 or 100 examples, then work out how to automate their evaluation. Run the checks as prompts, retrieval, and fine-tuning change. Manual inspection can remain a final check, but it cannot support every iteration at scale. Yan reports running tens of experiments each day and credits automated evaluations with making that pace possible. The value is both faster experimentation and a safer path to deployment.
The retrieval system does not need to start from scratch either. BM25, metadata fetching and matching, two-stage retrieval and ranking, and filtering already address the problem of selecting useful information. These established techniques aim to put the most relevant items first—the same property that matters when retrieved documents become model context.
The final building decision is how people encounter the model. Copilot puts assistance inside an IDE; ChatGPT makes it available through a chat window. Users can work in a familiar context without calling an API themselves. That interface also determines which choices become visible as feedback. UX therefore shapes both the immediate usefulness of the model and the evidence available to improve the product next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Seven production patterns spanning evaluations, retrieval, fine-tuning, caching, guardrails, UX, and feedback.
Annotated slides on prompt sensitivity, construct validity, and benchmark contamination.
Anthropic's account of practical difficulties in multiple-choice evaluations and broader model assessments.
Versioned API reference for reading Parquet files into pandas.
Further reading
Yan's slides and talking points, with links to supporting research.
- Lost in the MiddlePaper
Controlled experiments testing how answer position affects question answering over retrieved documents.
- SummaCPaper
Sentence-level natural language inference and aggregation for detecting inconsistent summaries.
- SelfCheckGPTPaper
A method that uses agreement across sampled generations as a hallucination signal.
Human evaluation of LLM summaries, benchmark references, and summaries commissioned from freelance writers.
Read the complete timestamped transcript
- 0:00
[upbeat music] Thank you.
- 0:15
Thank you, everyone. Um, I'm Eugene Yan, and today I want to share with you about some building blocks for LLM systems and products. Like many of you here, I'm trying to figure out how to effectively use these LLMs in production.
- 0:29
So a few months ago, to clarify my thinking, I wrote some patterns about building LLM systems and products, and the community seemed to like it. There's Jason asking for this to be seminar.
- 0:39
So here you go, Jason. [laughs] Today, I'm going to focus on four of those patterns. Evaluations, retrieval-augmented generation, guardrails, and collecting feedback.
- 0:50
All the slides will be made available after this talk, so I ask you to just focus. Buckle up, hang on tight, because we'll be going really fast. [laughs] All right, let's start with evals, or what I really consider the foundation of it all.
- 1:05
Why do we need evals? Well, evals help us understand if our prompt engineering, our retrieval augmentation, or our fine-tuning, is it doing anything at all, right? Consider eval-driven develop, uh, development, where evals guide how you build your system and product.
- 1:19
We can also think of evals as test cases, right? Where we run these evals before deploying any new changes. It makes us feel safe. And finally, if managers at OpenAI take the time to write evals or give feedback on them, you know it's pretty important.
- 1:38
But building evals is hard. Here are some things I've seen folk trip, trip up on. Um, firstly, we don't have a consistent approach to evals. If you think about more conventional machine learning, regression, we have root mean square error, classification, precision and recall, even ranking, nDCG.
- 1:54
All these metrics are f- pretty straightforward, and there's usually only one way to compute them. But what about for LLMs? Well, we have this benchmark whereby we write a prompt, there's a multiple choice question, we evaluate the model's ability to get it right.
- 2:07
MMLU is a example that's widely used, where it assesses LLMs on knowledge and reason ability, you know, computer science questions, math, US history, et cetera. But there's no consistent way to run MMLU.
- 2:20
Less than a week ago, Arvind and Suyash from Princeton, "Evaluating LLMs is a minefield." They ask, "Are we assessing prompt sensitivity? Are, are we assessing the LLM, or are we assessing our prompt to get the LLM to give us what we want?"
- 2:35
On the same day, Anthropic noted that the simple MCQ may not be as simple as it seems. Simple formatting ch- changes, such as different parentheses, lead to different changes in accuracy.
- 2:46
And no one is-- There's no consistent way to do it, to do this. As a result, it makes it really difficult to compare models based on these academic benchmarks.
- 2:55
Now, speaking of academic benchmarks, we may have outgrown some of them. For example, this task of summarization. On the top, you see the human evaluation scores on the reference summaries, and on the bottom, you see the evaluation scores for the automated summaries.
- 3:10
You don't have to go through all the numbers there, but the point is that all the numbers on the bottom are already higher than the numbers on top. Here's another one that's more recent on the XSum dataset, extreme summarization, where you see that all the human evaluation scores are lower than InstructGPT.
- 3:27
That-- And that's not even GPT-4. Now, finally, with all these benchmarks being so easily available, we sometimes forget to ask ourselves, "Hey, is it a fit for our task?"
- 3:37
If you think about it, does MMLU really apply to your task? Maybe if you are building a college-level chatbot, right? But here's Linus reminding us that we should be measuring our apps on our tasks and not just rely on academic evals.
- 3:54
So how do we do evals? Well, I think as an industry, we are still figuring it out. Bar pointed out it's the number one challenge out there, and we, we hear so many people talking about evals.
- 4:02
I think there are some tenets emerging. Firstly, I think we should build evals for a specific task, and it's okay to start small. It may seem daunting, but it's okay to start small.
- 4:12
How small? Well, here's Technim. You know, he releases a lot of open source, uh, models. He starts with an eval set of forty questions for his domain expert task.
- 4:21
Forty evals. That's all it takes, and it can go very far. Second, we should try to simplify the task as much as we can. You know, while LLMs are very flexible, I think we have better chance if we try to make it more specific.
- 4:34
For example, if you're using an LLM for content moderation task, you can fall back to simple precision and recall. How often is it catching toxicity? How often is it catching bias?
- 4:43
How often is it catching hallucination? Next, if it's something broader like writing SQL or extracting JSON, you know, you can try to run the SQL and see if it returns the expected result.
- 4:54
That's very deterministic. Or you can check the extracted JSON keys and check if the JSON keys and the values match what you expect. These are still fairly easy to evaluate because we have expected answers.
- 5:07
But if your task is more open-ended, such as dialogue, you may have to rely on a strong LLM to evaluate the output. However, this can be really expensive. Here's Jerry saying, you know, sixty evals, GPT-4, it costs him a lot.
- 5:21
Finally, even if you have automated evals, I think we shouldn't discount the value of eyeballing the output. Here's Jonathan from Mosaic. I don't believe that any of these evals capture what we care about.
- 5:33
They have a prompt to generate games for a [REDACTED:age] and a [REDACTED:age], and it was more effective for them to actually just eyeball the output as it trains through all the epochs.
- 5:44
Okay. That's it for evals. Now, retrieval-augmented generation. I don't think I have to convince you all here why we need retrieval-augmented generation, but, you know, it lets us add knowledge to our model as input context where we don't have to rely solely on the model's knowledge.
- 5:59
And second, it's far practical, right? It's cheaper and precise and continuously fine-tuning to add new knowledge.
- 6:06
But retrieving the right documents is really hard. Nonetheless, we have great speakers, Jerry and Anton, sharing about this topic tomorrow, so I won't go into the challenges of retrieval here.
- 6:16
Instead, I'd like to focus on the LLM side of things, right? And discuss some of the challenges that remain even if we have retrieval-augmented generation. The first of all is that LLMs can't really see all the documents you retrieve.
- 6:30
Here's an interesting experiment, right? The task is retrieval-augmented question and answering, you know, historical queries on Google and hand-annotated answers from Wikipedia. As part of the context, they provide twenty documents.
- 6:42
Each of these documents are at most one hundred tokens long, so that means two thousand tokens maximum. And one of these documents contain the answer, and the rest are simply distractors.
- 6:52
So the question they had was this: How would the position of the document containing the answer affect question answering? Now, some of you may have seen this before. Don't spoil it for the rest.
- 7:03
If the answer is in the first retrieved document, accuracies, accuracy is the highest. If it's in the last, accuracy is decent.
- 7:12
But if it's somewhere in the middle, it's actually worse accuracy than having no retrieval-augmented generation. So what does this mean? It means that even if context window sizes are growing, we shouldn't allow our retrieval to get ba- get worse, right?
- 7:30
Getting the most relevant documents to rank highly still matters, regardless of how big the context size is. And also, even if the answer is in the context and in the top position, accuracy is only seventy-five percent.
- 7:44
So that means even with perfect retrieval, you can still expect some mistakes.
- 7:49
Now, another gotcha is that LLMs can't really tell if the retrieved context is irrelevant. Here's a simple example. So here's, here are twenty top sci-fi movies, and you can think of these as movies that I like.
- 8:03
And I ask the LLM if I would like Twilight. So for folks not familiar with Twilight, you know, it's romantic fantasy, girl, vampire, werewolf, something like that. [laughing] [chuckles]
- 8:14
But I, I think I've never watched it before, so... [laughing]
- 8:17
But I have a really important instruction. If it doesn't think I would like Twilight because I've watched all these sci-fi movies, it should reply with not applicable. And this is pretty important in recommendations.
- 8:28
We don't wanna make bad recommendations. So here's what happened. First, it notes that Twilight is a different genre and not, not quite sci-fi, which is fantastic, right?
- 8:39
But then it suggests E.T. because of interspecies relationships. [laughing]
- 8:44
Um, I mean, I'm, I'm not sure how I feel about that. Uh. [laughing]
- 8:51
Yeah. I mean, how would you feel if you got this for a movie recommendation? The, the point is, these LLMs are so fine-tuned to be helpful, and it's really smart.
- 9:00
And they try their best to give an answer, but sometimes it's really hard to get them to say something that's not relevant, especially something that's fuzzy like this, right?
- 9:09
So how do we best address these limitations in RAG? Well, I think that there are a lot of great ideas in the field of information retrieval. Search and recommendations have been trying to figure out how to show the most relevant documents on top, and I think it worked really well.
- 9:23
And there's a lot that we can learn from them. Second, LLMs may not know that a retrieved document is irrelevant, right? I think it helps to include a threshold to exclude irrelevant documents.
- 9:34
So in the Twi- Twilight and sci-fi movie example, I bet we could do something like just measuring item distance between those two, and if it's too far, we don't go to the next step.
- 9:44
Next, guardrails. So guardrails are really important in production. We wanna make sure we, what we deploy is safe. What's safe? We can look at OpenAI's moderation API, hate, harassment, self-harm, all that good stuff.
- 9:58
But another thing that I also think about a lot is guardrails on factual consistency, or we call that hallucinations. I think it's really important so that you don't have trust bust- trust-busting experiences.
- 10:10
You can also think of these as evals for hallucination.
- 10:13
Fortunately [chuckles] or unfortunately, the field of summarization has been trying to tackle this for a very long time, and we can take a leaf from their playbook. So one approach to this is via the natural language inference task.
- 10:27
In a nutshell, given a premise and a hypothesis, we classify if the hypothesis is true or false. So given a premise, John likes all fruits, the hypothesis that John likes apples is true, therefore it's entailment.
- 10:39
Because there's not enough information to confirm if John eats apples daily, it's neutral. And finally, John dislikes apple is clearly false, therefore contradiction. Do you see how we can apply this to document summarization?
- 10:53
The premise is the document, and this hypothesis is the summary, and it just works. Now, when doing this though, it helps to apply at the sentence instead of the entire document level.
- 11:05
So in this example here, the last sentence in the summary is incorrect. So if we run the NLI task on the entire document and summary, it's gonna say that the entire summary is correct.
- 11:15
But if you run it at a sentence level, it's able to tell you that the last sentence in the summary is incorrect, and they include a, a really nice ablation study, right, where they inc- they check the granularity of the document.
- 11:27
As we got finer and finer from document to paragraph to sentence, the accuracy of detecting factual inconsistency goes up. That's pretty amazing. Now, another approach is sampling, right? And here's an example from ChatGP.
- 11:41
Given an input document, we generate a summary multiple times. Now, we check if those summaries are similar to each other, n-gram overlap, BERT score, et cetera. The assumption is that if the summaries are very different, it's probably means that they're not grounded on the context document and therefore likely hallucinating.
- 11:59
But if they're quite similar, you can s- you can assume that they are grounded effectively and therefore factual. And the final approach is asking a strong LLM. You know, conceptually it's simple.
- 12:09
Given an input document and summary, they get the LLM to return a summary score, and this LLM has to be pretty strong, and we have seen that strong LLMs are actually quite expensive.
- 12:18
Um, but in the case of factual consistency, I've seen similar, simple, simpler methods outperform LLM-basi- based approaches at a far lower cost. So try to keep things simple if you can.
- 12:32
Okay, now to close the loop, let's touch briefly about collecting feedback, and I'm gonna need audience, audience help here. So why is collecting feedback important? Because we want to understand what our customers like and don't like.
- 12:45
And then the magic thing here is that collecting feedback helps you build your evals and fine-tuning data set. New models come and go every day, but your evals and fine-tuning data set, that's your transferable asset that you can always use.
- 13:00
So, but collecting feedback from users is not as easy as it seems. So explicit feedback can be sparse. Sparse means very low in number, and explicit feedback is feedback we ask users for.
- 13:10
So here's a quick thought experiment. How many of you here use ChatGPT? Okay, I see a lot of you. How many of you here actually click the thumbs up and thumbs down button?
- 13:21
Accidentally. Accidentally. Okay, but these are the beta, beta testers, right? But you can see it's very small in number. So even if you include this thumbs up, thumbs down button, you may not be getting the, the feedback you expect.
- 13:32
Now, if the ex- if the issue with explicit feedback is sparsity, then the issue with implicit feedback is noise. So implicit feedback is the feedback you get as users organically use your product, right?
- 13:43
You don't have to ask them for feedback, but you get this feedback. So here's the same example. How often do you c- click the copy code button?
- 13:51
The rest of you just type it out like a madman? [laughs] Right. Okay. So, but does clicking the copy code button mean that the code is correct? In this case, no.
- 14:02
N rows is not a valid argument for pandas.read_parquet. But if we were to consider all code snippets that were copied as positive feedback, we would have a lot of bad data in our training.
- 14:14
So think about that. So how do we collect feedback? I don't have any good answers, but here are two apps I've seen do it really well. First one, GitHub Copilot or any kind of coding assistant, right?
- 14:23
For people not familiar with it, you type some functional signature, some comments, and it suggests code. You can either accept the code, reject the code, move on to the next suggestion.
- 14:33
We do this dozens of times a day. Imagine how much feedback they get from this, right? Here's a golden data set. Another example is Midjourney. For folks not familiar, Midjourney, you write a pro- you write a prompt, it suggests four images.
- 14:47
And then based on those images, you can either rerun the prompt, you can either vary the prompt, that's what the V stands for, or you can either upscale the image, uh, that's what the U stands for.
- 14:58
But do you know what an AI engineer sees?
- 15:02
Rerunning the prompt is negative reward, where the user doesn't like any of the images. Varying the image is a small positive reward, where the user is saying, "This one has potential, but tweak it slightly."
- 15:15
And choosing the upscale image is large positive reward, where the user likes it and just wants to use it. So think about this. Think about how we can build in this implicit feedback data flywheel into your products that you quickly understand what users like and don't like.
- 15:30
Oh, sorry. You can take your phone out.
- 15:33
All slides available after the talk. So that's all I wanted to share. If you remember anything from this talk, I hope it's th- these three things.
- 15:43
You need automated evals. You need automated evals. Just annotate thirty or a hundred examples and start from there, right? And then figure out how to automate it. It will help you iterate faster, right?
- 15:55
On your prompt engineering, on your retrieval augmentation, on your fine-tuning. Help you deploy safer. I mean, this is a huge conference of engineers. I don't, I don't think I have to explain to you the need for test- testing.
- 16:07
Eyeballing doesn't scale. It's good as a final vibe check, but it just doesn't scale. Every time you update the prompt, you just want to run your evals immediately, right?
- 16:15
I run hundreds of-- I run tens of experiments every day, and the only way I can do this is with automated evals. Second, reuse your existing systems as much as you can.
- 16:25
There's no need to reinvent the wheel. BM25, metadata fetching, uh, metadata matching can get you pretty far, and so do the techniques from recommendation systems, right? Two-stage retrieval and ranking, filtering, et cetera.
- 16:38
All these information retrieval techniques are optimized to rank the most, the most relevant items on top. So don't forget about them. And finally, UX plays a large role in LLM products.
- 16:51
I think that a big chunk of GitHub Po- Copilot and ChatGPT is UX. It allows you to use the LLMs in your context without calling an API. You can use an ID using a chat window.
- 17:03
Similarly, UX makes it far more effective for you to collect user feedback.
- 17:09
Okay. That's all I had. Thank you, and keep on building. [audience clapping] [upbeat music]