AI Engineer World's Fair 2025
Building Metrics That Actually Work — David Karam, Pi Labs
Read the talk
Building Metrics That Actually Work
Turn application requirements into inspectable scoring signals, calibrate them against user feedback, and reuse the resulting scorer for model comparisons, prompt changes, and response selection.
From a talk by David Karam
Before you start: Basic familiarity with LLM prompts, JSON, and Python is helpful; following the original hands-on exercises also requires a Google account and familiarity with Colab.
What counts as a correct answer?
A tax-research answer can be correct in several ways. A business question translated into SQL creates a similar problem: how do you define success, then improve the system without repeatedly writing unit tests, reviewing results, and starting again? That is the opening problem raised by an attendee in David Karam’s workshop. The difficulty begins before choosing an evaluation tool: the team must decide what it wants the tool to measure.
For agents, evaluation also supplies feedback from the world. An experiment needs an observable result that distinguishes success from failure; an agent needs the same thing. Yet each application brings its own subjective requirements and probabilistic behavior, making evaluation work difficult to reuse. One attendee estimates that testing consumed 30% of feature-development time in their earlier QA work, versus 80% for evaluation in their AI work. Those figures describe that attendee’s experience, but the underlying frustration is shared: building the evaluator can become most of the job.
Novel applications introduce another gap: there may be no real-world dataset to use as a baseline. Here, test examples and metrics are separate design problems. Synthetic data supplies cases to exercise; a metric checks what happened on those cases. Generating convincing examples does not establish that the checker is accurate. Another attendee has already tried AI-based judging and found its judgments unreliable enough to require manual review again.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make evaluation part of development
The presenters bring experience from more than a decade at Google, including work on Search, where stochastic behavior made quality measurement central to development. The recurring process was to establish a benchmark, improve until it stopped exposing useful differences, then move to a harder benchmark. Metrics were calibrated against human judgments and user data along the way. The workshop applies that methodology to language-model applications; it does not promise a single evaluation recipe that finishes the job.
Karam recalls roughly 300 Search metrics, challenging teams that think only in terms of a handful of generic scores. The point is the breadth of quality requirements an application can accumulate. Some teams first need to define metrics; others already have them and need to connect those scores to thumbs-up/down feedback. That connection carries evaluation out of the test suite and into the online product’s feedback loop.
The workshop’s shared Google Doc collects the exercise steps, code, spreadsheet, and slide deck; the presenters say it will remain available afterward. The accompanying Slack channel is workshop-metrics, intended for questions during and after the exercises.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with cheap signals, then add coverage
Live testing—trying the application, inspecting its responses, and changing prompts—can take a simple product surprisingly far. Multistep agents create more opportunities for failure, and eventually a developer needs checks that can run repeatedly without inspecting every trace. The available methods serve different purposes:
| Method | Useful contribution | Main constraint |
|---|---|---|
| Live testing | Fast discovery and prompt iteration | Requires direct inspection |
| Human evaluation | Expert judgment | Rater time and cost |
| Code checks | Verifiable properties | Only covers what can be specified |
| Natural-language judging | Semantic requirements | Consistency of judgments |
The presenters argue that generative decoder models, built to produce varied responses, can be awkward tools for consistent scoring. Their proposed next step is to combine multiple simpler signals into a scoring system.
Begin with five to ten easily derived signals that you have reason to associate with good results. Test the application, inspect failures, revise the application or its metrics, and measure again. The initial set does not need to anticipate every failure; it needs to make the next development decision more informed.
Karam treats these methods as a toolbox. Start with live testing and trace monitoring, then add sophistication as the system grows and manual inspection becomes impractical. Each additional technique has a cost and a return. A scoring system emerges by layering useful checks, rather than by replacing every earlier method at once.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put domain knowledge in the scorer
A trusted evaluator makes domain knowledge reusable. Once it can distinguish better and worse outputs, it can guide meta-prompts or optimizers such as DSPy, and filter synthetic data for fine-tuning or reinforcement learning. The scoring definition becomes an input to improvement, rather than merely a final acceptance test.
The same scorer can operate at inference time. Increase temperature, generate four or five candidate responses, score each one, and return the best. The presenter calls this online reinforcement learning; the procedure described is more specifically best-of-N sampling, which selects among responses without updating model parameters. It can change which answer reaches the user while leaving the prompt and model unchanged.
Generic helpfulness, harmfulness, and hallucination checks do not capture every product goal. A trip plan can satisfy basic guardrails and still be bland. If the application is supposed to make someone excited about a destination, that requirement needs a place in the evaluation. This is where application-specific judgment becomes valuable: the scorer must express what makes this particular product good.
The exercise therefore starts with a small scoring system and tests it against good, bad, and synthetic examples. Participants revise it in a copilot, then move it into a Python Colab notebook. A spreadsheet provides another interface for changing criteria and inspecting results without editing the whole workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Decompose subjective quality into inspectable signals
Search offers a useful analogy: score documents, then rank them to decide which result to show. That score combines signals such as popularity, title quality, content quality, feasibility, spam, and clickbait. Each signal answers a narrower question than whether the whole document is good. A scoring system for generated content can use the same decomposition.
At the bottom of the system, a signal may be an objective property checked with deterministic Python. Higher up, signals combine into a subjective judgment about the output. Keeping those layers visible makes the score inspectable: a developer can examine a failed criterion instead of trying to interpret a single opaque number from a complicated prompt.
Coverage matters as much as aggregation. Search cares about relevance, site quality, popularity, spam, and adult-content intent because each can affect the result a user should receive. Asking only whether an answer is helpful delegates the definition of helpfulness to the judge. Decomposition gives the application developer more control over that definition. Karam attributes lower variance and finer-grained analysis to measuring narrower, more objective properties. As failures reveal missing requirements, add signals that expose those failures.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a scorer for meeting summaries
The demonstration uses a meeting summarizer. Its input is a transcript of several people talking; its output is structured JSON containing a summary, action items, key insights, and a title. This is small enough to inspect directly, but it already mixes structural requirements with semantic ones.
The copilot can begin from three kinds of material:
- System prompt: Derive scoring dimensions from the application’s instructions.
- Explicit criteria: Start with requirements the developer has already identified.
- Input/output examples: Infer dimensions from concrete cases.
A reasoning model decomposes broad requirements and displays the resulting scoring artifacts beside the chat. Karam notes that the same iterative decomposition can also be attempted in a preferred chat interface. Asked whether the process can work backward from examples, the presenter shows an Example entry point for collections of cases; a few examples can also be pasted directly into the prompt. The resulting criteria are a starting point for revision.
Each dimension is a signal. One asks whether the output includes insights from the meeting. Others can contain editable Python. Natural-language questions go to specialized scoring models, while the code expresses properties that can be checked directly. The importance labels critical, major, and minor control how much dimensions matter. The presenter explains that the combination function can eventually be learned from examples; this exercise gives participants direct control over the weights.
After developing the scorer with synthetic examples, participants apply it to a separately labeled set of meeting summaries carrying real thumbs-up/down feedback. Agreement is not guaranteed: it depends on the criteria they built. Karam frames the distinction as calibrated versus uncalibrated metrics. Correlation analysis and confusion matrices help answer the operational question: when this scorer gives a high score, does that correspond to an output users consider good? Decomposition makes disagreements easier to investigate, but it does not remove the need to check them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Break examples deliberately
The first probe is to generate a synthetic example from the system prompt and score it dimension by dimension. Next, the presenter asks for a bad example. The copilot uses its accumulated context to create an output missing information, then generates another with broken JSON. These targeted defects let a developer check whether the expected dimension notices the failure.
The same interface can edit the checker: change Python, remove a dimension, revise a question, or add a requirement. The concrete addition in the demonstration is a title containing fewer than twenty words. The underlying structural checks are straightforward to express in Python:
python
import json
def structural_checks(output: str) -> dict[str, bool]:
try:
document = json.loads(output)
except json.JSONDecodeError:
return {"valid_json": False, "title_under_20_words": False}
title = document.get("title") if isinstance(document, dict) else None
return {
"valid_json": True,
"title_under_20_words": (
isinstance(title, str) and len(title.split()) < 20
),
}
output = '{"title": "Launch planning", "action_items": []}'
checks = structural_checks(output)
For this example, JSON validity and title length are separate signals. Neither establishes whether the summary preserved the meeting’s meaning; that requires other dimensions.
An audience question extends the idea to changing product scope: what happens when a trip planner also starts making bookings? Supply the changed requirements and new examples, then request dimensions that test the added behavior. The human-guided copilot is useful for this kind of inspection, while Colab can send larger datasets through longer-running jobs. The presenter describes alternating between those modes—periodically pasting in user examples to inspect the scorer, while clients also run substantial batch processes. The demo finishes this revision with the new title-length dimension in place.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compare scores with user feedback in Sheets
The next step exports the criteria into a copy of the prefilled workshop spreadsheet. The presenter briefly mentions an Unsloth reinforcement-learning integration as another destination for Pi Scorer, but the hands-on path here is Sheets: copy the criteria from the copilot and replace the sheet’s defaults.
The presenter describes the spreadsheet exercise as scoring about 120 examples with user thumbs-up/down labels. Its confusion matrix separates agreement on positive feedback, agreement on negative feedback, and the two directions of disagreement. Participants can revise dimensions and observe whether alignment improves.
The walkthrough makes the procedure concrete:
- Replace the default criteria with your own labels, questions, weights, and Python checks.
- Inspect the data sheet, which contains input/output pairs and recorded user feedback across mixed difficulty levels.
- Select the input/output ranges and invoke
Score selected rangesunderExtensions. - Inspect the resulting confusion matrix, then change criteria or deliberately damage an output to see how its score responds.
For example, corrupting JSON should affect the structural dimension. If a damaged response still looks acceptable in the aggregate, the dimension or its importance needs investigation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose where evaluation runs
An attendee asks how teams evaluate at scale: every example, or a subset? The presenter points to the Python Colab, SDK guidance, and batch workflows. Pi’s scorers are described as designed for low-latency online use, although the spoken latency estimate is too imprecise to establish a usable performance target.
For offline sets, mix easy, medium, and hard examples so the benchmark covers routine behavior as well as difficult cases. More extensive synthetic-data generation is left to follow-up documentation. For live applications, sample production logs or run scoring directly online. The presenter uses Google’s online spam checks and decision systems as examples, and identifies the cost of conventional LLM judging as an obstacle to applying it on every request. The placement of the scorer therefore depends on both the feedback needed and the cost of obtaining it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A scorer built to return numbers
Asked how the system differs from LLM-as-a-judge, the presenter starts with repeatability: Pi is described as returning the same score for identical inputs and remaining stable under small variations. The architectural explanation is bidirectional attention with a regression head, rather than autoregressive generation of score tokens. This also avoids the pattern of generating a score and then generating a post-hoc justification for it.
The presenter says the models were trained on billions of tokens dedicated to scoring varied content, including code. The intended interface is simple: supply a question and the data, receive an inspectable score. The scoring model is expected to understand the criterion without a separate cycle of judge-prompt tuning. These are the presenter’s descriptions of the system’s design and training.
Individual scores then feed an aggregation model described as an extension of a generalized additive model. Calibration uses thumbs-up/down examples to learn how signals should influence the result. Crucially, a signal’s effect can be asymmetric: spam can make the entire response unacceptable, while the absence of spam need not earn positive credit. This is richer than treating every successful check as another point in a sum.
The presenter attributes the scorer’s speed to denser embeddings and fewer parameters enabled by bidirectional attention. At the time of the workshop, the stated language support was English and a handful of other languages; broader multilingual support was forthcoming, and multimodal support remained on the roadmap.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry the same specification into Python
The final walkthrough moves into a prepared Colab that participants can finish independently. After the installation cells, the central object is the scoring specification. Copy it from the copilot’s code view into the notebook: it contains natural-language criteria with Python checks where needed. This is the definition to edit as requirements change, and the notebook can either be run cell by cell as supplied or modified.
The notebook loads the public Hugging Face dataset described in the workshop as containing the same thumbs-up/down examples used in Sheets. It runs the scorer, collects scores and predicted ratings, and constructs a corresponding confusion matrix. The displayed evaluation output puts the feedback, transcript, extracted output, score, and predicted rating together, making the Python workflow a continuation of the spreadsheet exercise.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compare models and catch prompt regressions
With the scorer established, the notebook uses it to compare generators. The presenter reports that the model labeled 2.5 scores slightly higher than 1.5 on this meeting-summary task. The model family and exact variants are not identified in the narration. The presenter attributes the small gap to the task’s limited reasoning demands, and suggests that a smaller model such as Claude Haiku would expose a larger difference. The model-comparison demonstration uses about ten examples and five models, generating responses and scoring them with the participant’s criteria. Its result is a task-specific comparison under that scorer.
Prompt comparisons reuse the same structure: hold the scoring specification and test set fixed, change the system prompt, and inspect whether scores decline. On ten examples, the deliberately bad prompt receives lower scores than the good prompt in the demonstration. The contrast is intentionally pronounced to show how the workflow detects regressions before a prompt change is adopted.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the score to select the response
The last exercise holds one meeting transcript fixed and varies the number of generated responses. The single-response baseline uses temperature 0.7. With more samples, the system produces several distinct summaries, ranks them using the scoring system built during the workshop, and returns the highest-scoring candidate.
The selection step is small enough to keep separate from both generation and scoring:
python
from collections.abc import Callable, Sequence
def select_best_summary(
transcript: str,
candidates: Sequence[str],
score: Callable[[str, str], float],
) -> tuple[str, float]:
if not candidates:
raise ValueError("At least one candidate is required")
scored = [
(candidate, score(transcript, candidate))
for candidate in candidates
]
return max(scored, key=lambda item: item[1])
All candidates receive the same transcript and scoring function. The operation chooses an existing response; it does not rewrite the summary or train the generator.
The presenter reports rising scores as sample count increases in this single-transcript exercise, interpreting the change as better response quality. The demonstrated measurement is the scorer’s own output, rather than a separate user-quality measurement. That makes calibration consequential: once a score determines which answer the user receives, its definition of quality becomes part of the application’s behavior. The prepared Colab leaves participants with this complete generate-score-select workflow to run and explore after the workshop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Framework for building and optimizing language-model programs against task-specific metrics.
Further reading
Configuration guide for evaluating input/output pairs with Pi scoring criteria. Requires a separate Pi API key.
Read the complete timestamped transcript
- 0:00
[upbeat music] Uh, we're a bit early, but I guess everyone's here, so we will get started.
- 0:19
Um, maybe we'll spend, uh, the first few minutes to get people a little oriented. And actually we, we are, we are quite curious, like, what brings people here.
- 0:30
Yeah, why, wh- why are you here? [laughs] [laughs]
- 0:33
Uh, what, what about evals, uh, did-- do you... Like, maybe by a show of hands, a few people, like, have you done evals before? Okay. And okay. H- have you struggled with them?
- 0:48
Is it-- Was it hard? Maybe by, again, show of hands.
- 0:50
Away from the speaker.
- 0:52
Um, are you-- Well, so people who have not done evals before, what bring, what brings you here?
- 1:00
Somebody? Anyone wants to volunteer?
- 1:03
Yeah, feel, feel free to raise your hand.
- 1:05
All right.
- 1:05
Sorry. Should I move away from the speaker?
- 1:09
A little bit maybe.
- 1:10
A little bit. All right, I'll stand on this side.
- 1:11
He's trying to understand, like, how to approach it better.
- 1:14
Oh, I, I see. Oh.
- 1:16
Okay, so, so trying to understand how to, how to approach evals. Um, for people who have tried before, what have they struggled with? What has been the... Go ahead.
- 1:28
I think the struggle for me is to define the metrics that... Because it's hard to... Like, I do, um, tax research equations. So first, it's hard to define what the correct answer is.
- 1:39
It can be many different ways. And two is, um, there are end-based business questions and that you can turn into SQL. And how do you pragmatically
- 1:51
improve it versus I design unit tests, and I review it, and I do it again, and I feel like that's still very, um, manual and labor-intensive to me.
- 2:01
Right.
- 2:03
Yeah.
- 2:03
Yeah, so the evals can be labor-intensive. Evals can be hard to, hard to set up. Um, they can be painful to get st- Oh. Get started with. Um, any other thoughts?
- 2:17
Go ahead.
- 2:19
Well, it's like machine learning is dependent upon training data. Agents are depending upon evals to provide feedback from the world to let them know whether or not they're getting it right.
- 2:30
And clearly, it's, it's a sophisticated and challenging problem. So, like, how to do science, you have to have experiments. This is like that. How do you experiment?
- 2:39
Right. So we are all-
- 2:40
I'm just saying it's super important.
- 2:41
Yeah, we are all getting pulled into the, the world of data science and machine learning in some ways, even if we, we don't want to. Go ahead.
- 2:50
Yeah, sorry. Uh, before engineering, I was a part of quality assurance team. And you know, all these years people wasn't happy that testing, testing takes, I don't know, thirty percent of feature development time.
- 3:03
And now with AI, I think that evaluation took eighty percent of feature development time, and [laughs] people even more not happy about it. And we are trying to find ways, uh, also to, to find the shortcuts, but it seems that it's not always possible because each case is so unique that you just can't to reuse some previous works,
- 3:26
and you have to create many things from scratch on different levels. And, and because of this unclear nature, pro- provable nature of, uh, evaluation, it's every time you spend such a way-
- 3:38
Speak up. The audience can't hear.
- 3:40
-to find problem, the more optimized way to do evaluation.
- 3:45
Right. So basically, um... You said speak up or speak low? Okay, speak up. Okay. [laughs] Um,
- 3:55
yeah, so, so basically, like, um, it's custom sub- subjective evals, like for your specific use case. You just copy-paste something that exists. Much like tests, we used to do testing.
- 4:07
Some people did testing, some didn't. But now for, for... in the world of AI, everyone has to do evals. Everyone is spending a lot of time doing evals. Uh, the best practices don't exist.
- 4:18
So I... Go ahead, please.
- 4:20
Well, uh, I was just going to add that I'm looking first to understand how synthetic data is playing a role there, and especially for novelty like type of problems where you don't have access to real, like real world data to use as a baseline.
- 4:33
So how much of this can impact the eval system?
- 4:37
Right. Uh, there are like two aspects of evals. There's the synthetic data, and there are the metrics. Metrics are checking, synthetic data is testing it. Uh, how-- what does synthetic data do?
- 4:49
How do you generate good synthetic data to test and evaluate? Um, go ahead.
- 4:55
Mine goes along with the first comment. We've been trying local prompt on LLM models and use AI to evaluate, but in the end, it's just not accurate. So we have to do it manually and so hoping to learn then what are some acceptable ways to-
- 5:11
Right
- 5:11
...[inaudible].
- 5:13
Right. Like, uh, au- automatic evals are, are something everybody desires because you don't want, like human beings to sit and do it. Uh, but the options that are, are somewhat limited.
- 5:26
Typically, for most things, you can lean on AI to do human tasks. For, for some reason, evals tend to be hard for LLMs to do right. Um, so I guess that's, that makes a lot of sense.
- 5:39
I-- we're not gonna solve all your eval problems here. [laughs]
- 5:43
I think this is a, on a very hard research area that we would probably continue working on in the next few months. Um, but David and I were, were at Google for, for over a decade, and, uh, we have been doing evals for, for a long time.
- 5:59
Uh, we, we've dealt with stochastic application. We both search. We were building search. Search was also similarly stochastic, and most of, uh, the core effort in Google was to do a good job of evaluating where we are and improving it.
- 6:14
Um- And some of the techniques we learned at Google, we are bringing in here. Um, so we, we built a product around it, a bunch of technology around it, but really the biggest takeaway I want people to have is the methodology and the stuff that we have learnt, uh, whatever manifestation it takes, right?
- 6:32
So I'm hoping that you get to play with some of our stuff, but also to learn some, some good ideas. Um, David, do you wanna add?
- 6:42
Yeah, uh, I mean, one, one thing I would add is at, at Google we used to call this quality and evals was part of it, and there was this constant, uh, idea like benchmarking and, you know, exhausting a benchmark and then moving on to the next benchmark.
- 6:53
Whenever we work with clients right now, we, we take a similar approach. So like that Chin meant to- mentioned, so much of evals is just good methodology, setting up benchmarks, trying to figure out the metrics that work, calibrating metrics with humans, calibrating metrics with user data.
- 7:05
Uh, what I'm trying to say is that it can get arbitrarily complex. Like evals is not like, oh, there's one way to do it and then you're done. It's, it's just part of how you do development, uh, which, you know, everybody's been struggling with because we've all been trying to learn what it means to develop on top
- 7:16
of this new stack. But again, today, in today's session, we're just hoping to bring you some of these ideas where like methodologies that make sense, like how to think about your metrics.
- 7:24
I mean, people think of like four metrics. So I would challenge you, Search had three hundred metrics. So like you start to think about like how can you expand the scope of how you're doing this work?
- 7:31
Uh, and you know, part of what we're doing is trying to build technologies that make it easy to adopt these, these technologies, how to create benchmarks that are interesting, how to like make it harder, um, and then we can talk about, about this as we go.
- 7:43
Uh, th- this workshop will be mostly hands-on. Uh, we'll show a few slides and then we'll dive into the code 'cause I think the best way to, to learn about these things is just to, to do them.
- 7:51
Um, I think as we go, we'd love to just pass it around and, you know, if people have, you know, again, uh, people are at different parts of journeys.
- 7:58
For some people it's like, "Hey, I don't have metrics and I'll just develop my own metrics." Uh, we worked with clients that have metrics, but they wanna, for example, correlate them to user behavior.
- 8:06
They have a lot of thumbs up, thumbs down data. Um, so that goes all the way into a feedback loop. So you see like evals is not just one thing related to testing.
- 8:13
It goes all the way into your online system and your feedback loops and all that. So but hopefully a lot of the mental models today will help you kinda gauge that.
- 8:20
Um, just two, two ish things. The Slack channel is, uh, workshop-metrics, uh, so you can join there on the Slack and then we'll just be posting things and then we can have discussions and continue the conversations, uh, even after the, the, the workshop.
- 8:33
And, uh, the second one is, uh, we have a document that will have all the steps, uh, of the workshop, uh, all the places where you can get the code, where you can get the sheet.
- 8:43
Just go withpi.ai/workshop. You will land in a Google Doc, uh, so there's a lot of links. The Google Doc also has a link to the slide deck we're presenting so you can have it.
- 8:52
We'll keep it online even after the workshop so you guys can reference it. Um, all right, we'll get started to, to keep you guys on time and maximize sort of coding time.
- 9:01
Yeah, I, um, there are four of us. A couple o- other people will just walk around if you guys get stuck on any step, if you're having trouble with anything-
- 9:07
Yeah
- 9:08
... just raise your hands and then we'll come and-
- 9:10
Just raise your hand
- 9:10
... come over. Um, all right, I'll give you a very like maybe five-minute, uh, quick blurb, show a quick demo, and then get started. Um,
- 9:21
so can you guys still hear? Okay. I think we went, we went through this, but basically like most people start with live testing. That's like, you know, try it out, see how it works, change prompts, and honestly for many applications you can go pretty far with just that.
- 9:37
Like I think AI, AIs have become much quite good. I think agent systems as you, uh, you were saying, agent systems are, are more complex because of multiple steps, things fail more often.
- 9:49
But live testing gets you pretty far. Human evals are expensive. Typically, most companies don't bother setting up human rater evals. Some do. Some have subject mata- matter experts. Uh, code-based evals is where I think majority of the people are spending time.
- 10:04
They're writing some sort of a code to test some verifiable things that they can and, and people are moving into natural language, like LLM as a judge type things where it gets...
- 10:16
This is not a very good task. We'll get into some of it like for, for t- typical decoder models, the AI models, and so you kinda have to fight against, um, the, the, these models are designed to be creative.
- 10:27
They're designed to be, um, that's not what you want from a judge, uh, typically. The s- the scoring system, uh, the scoring system is this idea that you, you, you're not trying hard to build a comprehensive set of metrics from the beginning.
- 10:43
You start with some correlated signals. Maybe you start with five, 10 signals that you know for a fact are correlated with goodness. They're very simple signals you can easily derive, and then time you build upon those as you see problems, as you s- as you debug.
- 11:01
And, and then you test your app, works well or not. You learn from it. You build your application. You measure it again. So that grip makes it a much more of a feedback loop type process.
- 11:13
Uh, that's the thing that we're trying to introduce in terms of methodology.
- 11:17
Uh, the thing that David was saying-
- 11:19
Uh, maybe-
- 11:19
Go ahead.
- 11:19
Yeah. One, one thing I will add to that is like there's no right or wrong. Like live testing actually gets you a very long way. Um, sorry, we, we have a lot of feed echo.
- 11:27
Um, maybe you wanna turn that off and then we can switch the mic as well right now. Let's do that. Yeah. So ju- just to say like it, it's, there's no right or wrong.
- 11:35
I think you should absolutely start with live testing. A lot of people use tracing and they just ma- monitor traces. I think a- as your system scales, you do wanna get a little bit more sophisticated.
- 11:44
Uh, so you, you start to layer those things in order of complexity. That's a lot of how we talk about quality generally is this idea of like there's complex things, but they give some amount of investment, uh, of some amount of return.
- 11:55
So it's like a little bit of an ROI. Some things are very cheap to do and then you do them, and as your system scales they become impractical and then you just layer in more techniques.
- 12:02
So there's not a statement that like any of these things are good or bad. You just need all of them. So think of these things as tools in your toolbox.
- 12:08
But eventually what you really want is a scoring system. Now how it manifests for you, you should, you should be the judge of it. But like th- this is one thing to get away with, like just keep laying into your tools and with increasing levels of sophistication you get probably to a system that will look like the
- 12:22
one we'll develop today. Um-
- 12:26
Yeah, and a-as David was saying, like, um, e-evals are such a, such an investment, so they might, might ask like, "Well, why am I spending so much time on just evals?
- 12:34
Why am I not building?" And I think one of the things that, that was core to, to Google, and I think we-- this industry is going to adopt this over time, is that evals are actually the only place you're going to spend most of your time because that's where domain knowledge is going to live.
- 12:48
Everything else will just work off of those evals. And so, um, so for example, if you have really good evals, you don't have to write prompts. You can write...
- 12:58
Like, you can find problems and use meta prompts or optimizers like DD, DS Pi and others to improve your prompts yourself. You can filter out synthetic data and then use that for fine-tuning if you're really interested in fine-tuning or, or reinforcement learning.
- 13:12
But you can also use these techniques online. Like, one of the things that you're gonna test to try today is a very simple but very effective technique that almost all big labs use, um, Google used it extensively, which is the, the idea that you crank up the temperature, you generate a bunch of responses instead of just one
- 13:29
response. You can think of this as online reinforcement learning. Generate four or five responses, and then score those responses online and see which one's the best one. And you get a, a pretty decent lift just by doing this, like, without actually doing any changes to your prompts or models and so on.
- 13:44
So that-- these are the kinds of things you can do, uh, with-- when, when you have a really good scoring system that you can lean on. So you get to try some of these techniques today.
- 13:53
Um, but, but the key point is, don't think of evals as testing in the classic sense. Think of these as the primary place where domain knowledge lives. Um, and then, and then one of, one of the things like that I, I think, um, one of the attendees was pointing out is that t- right now, at this point
- 14:11
in the AI industry, we have figured out a handful of standard evals, like you can think of simple helpfulness, harmfulness, hallucinations, and that doesn't really get you far enough.
- 14:23
That's good enough to just sort of do a guardrail and make sure you're not doing anything wrong. But we're, we're moving into the world where we wanna build really good...
- 14:31
Like, for example, if you're trying to build a trip plan, you, you, you're curious about like, you know, how, how to make that trip plan really be perfect. Like, one of the things I, I really hate about trip plans, if they're too-- like, not interesting enough.
- 14:43
I'm looking at a trip plan, I wanna be excited about this place. Typically, when LLMs give me a trip plan, it's a very kinda, a very plan-ish, right? So now you're trying to build these sort of nuances in your applications, and that's the kind of stuff if you wanna evaluate.
- 14:57
That's where, that's where the industry is gonna go. So how do you build like these much more nuanced evals? Uh, that, that's one of the places where these traditional evals fail.
- 15:06
So as, as I went over, like start simple, iterate, see what's broken, and then improve, right? And so that's the other thing you get to try today is in a copilot-like setting, start with something, test it out with a handful of examples, maybe s- generate some synthetic examples, test it out with good examples, bad examples, see what's
- 15:27
working, what's not, and then iterate. We have picked a relatively simple example for today because for workshop purposes, we wanted to keep it simple. But you can absolutely go and try it on fairly complex things, and, and there you can see a lot more of the nuances.
- 15:41
Um, so yeah. So there are two parts of today's workshop. The first part is, um, setting up your scoring system, and the second part is once you've set up the scoring system and played with it and iterated on it in a copilot, trying to use it in a colab.
- 15:58
You would need a Google account and some, some proficiency in working with colabs py-- and Python code. Um, we'll also introduce a spreadsheet component to this so that you can actually try to play around with this in a spreadsheet, um, so that y-you can easily make changes and test things out.
- 16:18
Uh, one last thing I wanna hit on before we jump into the, the, the workshop is what, what is the scoring system? What is this idea of scoring system, right?
- 16:28
Like, so just to give you a mental framing of it, um, s- ranking is scoring is evals, effectively. Um, well, that's one way to think about it. When Google does a search, what it's trying to do is it's scoring every document and seeing whether this is good or not, and then giving you the best document that's the
- 16:47
best for you. And that's not that different from ch-checking and scoring an LLM-generated content. And the way Google does it is by breaking this problem down into a ton of signals, right?
- 16:59
So you can imagine you're looking at SEO, you can look at document popularity, you're looking at title scores, whether the content is good or not, um, maybe feasibility of things, um, spam, you know, clickbaitiness, stuff like that, and then brings all of these signals together into a single score that combines these, these ideas.
- 17:17
Individual signals that you have are very easy to understand. You know what that is. You can inspect it very easily. It's not a complex prompt with some random score.
- 17:25
It's something that you can easily understand. But it all comes together into a single score. And that's the idea we are sort of bringing in. That's what we call a scoring system.
- 17:34
At the bottom level, things are very objective, tend to be deterministic, sometimes just Python code. But as you bring this up to the top, it becomes fairly subjective, and you can like bring these together into very complex ways.
- 17:47
Yeah. And I would just, uh, encourage you to think about like wh-why this, why, why this kinda-- why this could work, and, and we know it does work. Um- Some of what you will be struggling with with evals is just, like, you're not measuring the things you need to be measuring, so you have a little bit of
- 18:01
a comprehensive initi-issue. And so, like, the question is, like, how many metrics do you need to add? So I give the example, Google Search uses around three hundred signals.
- 18:08
Now, maybe you don't wanna be that sophisticated, but, like, Google does care about all those things. They care about, like, you know, the score of the site, they care about popularity, they care about content relevance, spam, porn seeking.
- 18:17
Well, they have all these classifiers, all these ways that they understand the content to then bring it together. What this gives you is, like, really visibility over your application and more and more ways to marry your own judgment into it.
- 18:27
So if, if instead you just go and say, "Hey, is this a helpful response?" Mostly what you're doing is delegating that, that eval to the LLM itself, right? Or to a rater who you're asking, like, "Hey, see if this is good."
- 18:38
But when you break this all down, you get some really nice properties where, like, your variance goes down a lot just because you're measuring way more objective things, so things are not, like, going back and forth all the time.
- 18:48
And it's very precise because, like, you're-- you get all these things and you add them together to a more high-fidelity score. And when you are analyzing the data, then you can, like, slice and dice by way finer grain things.
- 18:59
And that's kinda like why the system tends to work much better. And the best part of it is, as you iterate, you just add more signals. Like when... Th-this now doesn't leave you as like, "Oh, I either have evals or I don't have evals."
- 19:09
But, like, rather you just have a set of metrics that you keep adding over, over time as you discover what actually matters about your app. Oh.
- 19:26
Okay, uh, how do I do this?
- 19:30
Here, I'll hold it. [laughs] [laughs]
- 19:33
Um, okay, so I'll just quickly show you a demo of where you're gonna start today. Uh, give you some sort of basic ideas of what the kinds of things you would be doing, uh, and then, and then I'll just...
- 19:43
we, we, we should just get started, right? And so this is, uh, this is a copilot that helps you put together evalu-evals. That's the idea. Where I'm starting is basically just a system prompt.
- 19:53
This is... This application is basically a simple meeting summarizer. It takes, uh, a meeting script, talk-talking between multiple people, and then generates some sort of a structured JSON in the end, which is a summary with very specific, um, action items, key, key, uh, key insights, and then a title, right?
- 20:13
So it's a relatively simple thing. It's easy for you to inspect and see where things are not going, working and not working well. Um, so typically you would start with something like a system prompt.
- 20:22
You can also start with examples if you have a bunch of examples, or you can start with criteria itself. Um, and the, the first step is it would try to use this to build your scoring system.
- 20:36
Um...
- 20:39
Yeah. And, uh, this is doing exactly what was in that slide, which is trying to say, like, from that coarse-grained subjective thing, what are all the smaller things I can, I can suss out of, out of it?
- 20:49
Now, this uses a reasoning model. Like, if you drop this into a ChatGPT or so, uh, we just try to replicate that experience where you get these artifacts on the right-hand side, which are your...
- 20:57
is your actual scoring system. But if you think you just wanna do it, like, iteratively through your favorite, like, sort of chat interface, you can also do it. Just say...
- 21:05
Oops.
- 21:05
Go ahead.
- 21:06
Can we do it in reverse where we provide examples of input and output and ask it to score it?
- 21:10
Yes, exactly right. So right in front there is an Example button. You can start with example. You can give it actually hundreds of examples if you want, twenty examples.
- 21:20
Uh, and then it gets into a much more complex process of, like, figuring out these dimensions based on example. Or you can just copy-paste one or two examples in the prompt itself, and it'll, it'll generate it.
- 21:29
Um, the... this is just a starting point, by the way. That's the idea. It, it starts you somewhere, and now you're gonna iterate over it, and that's the exercise you will, you will spend time on.
- 21:38
I'll show you few things. Like, this is your scoring system. These are your individual, um, these are individual dimensions, is what we call it, or you can think of these as signals.
- 21:49
Uh, they're all questions, um, effectively. For example, this is a-- does the output include any insights from the meeting? That's a natural language question. Um, or you can a-have code, um, just Python code that we have generated.
- 22:04
You can edit this code however you want. Uh, y- this, this, uh, this, this is actually as simple as what this, this says. It's... When you, when you look at the actual code, the code is effectively just a bunch of questions, and you're sending these questions to our specialized foundation models that are designed for scoring and evaluation.
- 22:24
Um, so you'll get to play with this a bunch in the Colab. You can see what the form of it looks like. There's another thing that you would notice, that there's this idea of critical, major, and minor.
- 22:33
These are just weights. These are just ways-- ways for you to control what's important and what's not. The combination of this is done through, uh, a mathematical function that, that you can learn over time.
- 22:42
So you actually, eventually you would give it a bunch of examples and it'll learn it. But in this particular exercise, you have a little bit more control. And finally, once you have your scoring system, uh, done, uh, there is a way for you to integrate it into Google Sheets.
- 22:56
That's the thing that you're gonna play around with today, which is basically taking this criteria, moving it to a Google Sheet, and testing it against real examples. So here you're gonna work with synthetic examples.
- 23:07
You'll develop your scoring system, and then completely blinded to this, we have labeled dataset where users are set thumbs up, thumbs down on the summaries. You're gonna apply this, that scoring system and see how well it aligns with real, real thumbs up, thumbs down.
- 23:21
Um, so we don't know. Uh, it depends on you. You, you're, you're building your own scoring system, whether it aligns or not. Um...
- 23:28
Yeah. And, and th-this is a really interesting point and w-why evals start to get hard. Like, we, we call this workshop like solving the hardest challenge, which is metrics that actually work.
- 23:36
So this idea of, like, correlation ends up being really, really important. Metrics that work are not necessarily good metrics or bad metrics. They're either calibrated metrics or uncalibrated metrics.
- 23:45
And this is where, like, uh, at Google, for example, we had a lot of data scientists that we worked with, right? Because they were doing, like, all these correlation analyses and confusion matrices and such.
- 23:54
So pa-part of the challenge of, of, of good evals is just getting comfortable with the numerical aspect of these things. And of course, again, having a scoring system that dissects things into much simpler things makes it easier to analyze.
- 24:06
But you still have to think about those things, like does this actually correlate with goodness? Like, if it gives a high score, is this an actually a good score?
- 24:12
And that's a big part of the methodology. But the good news is, as, as Achin showed in the previous slide, once you have metrics you can trust, uh, like almost all of the rest of your stack gets radically simplified as a result.
- 24:24
Okay, so how do you use this copilot? It's created this. Maybe a place to start would be, you know, generate an example. This is a synthetic generation, uh,
- 24:34
um, happening behind the scenes. It's taking, uh, your system prompt and other information and trying to generate, uh, some sort of an example. And, and then, and then this example that is generated is scored, and you can see how these individual scores work.
- 24:49
Now you can start, uh, kind of testing this a little bit more. You can say, um, "Can you generate a bad example?" Um, and then it will, it will try to generate an example that's broken in some particular way.
- 25:03
It-- The, the copilot understands what you've done so far. It has the full context. It's using all of this information to kind of like sort of walk you through this.
- 25:11
So in this particular way-- like in this particular case, it created something that is missing a bunch of information. But you can even do things like, you know, specific things like, "Can you create, uh, an example that has broken JSON?"
- 25:27
Uh, so this is like basically example generation. That's one, one thing you can do here. The other thing you can do is you can make changes to your scoring system itself through the copilot.
- 25:39
So you can either go and ask the copilot to make changes to your, um... Anyway, this is a broken JSON example. You can go to the copilot and either ask it to change the Python code itself.
- 25:50
You can say, you know, "Can you update the Python code or-- and, and such of any of these things?" You can also ask it to remove or add dimens-dimensions.
- 25:57
Maybe you can say, "Can you generate a dimension that checks for the title to be less than twenty words?" And then the copilot can add these dimensions.
- 26:14
Go ahead. So let's say we add a feature to the product, the, the trip planner that also now does bookings for you. Uh-huh. So are you then adding that into the- Exactly ...
- 26:25
model as like a higher level? So now it's like more of an agent- Yes ... rather than that. Exactly. And, and what do you do? Do you just add it to your- You just ask the copilot and say, "I've updated.
- 26:34
Here's some, like, new examples. Can you add new dimensions to test these new examples?" Or, "We have changed things," and it automatically... I mean- It knows what layer? It knows what layer you're...
- 26:44
Exactly. Layer. Yeah. The other thing I would say is the copilot is very helpful if you want this sort of human-in-the-loop type process. Uh, but once you get comfortable with the system, what we see people doing is just use Colab to send large amounts of data and let our system figure things out on its own.
- 27:04
Uh, I, I find-- By personally, I find it very good to play with my scoring system here every so often to see if it's like working well, like maybe paste an example from, from a user and see how well it's working, and so on and so forth, so you can kinda go back and forth.
- 27:18
Uh, but, but most of our clients actually just fire off these, uh, long, long-running processes. Um. Yeah, I think maybe you wanna- So just- ... spreadsheets. Um, so anyway, so that's, uh...
- 27:31
So it just created this new, uh, title length thing for you. So you can like sort of play around with this, um, and it'll-- you can change the questions, you can remove dimensions, so on and so forth, right?
- 27:41
The last thing I would show before we get going is, uh, one of the things that you would do is we, we, we have pre-filled a spreadsheet in the, in the, in the workshop directions.
- 27:52
You will make a copy of that spreadsheet, and we have a spreadsheet integration where you can run our score inside a spreadsheet itself. You see there are other, other sort of places where you can use it.
- 28:03
I'm not gonna get into this, but you can use it for reinforcement learning, for example, using onslaught integration. But there are other places you can actually integrate, uh, with PyScorer, uh, if...
- 28:13
Not sure if you're familiar with these. But basically, in the, in the Sheets integration, what you are going to do in this workshop is you can actually just copy this wholesale and put it into a new spreadsheet with the examples that are here.
- 28:26
What we are trying to do is just copy the criteria. So you're gonna build a criteria. Here's a copy signal, uh, s-sim, uh, icon. Just click on the copy icon and then go to the spreadsheet that we will-- which, which you have, replace the criteria that exist there with this criteria that you've created.
- 28:43
And then under Extensions, these directions are all in the, in the docs. I don't have to remember it. Under Extensions, you can go and call the scorer. The scorer's gonna run across about a hundred and twenty examples, and then you'll see a confusion matrix, which shows you how many times there's alignment on thumbs up, how many times
- 29:00
there's alignment on thumbs down, and how many times there's no alignment. And then you can play around in the spreadsheet itself, make changes to the dimensions, and see if you can bring the alignment closer.
- 29:09
So this gives you a sense of, um... Yeah, so this is the-- this is what a spreadsheet would look like. This is your criteria, um, which is basically just the English form, uh, the, the spreadsheet form of the English that you saw.
- 29:22
Can you zoom in? Oh, sorry. Is that better? Yeah. Okay. So like you can see, this is the label, the question. These are the weights. In the case of Python, there's Python code.
- 29:34
So this is what your criteria sheet looks like right now. This is the default one that we put in there for you, but you would replace it with your own.
- 29:41
This is the data. This, this basically has actually feedback, which is a thumbs up, thumbs down, which we got from users. Um, some of this is fairly complex, some of this is easy, so it's all kinds of different sort of mix of things.
- 29:52
And what you're gonna do is you're gonna select these two rows. This is the input, the output. And then in the extensions, uh, under here, you will have score selected ranges, which is gonna create a score.
- 30:02
And then you can look at the confusion matrix and see how it works. So that's sort of how... That's one part of the exercise. But you can easily go and make changes here or even test out by making changes to the data, you know, messing up your like, your JSON and seeing whether how that impacts things, and
- 30:18
so on and so forth. So that's, that's the first phase of our copilot, the, as of, of, of our workshop, and we'll talk about the second phase when we get started with that one.
- 30:26
Go ahead. Yeah, I'm just-
- 30:34
Examples or do teams run this on a subset or just, like, how do you do this at scale?
- 30:41
Right. So that, we'll, we'll hit a lot of that in the second part of our workshop. We will directly go into Python Colab code. Uh, we have, um... The, the SDK goes through a bunch of this, um, the details and best practices on how you do it.
- 30:57
But you will get to play with this o- uh, in, in this workshop. You, you c- you will... Our, our, our scorers are specifically designed for online workflows. These, like, twenty dimensions that you have, they score them all in sub twenty, like, fifty milliseconds.
- 31:13
So you can run on a very large scale. You can run it online, um, fairly easily, and we have batch processes and stuff like that set up for you, so you'll be able to play with that, create sets.
- 31:24
Um, typically you want to create eval sets which have a combination of hard, easy, medium, that kind of stuff. We're not gonna get into data generation, synthetic data generation that much, but our...
- 31:36
Uh, you'll play with it in the copilot. But the actual data generation stuff there, there's actually, um, documentation that we- you can follow up afterwards on how to create, like, easy to set hard, medium sets for your testing purposes.
- 31:49
Uh, the best thing is to s- to sample from your logs some number of things and evaluate or just run it online. Majority of this, these kinds of quality checks at Google were run online, whether it's spam detection, whether it's, like, you know, which...
- 32:03
what, what decisions to make. That's the ideal place you wanna be. Most people, it's very difficult to kind of like sort of implement at that point. I think the challenge-- one of the biggest challenges with LLM as a judge is it's so expensive you can't run, run online.
- 32:15
So that's one of the things that this solves. Um,
- 32:19
all right. So... Oh, go ahead.
- 32:21
How this is different than LLM as a judge? Are you using additional API models or smaller APIs?
- 32:26
Um, so, so we had a bunch of discussion on this. Maybe I'll go very quick- quickly through these slides. Um, so,
- 32:36
so the, the, the-- these models are designed for high precision. Like, for example, if you run the same scorer twice on the same thing, it's not gonna give you different scores, exactly the same score.
- 32:44
With small variations, keep the same scores, right? It's highly designed for, like, super high precision. It's just the architecture of these, these models is, is desi- is, is, is basically, um, l- very low variance.
- 32:58
Um, they... The, the... Part of the reason is that they're using this bidirectional attention instead of, like, the typical decoder models attention. They have a regression head on top instead of, uh...
- 33:07
To- token generations, it's not autoregressively generating tokens, which has a lot of, like, weirdness that happens to it. They did a lot of post-hoc, uh, explanation for scores. They'll come up with a score, then they'll try to justify the score.
- 33:18
That doesn't happen. The other thing is these models have been, uh, trained on, um, a lot of data, like, you know, billions and billions of tokens, which are only for scoring the different kinds of contents, a coding content, other type of content.
- 33:31
So these are-- these generalize really well across, but that also stabilizes them quite a bit. Um, the, the one thing that I would say, which is really nice, is the interface is very nice.
- 33:43
It basically just... You ask a question and give it the, the data, and it will answer the question with a score, and then you can inspect the score and understand why, why it gave you that score.
- 33:52
So it's a fairly simple interface. There's no, like, prompt tuning and so on and so forth. Because in this particular case, when you're evaluating, prompt tuning doesn't really like...
- 34:00
It's not very natural. Like, you know, how do you explain a rubric to, to a model? So these models understand internally why should this score high, why shouldn't they score high.
- 34:10
And then these things come together, uh, using, uh, um, a, a fairly sophisticated model which is like a extension of generalized additive model, which brings all of these different signals together, um, by weighting them based on your thumbs up, thumbs down data.
- 34:25
So this is a process called calibration, where you give it a bunch of data and it understands what's important, what's not. What should I-- Some-- If something fails, I should fail everything, for example, spam.
- 34:35
But if it succeeds, I shouldn't contribute. Those kinds of decisions, it makes those decisions for you based on the data. So that's sort of... It's, it's a much more advanced way of doing evals, and they're sort of ground up built for eval purposes.
- 34:50
It gives them that sort of stability that you desire. And the, and the reason they're very fast is because when you use bidirectional attention, you can build much more denser embeddings.
- 34:58
So with fewer parameters, you can get fairly high quality scores. Go ahead.
- 35:03
Does it work well with other languages?
- 35:06
Uh, right now we have trained it with a handful of languages. Uh, we are, we are pretty soon going to release a model, um, with multi-lang- multilingual capabilities. We don't support multimodal yet, but this is in our roadmap, right?
- 35:22
So right now it's English and a few languages, and then we're gonna expand it beyond that.
- 35:26
Mm-hmm. Um, let- let's kick it... We should kick off the workshop, and then we'll pass it on, and we can just answer all the questions as well. Um, do you just wanna share the doc?
- 35:34
I'll just, uh-
- 35:36
Again, reminder, it's like withpi.ai/workshop.
- 35:40
I'll just share the doc here as well so that you all can see it.
- 35:43
Yeah. I'll also put it in the Slack channel for everybody.
- 35:45
Um, but please get started and let us know.
- 35:55
Um, sorry. Uh, some people maybe have already started working with the colabs, I've seen. Um, but, uh, if... I'll quickly show others, uh, what,
- 36:07
um, what, what the second phase of the exercise is about. Um, but of course, we can continue all of this going forward. Uh, how do I project this?
- 36:19
Um, so-
- 36:19
Um, so the s- the second part of the exercise... By the way, we, we may not have enough time to wrap it all up here, but the second part of the exercise, feel free to do it on your own.
- 36:29
The colab is available for you, so you can, you can, you can use this. Um, this colab, um,
- 36:36
this, this is a pre-prepared colab, um, and
- 36:44
I don't know if you can see this, but basically this particular Colab will take you through these multiple steps. And a lo- lot of people ask me questions about how to use it in code, so I just wanna quickly go over it.
- 36:55
Um, this is some basic installation. This is your scoring spec. This is, this is where all of your intelligence is gonna live. Again, this is a relatively simple example, so it's a relatively simple scoring spec.
- 37:05
But the way you get this spec is basically over here, you get into code, you copy it, and you put it into your Colab, right? So this is basically how you get the spec here.
- 37:16
It's all in natural language spec, uh, with some Python code in there.
- 37:21
This is what you'll change. This is what you'll tweak. This is where everything is going to be. And what you're now doing in this particular Colab, which you can literally click through it, and you don't have to do anything more than that, but or you can play around with it a whole bunch, is...
- 37:36
One is we have some datasets, public, public datasets on Hugging Face, which has the thumbs up, thumbs down data. This is the same data if you've seen the sheet.
- 37:43
That's the same data over here. You-- it-- you, you-- this Colab is loading that data, running the scorer on it, returning your results, um, and then building a similar confusion matrix that you saw there, which indicates how well, well-aligned your scores are.
- 37:59
So just getting you warmed up. Uh, then you can use it to compare models. This is where, like, really interesting stuff starts happening, right? So in this particular case, we are comparing one point five and two point five models.
- 38:11
You can see that two point five has a slightly higher score than one point five for this particular task. But because two point five is mostly for reasoning, there's not that much of a delta here.
- 38:21
If you go to smaller models, uh, like some of the, the, the, the mini models like the Claude Haiku, you could see a-- you'll see a much bigger delta between the quality based on your own scoring system.
- 38:31
So now you can eva-- use your scoring system for evaluating different models. What this is doing is it's taking about, you know, ten examples,
- 38:40
calling these five different models, generating responses, and then scoring them using our scoring system, right? So that's a model comparison. The other upset is to try different prompts. Um, people change their system prompts, but they're worried about, um, when they change it, what would happen.
- 38:57
This is the right way to kinda make sure that you're not regressing. You have your scoring spec. You try different system prompts, see if your scores are going down on your, on your test set.
- 39:07
So again, taking ten examples just to demonstrate how you compare them. Uh, here's-- like we, we created a bad and a good prompt just to kinda accentuate this. So like bad prompts getting much lower score than good prompts on this particular task.
- 39:21
Uh, this is the one that I, uh, that, that I'm very excited about because it brings you into the online world and how to actually do this online, where you're taking this one particular transcript and you are testing it out with different number of samples.
- 39:34
So if you're-- if you use just one sample, which is typically what you do, generate one response with a temperature of point seven, you get a particular score. But as you up the, the number of samples, what it's doing behind the scenes is creating three or four, four of those responses.
- 39:52
Each one is a different response, and then ranking them using our-- the Pi scoring system that you just built and picking the one that's best. And what you would see is as you increase the number of examples, you'll see the score steadily going up, the response quality going up.
- 40:05
So this is, uh, m-- literally, like you can click through this in the Colab and run through it, and you don't-- won't need anything else. But it is a bunch of options for you to play around with this right now, later.
- 40:17
Just wanna introduce this to you guys before, um, before you all disappear. [upbeat music]