How to evaluate a model for your use case

How do you know whether a language model performs well on your particular task? Standard metrics and benchmarks offer useful starting points, but they do not supply a universal evaluation procedure. Emmanuel Turlay, introducing himself as CEO of Sematic, the company behind Airtrain, starts with this gap between general capability and application-specific performance.

Model evaluation is statistical measurement on a dataset independent of the training data. The question is how well a model handles a particular use case across a large collection of examples, rather than whether it produces one convincing answer. Evaluation belongs after training or fine-tuning and should recur throughout development as a check on performance and safety. Turlay compares it to running a test suite in a continuous integration pipeline: a repeatable part of the workflow, rather than a final spot check.

Traditional supervised learning often provides a well-defined output structure and established ways to measure errors.

These metrics give teams a clear way to grade predictions against expected outputs.

Language models generate text, whose acceptable forms are much less constrained. An output that differs from a reference answer is not necessarily wrong. A useful evaluation therefore has to distinguish differences in wording from differences in correctness or suitability for the task.

The first practical distinction is whether you have labeled reference answers. With references, you can compare generated text against expected text using token overlap.

• BLEU: A precision-oriented metric based on matching n-grams—sequences of tokens—between an output and a reference. It is commonly used for translation and can also be applied to summarization. Overlap alone, however, does not establish intelligibility or grammatical correctness.
• ROUGE: A family of metrics presented here through recall: how much of the reference’s token sequences appears in the generated output. Its principal application in this discussion is summarization.

Both provide useful measurements, but their relationship to success depends on what the application actually needs.

Without a reference summary, you can still compare a summary with its source text. Turlay introduces density and coverage for this purpose. The published definitions in Newsroom make the distinction precise: coverage is the fraction of summary tokens belonging to fragments shared with the source; density sums the squared lengths of those fragments and divides by summary length. Longer copied passages therefore contribute more strongly to density. These measures require the source text, but no reference summary, and describe extractiveness rather than overall summary quality.

This leaves a substantial boundary around what the metrics tell you. They are useful for high-level tasks such as translation and summarization, but a measurement tailored to those tasks need not capture success in another application. Public benchmarks broaden the comparison, while retaining their own task boundaries.

A benchmark is a standardized test for particular capabilities. GLUE, the General Language Understanding Evaluation benchmark, combines nine language-understanding tasks, including paraphrase detection and sentiment analysis. HellaSwag tests commonsense inference by asking a model to choose a plausible continuation. The slide’s example concerns bathing a dog; Turlay identifies answer C as the reasonable continuation.

TriviaQA tests knowledge and reading comprehension using trivia questions and associated evidence. Its original paper distinguishes 95,000 question-answer pairs from more than 650,000 question-answer-evidence triples; the talk’s description of almost a million questions conflates these units. The questions were written by trivia enthusiasts, with evidence documents collected separately.

ARC tests reasoning through multiple-choice science questions. Its documented scope is grade-school science, rather than the high-school level mentioned in the recording, and it separates questions into Easy and Challenge sets. Together, these benchmarks help map how language models compare across different capabilities.

The application may ask for something much more specific than a benchmark measures:

• Clinical extraction: Extract symptoms from a doctor’s notes.
• Recipe extraction: Extract ingredients from a recipe.
• API interaction: Form a JSON payload to query an API.

A general benchmark score does not tell you how a model performs these operations on the inputs your application will supply. Each application needs an evaluation procedure tied to its own task and data. Building that procedure takes work; the next question is how to make its scoring practical.

A second language model can grade the first model’s output. Describe what the application is trying to accomplish, specify the properties you want to measure, and ask the grading model for a numerical assessment. This creates a specialized metric whose definition comes from your application’s criteria.

The evaluation pipeline has five steps:

1. Run the evaluation dataset through the candidate model.
2. Collect the generated outputs, or inferences.
3. Place each output inside a scoring prompt that describes the task, the property being graded, and the scale—for example, 1 to 10.
4. Send that prompt to a scoring model and obtain a numerical score.
5. Repeat across the dataset to produce a distribution of scores for that property.

The distribution connects the grading rule to performance across the evaluation set, extending the assessment beyond an isolated output. The completed diagram shows the entire path through the scoring model to its illustrative result.

Consider a model generating closing words for professional emails. The property to evaluate is politeness, and the grading scale runs from 1 to 10. Turlay contrasts “Please let us know at your earliest convenience,” which scores highly, with “Tell me ASAP,” which scores poorly. The difference is meaningful for this application even though both statements request a response.

A small Python prompt builder makes the components of this scoring step explicit. It keeps the task, property, and scale fixed while changing the closing being evaluated:

import json

closings = [
    "Please let us know at your earliest convenience",
    "Tell me ASAP",
]


def politeness_prompt(closing: str) -> str:
    return (
        "Task: Generate closing words for a professional email.\n"
        "Evaluate the politeness of the closing below.\n"
        "Scale: 1 = very impolite; 10 = very polite.\n"
        "Return only one integer from 1 to 10.\n"
        f"Closing: {json.dumps(closing)}"
    )


scoring_prompts = [politeness_prompt(text) for text in closings]
for prompt in scoring_prompts:
    print(prompt)
    print()

These prompts are ready to send to a scoring model; constructing them does not assign scores. Once the grader returns its judgments, those values can contribute to the dataset-wide politeness distribution.

Turlay reports that GPT-4 was the best grading model his team had found at the time, but could be costly for large datasets; Flan-T5 offered a useful trade-off between speed and correctness. He supplies no quantitative comparison or evaluation protocol, so this is a historical account of the team’s experience rather than a current model ranking.

Airtrain packages the workflow around those dataset-wide comparisons: upload a dataset, select candidate models, describe the properties to measure, and visualize the resulting metric distributions. Turlay names Llama 2, Falcon, Flan-T5, and a user’s own model as comparison options. The recording closes with an invitation to Airtrain’s then-early-access offering. The intended decision rests on statistical evidence from your dataset and your criteria—the same application-specific foundation that generic leaderboards cannot supply.