AI Engineer Summit 2023
Best Practices for Evaluating Large Language Model Applications with llmeval: Niklas Nielsen
Read the talk
Evaluating LLM Applications with llmeval
A correct answer can still break an application. Niklas Nielsen demonstrates how to test output contracts, diagnose prompt regressions, and extend evaluation from Python checks to human-informed model judges.
From a talk by Niklas Nielsen
Before you start: Basic familiarity with Python, command-line tools, and LLM prompts will help you follow the examples.
What counts as a good response?
How do you safely change an LLM application when you cannot yet say what a good response looks like? After shipping features built on GPT, teams face decisions about prompts, configurations, model providers, self-hosting, and fine-tuning. Each change can alter the generated output, and without an evaluation criterion, it is difficult to tell whether the application improved or regressed.
Niklas Nielsen, CTO and co-founder of Log10, introduces llmeval as a local command-line tool for making those decisions more concrete. Its starting point is to give prompts, test cases, and acceptance criteria a repeatable structure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prompts, Python metrics, and repeated trials
Nielsen presents the initial setup as four lines of code. Initialization creates a folder structure for prompts and tests, while Meta’s Hydra supplies the configuration system. The evaluation logic lives in Python, so a metric can perform a simple comparison, run more involved validation, or call another LLM. Reports then summarize the application and its individual tests.
A test needs both a response criterion and a rule for aggregating attempts. Because model responses vary, Nielsen proposes flexible acceptance criteria: for example, require three of five attempts to pass. That is an example policy for accepting a test, rather than a measured reliability rate. Repeated trials let the report describe behavior that a single response cannot capture.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Define an exact-output contract
The terminal demonstration begins with a small setup sequence:
- Create a working directory and enter it.
- Create a Python virtual environment.
- Install
llmevaland initialize its prompt and test folders.
These are the historical CLI workflow and examples from the recording; the package is now archived.
The first prompt asks what A plus B is, then adds an output constraint: “Only return the answer without any explanation.” That second sentence matters because the test compares the actual response with the expected response as a string. A mathematically correct explanation can therefore fail the application’s contract.
The comparison allows one narrow normalization: it strips leading whitespace. Nielsen explains that Claude sometimes prepends spaces in this example. The test then exposes a metric for the report and a separate pass/fail result. Its concrete inputs are 4 and 5, with the expected output "9". The comparison can be expressed directly in Python:
python
def exact_match(actual: str, expected: str) -> dict[str, bool]:
matches = actual.lstrip() == expected
return {"metric": matches, "result": matches}
expected = "9"
assert exact_match(" 9", expected)["result"]
assert not exact_match("4 + 5 = 9", expected)["result"]
This preserves the distinction between tolerating leading whitespace and tolerating extra explanation. The latter still changes the required output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Remove one instruction and inspect the failure
To expose what the output constraint does, Nielsen removes the answer-only instruction and runs the arithmetic test. The CLI can execute the entire suite or select the math prompt through a configuration override:
bash
# Run the complete suite.
llmeval run
# Run only the math example.
llmeval run prompts=math
Nielsen says the default is five samples per test; for this demonstration, he overrides it to one sample. The defaults live in the llmeval YAML configuration. This run samples Claude, GPT-4, and GPT-3.5 once each.
Generating a report makes the failure visible. GPT-3.5 supplies an explanatory sentence, while Claude writes out the equation. Both convey the correct sum, but neither meets the exact-output requirement. The displayed report distinguishes the responses as follows:
| Model in the report | Response format | Exact match |
|---|---|---|
| GPT-4 | Bare 9 | Pass |
| GPT-3.5-turbo | Answer with explanatory text | Fail |
| Claude-2 | Written-out equation | Fail |
The useful finding is a formatting dependency: removing a prompt clause changes whether the output can satisfy the consumer’s contract.
Nielsen restores the instruction and reruns the test. The latest run passes, although generating the report can still announce failures because earlier runs remain in its scope. Per-run results and the overall report answer different questions: the latest result tells you whether the change repaired this attempt; the overall summary tells you whether any included run failed. Reading the aggregate without checking the run can make a successful fix look unsuccessful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Check whether generated Python parses
The next example, framed as tool use, asks the model to generate Python. Here too, the prompt contains several clauses suppressing surrounding explanations. The acceptance criterion changes, however: instead of requiring one exact string, the test asks whether the response parses as a Python program. A Python metric can implement that boundary with ast.parse:
python
import ast
def parses_as_python(output: str) -> bool:
try:
ast.parse(output)
except SyntaxError:
return False
return True
This accepts different valid programs without requiring identical source text. It checks syntax; it does not execute the program or establish that the program computes the requested result.
The displayed report shows the generated-Python tests passing. Its rows contain examples for adding two numbers and finding an inverse square root across the three models, with True values and green pass indicators. Those results complete the demonstrated parsing check.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use a model when the criterion needs judgment
Exact matching and parsing give precise answers to narrow questions. For a more interpretive task, Nielsen introduces a model-based evaluation of Mermaid diagrams: define a set of criteria, ask for a score from one to five, and request a reason. Writing tests and collecting cases still take work, but a judge model provides a way to evaluate qualities that are harder to capture with a direct comparison.
In this arrangement, typically a larger model evaluates another LLM’s output. The judgment can take several forms:
- Pass/fail: Decide whether the output meets the criteria.
- A grade: Assign a score on a small scale, such as one to five.
- A preference: Choose between alternatives and explain the choice.
The reasoning accompanying a judgment can make the result more useful for diagnosis than a bare score.
The judge also introduces its own failure modes. Nielsen cautions that, when comparing models, judges can favor output from their own model. He also warns about the quality of point scores on zero-to-one or zero-to-100 scales. These cautions affect how much confidence to place in a ranking or numerical difference: a more nuanced-looking result still depends on the evaluator’s behavior. The next step in the demonstration is to bring human feedback back into that process.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate suggestions from a reviewer’s history
Nielsen’s final demonstration bridges model evaluation and human review. Suppose John has already reviewed a collection of completions. The proposal is to model that accumulated feedback as an “Auto John,” which generates suggested reviews for incoming completions. Human judgments become an input to the evaluator instead of being removed from the feedback process.
The interface shows two existing human feedback entries: one with a score of five and another with a more nuanced review. A further item is pending feedback. Clicking it reveals an AI-suggested answer. The visible transition is from a pending review to an available suggestion; it does not establish that a human accepted the suggestion. This leaves a practical role for automation: extend a reviewer’s prior judgments to new work while keeping the distinction between human feedback and generated advice visible.
Nielsen closes by directing viewers to the llmeval documentation and inviting viewers to contact him by email. He also gives Niklas Crafoord as the name to look for on X.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Historical Log10 evaluation package released in October 2023. The project is now archived and no longer receives updates.
Introduction to configuration composition, command-line overrides and parameter sweeps. The page currently displays development-version documentation.
Further reading
Nielsen's 2023 walkthrough of llmeval, with prompt tests, Python parsing checks, model judges and majority-based reporting.
Research on model judges, human preference agreement and limitations including self-enhancement, position and verbosity biases.
Updates since the talk
- Log10 client and AutoFeedback examplesRepository
Later client documentation for generating feedback from existing reviews and custom rubrics. The repository was archived in May 2025.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hi, this is Niklas.
- 0:17
I'm CTO and co-founder of Log10, and we want to talk about how you can scale reliability of L- LLM applications using, uh, a new tool that we've built. During this year, I think we all can agree that there's been, like, this kind of craze in the industry, and we've been rolling out a ton of intelligence features based
- 0:37
on GPT. And we're now kind of finding ourselves in a now what moment because without knowing what good means in a generative setting, it's really hard and risky to evolve your applications, like changing your prompts, configurations, let alone considering going from one model provider to another to more advanced use cases like cell posting or fine-tuning.
- 1:02
We want to introduce a new tool today called llmeval that enables, uh, teams to ship reliable LLM products.
- 1:13
It, it is command line tool that you can run locally, and with these four, uh, lines of code, uh, you should be good to go. Um, the initialization creates a folder structure, um, and best practices for storing prompts and, and tests.
- 1:33
And then this is based on a super configurable system from Meta called Hydra. So you could basically extend it to your heart's desire, and the metrics that we have wired up are in Python, so they could be any logic.
- 1:50
Could be call out to all the LLMs, whatever you want. And after these evaluations have been run, you generate some reports that basically gives you, like, a brief overview of how the entire app and all the tests are looking, but still supports flexible test criteria because, like, these models are very fussy.
- 2:11
It's very hard to say with a guarantee that it's gonna be one or the other, but it's fairly safe to say that the majority cases, or say three out of five, should pass.
- 2:22
And we're gonna jump into command line and taking a look.
- 2:28
We're just gonna create a directory for today.
- 2:35
And go into this directory and create ourselves a virtual environment.
- 2:42
From here, we're gonna install llmeval and initialize the folder structure. What we should be able to see here is
- 2:58
a directory structure where we have our prompts. Let's say a simple case could be this, where we have this message template saying like, "What is A plus B?" Only return the answer without any explanation.
- 3:11
So in this case, we know that we have to prompt engineer further in order to get an exact output. 'Cause let's take a look at how the test looks like.
- 3:21
In this case, we're taking, like, the actual output from the LLM and comparing it with the expected, and this is like a strict comparison. [clears throat]
- 3:31
What we had taken the liberty to do is to strip any spaces that might be... come from, from the left, and that's because some models, in this case, Claude, tends to prepend spaces.
- 3:44
And so it's things like that that you have to watch out for. Then we have the metric, which could be any metric that you wanna surface in the report, and then the result, which is then a pass or fail.
- 3:56
And in this case, we wanna add four and five, and we expect it to be nine.
- 4:01
And I'm just gonna try to run this test here and try to re-revert some of the prompt engineering that we did earlier. So I'm gonna re-remove
- 4:12
only return the answer without any explanation. And the way you can start it is llmeval run.
- 4:24
But if you want to override anything-- If you just do llmeval run, it runs everything. But if you do like prompts equals math, then it's only gonna run the math example.
- 4:34
If you do n tries one, then it's just gonna do one sample. By default, we do five samples, so, so we get, like, a better read on the stability of, of each test, but it might be too much for you.
- 4:49
But you can override anything. You can find these default settings here in the llmeval YAML. And but let's try to run this and see what happens. And so this ran across Claude, GPT-4, and, and GPT-3.5 once.
- 5:07
So we can go in and generate a report
- 5:12
and see, like, actually something failed. What was it that failed? So let's take a look at the output here. And in this case, because we've removed our prompt engineering, GPT-3.5 starts being a bit chatty and says like, "Four point five equals nine."
- 5:27
Claude does something similar. So it kind of writes out the... writes out the equation. And now I'm gonna try to revert
- 5:36
and see... Let's, let's get this in. And we try to run one more time.
- 5:45
Great. Now, when we generate the report, it could say some tests failed, but the most recent test that ran passed. So when you do the report, it's gonna generate a summary.
- 5:55
It could generate a report per, per run, but then also say, overall, was there anything that, that failed out of these reports?
- 6:04
If you wanna go a bit more advanced, let's say you wanna use tools, y- we c- we have an example here where we are generating some Python code, and again, we had to add a number of different, um, clauses to make sure that it only outputs Python.
- 6:19
It tends to be very happy generating, um, s- surrounding explanations.
- 6:24
Uh, so in this case, we are gonna see whether or not, um, it returns an actual Python program that could be, that could be parsed. So let's try to run that.
- 6:37
If you go in and take a look at this report,
- 6:41
you can see that these tests actually end up, end up passing our tool use. And to, to round up,
- 6:48
we have model-based evaluation as well, where you can test using other models. And so in this case, say with grading, we can go in and define, like, a full set of criteria.
- 7:02
Here, we're about evaluating Mermaid diagrams, giving a score between one and five, and the reason, and that, that is also supported in llmeval. One thing about the previous approach is that it takes quite an amount, amount of work to set up these tests and gather your test cases.
- 7:21
And one really compelling answer to evaluation has been model-based evaluation, and it's, uh, it's a setting where you have, well, typically a larger model discriminate or kinda grade or be a judge over the output from another LLM.
- 7:36
And that makes it so you can get more nuanced output like pass/fail or a grade from one to five or preferences between different options and its reasoning behind it.
- 7:47
There's a number of pitfalls, unfortunately, around this approach around biases towards the output from the model itself. If you're sweeping different models, they tend to prefer their own, own output.
- 8:00
They're not very good at giving, uh, point scores, saying anything between zero and one, or larger scores between zer- zero and 100. [clears throat]
- 8:09
But there are different ways where you can start increasing the accuracy of the kind of feedback that's been generated.
- 8:17
And [clears throat] we've been working on this, where you basically start bridging between model-based and human feedback. So instead of removing the human completely from the feedback, you start taking in all feedback that might have been given prior and start modeling it and say, like, if you have all the feedback from John, then we create an Auto John that
- 8:38
will start create-- generating feedback per review, um, for any incoming completions. And so in this case here, we have two pieces of feedback that's been already given by human.
- 8:48
See here, it was all just, like, a score of five, or here, just, like, a bit more nuanced. But here, we are kind of pending feedback. And if you click this, we have AI suggested an answer to, to this.
- 9:06
And that's all I had today. Um, if you want to get started on, um, llmeval, we have our documentation at our usual documentation site. And you can find me at, uh, Niklas Crafoord at, on, uh, X, or formerly, formerly known as Twitter, or shoot me an email at, uh, [REDACTED:email_address].
- 9:25
Thank you. [outro music]