# Best Practices for Evaluating Large Language Model Applications with llmeval: Niklas Nielsen

Niklas Nielsen, Log10

AI Engineer Summit 2023

[Talk page](https://ai.engineer/talks/fiXjTif1nS4-best-practices-evaluating-large-language-model) · [Video](https://www.youtube.com/watch?v=fiXjTif1nS4)

## Evaluating LLM Applications with llmeval

A correct answer can still break an application. Niklas Nielsen demonstrates how to test output contracts, diagnose prompt regressions, and extend evaluation from Python checks to human-informed model judges.

Basic familiarity with Python, command-line tools, and LLM prompts will help you follow the examples.

### What counts as a good response?

[0:17](https://www.youtube.com/watch?v=fiXjTif1nS4&t=17s)

How do you safely change an LLM application when you cannot yet say what a good response looks like? After shipping features built on **GPT**, teams face decisions about prompts, configurations, model providers, self-hosting, and fine-tuning. Each change can alter the generated output, and without an evaluation criterion, it is difficult to tell whether the application improved or regressed.

![Slide with three numbered points about evaluating GPT applications, risky changes, and introducing llmeval, beside a photograph of a van.](https://ai.engineer/api/talk-essay-media/199dc106a53a1b974243eebc48a824f99a72457e42a8394ffdd8caa0c6580548)

We are stuck without evaluation.

Niklas Nielsen, CTO and co-founder of Log10, introduces [llmeval](https://pypi.org/project/llmeval/0.4.1/) as a local command-line tool for making those decisions more concrete. Its starting point is to give prompts, test cases, and acceptance criteria a repeatable structure.

### Prompts, Python metrics, and repeated trials

[1:13](https://www.youtube.com/watch?v=fiXjTif1nS4&t=73s)

Nielsen presents the initial setup as four lines of code. Initialization creates a folder structure for prompts and tests, while Meta’s [Hydra](https://hydra.cc/docs/intro/) supplies the configuration system. The evaluation logic lives in Python, so a metric can perform a simple comparison, run more involved validation, or call another LLM. Reports then summarize the application and its individual tests.

**A test needs both a response criterion and a rule for aggregating attempts.** Because model responses vary, Nielsen proposes flexible acceptance criteria: for example, require three of five attempts to pass. That is an example policy for accepting a test, rather than a measured reliability rate. Repeated trials let the report describe behavior that a single response cannot capture.

### Define an exact-output contract

[2:28](https://www.youtube.com/watch?v=fiXjTif1nS4&t=148s)

The terminal demonstration begins with a small setup sequence:

1. Create a working directory and enter it.
2. Create a Python virtual environment.
3. Install `llmeval` and initialize its prompt and test folders.

These are the historical CLI workflow and examples from the recording; the package is now archived.

The first prompt asks what A plus B is, then adds an output constraint: “Only return the answer without any explanation.” That second sentence matters because the test compares the actual response with the expected response as a string. A mathematically correct explanation can therefore fail the application’s contract.

The comparison allows one narrow normalization: it strips leading whitespace. Nielsen explains that **Claude** sometimes prepends spaces in this example. The test then exposes a metric for the report and a separate pass/fail result. Its concrete inputs are `4` and `5`, with the expected output `"9"`. The comparison can be expressed directly in Python:

```python
def exact_match(actual: str, expected: str) -> dict[str, bool]:
    matches = actual.lstrip() == expected
    return {"metric": matches, "result": matches}

expected = "9"
assert exact_match(" 9", expected)["result"]
assert not exact_match("4 + 5 = 9", expected)["result"]
```

This preserves the distinction between tolerating leading whitespace and tolerating extra explanation. The latter still changes the required output.

![VS Code displays math.yaml with an exact_match metric, a whitespace-stripping comparison, inputs a: 4 and b: 5, and expected output "9".](https://ai.engineer/api/talk-essay-media/55bf1c340771f1ed37ba3d459abc7fc750f67d958472ce76af78ca821418cbb4)

An exact-match test checks inputs 4 and 5 against the expected answer 9.

### Remove one instruction and inspect the failure

[4:01](https://www.youtube.com/watch?v=fiXjTif1nS4&t=241s)

To expose what the output constraint does, Nielsen removes the answer-only instruction and runs the arithmetic test. The CLI can execute the entire suite or select the math prompt through a configuration override:

```bash
# Run the complete suite.
llmeval run

# Run only the math example.
llmeval run prompts=math
```

Nielsen says the default is five samples per test; for this demonstration, he overrides it to one sample. The defaults live in the `llmeval` YAML configuration. This run samples Claude, **GPT-4**, and **GPT-3.5** once each.

Generating a report makes the failure visible. GPT-3.5 supplies an explanatory sentence, while Claude writes out the equation. Both convey the correct sum, but neither meets the exact-output requirement. The displayed report distinguishes the responses as follows:

| Model in the report | Response format | Exact match |
| --- | --- | --- |
| GPT-4 | Bare `9` | Pass |
| GPT-3.5-turbo | Answer with explanatory text | Fail |
| Claude-2 | Written-out equation | Fail |

The useful finding is a formatting dependency: removing a prompt clause changes whether the output can satisfy the consumer’s contract.

![llmeval report table shows three model outputs for adding 4 and 5, with a green pass for the bare answer 9 and failures for answers containing extra text.](https://ai.engineer/api/talk-essay-media/0179b3c421915257dc367ce0b72a7cd5f4fd2f214fbb89a68142ae0a74607ca7)

The arithmetic report shows GPT-4 passing while GPT-3.5-turbo and Claude-2 fail exact matching.

Nielsen restores the instruction and reruns the test. The latest run passes, although generating the report can still announce failures because earlier runs remain in its scope. **Per-run results and the overall report answer different questions:** the latest result tells you whether the change repaired this attempt; the overall summary tells you whether any included run failed. Reading the aggregate without checking the run can make a successful fix look unsuccessful.

### Check whether generated Python parses

[6:04](https://www.youtube.com/watch?v=fiXjTif1nS4&t=364s)

The next example, framed as tool use, asks the model to generate Python. Here too, the prompt contains several clauses suppressing surrounding explanations. The acceptance criterion changes, however: instead of requiring one exact string, the test asks whether the response parses as a Python program. A Python metric can implement that boundary with `ast.parse`:

```python
import ast


def parses_as_python(output: str) -> bool:
    try:
        ast.parse(output)
    except SyntaxError:
        return False
    return True
```

This accepts different valid programs without requiring identical source text. It checks syntax; it does not execute the program or establish that the program computes the requested result.

The displayed report shows the generated-Python tests passing. Its rows contain examples for adding two numbers and finding an inverse square root across the three models, with `True` values and green pass indicators. Those results complete the demonstrated parsing check.

![VS Code report displays six generated-code rows for adding two numbers and finding an inverse square root across three models, each with True and a green pass indicator.](https://ai.engineer/api/talk-essay-media/471706d5b0d72c53337f3c9b11f7982717ba2d0d9d3ca58f4472de047592827d)

Generated Python examples receive passing results in the evaluation report.

### Use a model when the criterion needs judgment

[6:48](https://www.youtube.com/watch?v=fiXjTif1nS4&t=408s)

Exact matching and parsing give precise answers to narrow questions. For a more interpretive task, Nielsen introduces a model-based evaluation of Mermaid diagrams: define a set of criteria, ask for a score from one to five, and request a reason. Writing tests and collecting cases still take work, but a judge model provides a way to evaluate qualities that are harder to capture with a direct comparison.

In this arrangement, typically a larger model evaluates another LLM’s output. The judgment can take several forms:

- **Pass/fail:** Decide whether the output meets the criteria.
- **A grade:** Assign a score on a small scale, such as one to five.
- **A preference:** Choose between alternatives and explain the choice.

The reasoning accompanying a judgment can make the result more useful for diagnosis than a bare score.

The judge also introduces its own failure modes. Nielsen cautions that, when comparing models, judges can favor output from their own model. He also warns about the quality of point scores on zero-to-one or zero-to-100 scales. These cautions affect how much confidence to place in a ranking or numerical difference: a more nuanced-looking result still depends on the evaluator’s behavior. The next step in the demonstration is to bring human feedback back into that process.

### Generate suggestions from a reviewer’s history

[8:17](https://www.youtube.com/watch?v=fiXjTif1nS4&t=497s)

Nielsen’s final demonstration bridges model evaluation and human review. Suppose John has already reviewed a collection of completions. The proposal is to model that accumulated feedback as an “Auto John,” which generates suggested reviews for incoming completions. Human judgments become an input to the evaluator instead of being removed from the feedback process.

The interface shows two existing human feedback entries: one with a score of five and another with a more nuanced review. A further item is pending feedback. Clicking it reveals an AI-suggested answer. The visible transition is from a pending review to an available suggestion; it does not establish that a human accepted the suggestion. This leaves a practical role for automation: extend a reviewer’s prior judgments to new work while keeping the distinction between human feedback and generated advice visible.

Nielsen closes by directing viewers to the llmeval documentation and inviting viewers to contact him by email. He also gives Niklas Crafoord as the name to look for on X.

## Resources

- [Ready, Set, Test: Building Evaluation into Your LLM Workflow](https://arjunbansal.substack.com/p/ready-set-test-building-evaluation) — Nielsen's 2023 walkthrough of llmeval, with prompt tests, Python parsing checks, model judges and majority-based reporting.
- [llmeval 0.4.1](https://pypi.org/project/llmeval/0.4.1/) — Historical Log10 evaluation package released in October 2023. The project is now archived and no longer receives updates.
- [Hydra configuration documentation](https://hydra.cc/docs/intro/) — Introduction to configuration composition, command-line overrides and parameter sweeps. The page currently displays development-version documentation.
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685) — Research on model judges, human preference agreement and limitations including self-enhancement, position and verbosity biases.
- [Log10 client and AutoFeedback examples](https://github.com/log10-io/log10) — Later client documentation for generating feedback from existing reviews and custom rubrics. The repository was archived in May 2025.

## Slides

### Scaling Reliability of LLM Applications

Niklas Nielsen, Log10’s CTO and co-founder, introduces a talk on scaling LLM application reliability using a tool his team built.

[0:14](https://www.youtube.com/watch?v=fiXjTif1nS4&t=14s) [Title slide for Niklas Nielsen’s talk on scaling the reliability of LLM applications.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d89576aa4716dcfd54eea280eda16b0249d1bc0cb78d5de38c1591989a0c323.webp)

Scaling Reliability of LLM Applications

Niklas Nielsen
CTO & Co-founder

LOG10

### We Are Stuck Without Evaluation

Without a way to define good output, teams risk regressions when changing prompts, configurations or model providers. Nielsen introduces llmeval to help teams ship reliable LLM products.

[0:27](https://www.youtube.com/watch?v=fiXjTif1nS4&t=27s) [Slide describes the risks of changing LLM applications without evaluation and introduces llmeval.](https://assets.ai.engineer/sites/aiecode2025/sha256/e00f9952de9329e614a422d056c1dc1d682fafc1422c6d2e6a3215dd1c6b0197.webp)

We are stuck without evaluation

01
After deploying GPT based applications in production, we are having a collective “now what” moment.

02
Without knowing what good means, it is hard and risky making changes in prompts, configurations and models provider i.e. OpenAI -> Anthropic -> Self hosting

03
Introducing llmeval today so your team can ship reliable LLM products 🚢

02

### llmeval Overview

llmeval runs evaluations locally using configurable Python metrics, then generates Markdown reports. Its rollups can require X out of Y runs to pass, accommodating variable LLM outputs.

[1:10](https://www.youtube.com/watch?v=fiXjTif1nS4&t=70s) [Slide shows llmeval command-line steps, configuration and metrics features, and an example evaluation report.](https://assets.ai.engineer/sites/aiecode2025/sha256/c7c061213f65497b4efe6d6054dcd956879ba3e86804b395add6b1ad8dc52aba.webp)

llmeval

$ pip install llmeval
$ llmeval init
$ llmeval run
$ llmeval report

01
CLI and Simple folder structure where you can import prompts, associated tests separately.

Run everything locally.

02
Extensible - hierarchical configuration using a system from Meta called Hydra.

Metrics are backed by python - use any tool or LLM to evaluate.

03
Generate markdown reports with custom roll up.

Support X out of Y passes, or any logic you prefer to deal with probabilistic nature of LLMs.

Embedded report:
[illegible]role': 'human', 'content': 'what is the capital of {country}?'}]
model | temperature | country | Expected | Actual #0 | correctness (metric) #0 | correctness (pass / fail) #0 | Actual #1 | correctness (metric) #1 | correctness (pass / fail) #1 | Actual #2 | correctness (metric) #2 | correctness (pass / fail) #2
gpt-3.5-turbo | 0.9 | Denmark | Copenhagen | The capital of Denmark is Copenhagen. | 1 | ✅ | [illegible] | 1 | ✅ | The capital of Denmark is Copenhagen. | 1 | ✅
gpt-3.5-turbo | 0.9 | Germany | Berlin | The capital of Germany is Berlin. | 1 | ✅ | The capital of Germany is Berlin. | 1 | ✅ | The capital of Germany is Berlin. | 1 | ✅
[illegible]pt-4 | 0.9 | [blank] | [blank] | The capital of Denmark is Copenhagen. | 1 | ✅ | The capital of Denmark is Copenhagen. | 1 | ✅ | The capital of Denmark is Copenhagen. | 1 | ✅
[illegible] | 0.9 | Germany | Berlin | The capital of Germany is Berlin. | 1 | ✅ | The capital of Germany is Berlin. | 1 | ✅ | The capital of Germany is Berlin. | 1 | ✅

[illegible]'content': '{question}'}]
[illegible]tion | Expected | Actual #0 | exact_match (metric) #0 | exact_match (pass / fail) #0 | Actual #1 | exact_match (metric) #1 | exact_match (pass / fail) #1 | Actual #2
[illegible] | The meaning of life is a philosophical [illegible] | False | 🔴 (skipping) | The meaning of life is a philosophical question th... | False | 🔴 [illegible]

03

### Set Up a Virtual Environment

Nielsen creates the ai.engineer directory, then creates and activates a Python virtual environment inside it.

[2:37](https://www.youtube.com/watch?v=fiXjTif1nS4&t=157s) [VS Code terminal shows a project directory and an activated Python virtual environment.](https://assets.ai.engineer/sites/aiecode2025/sha256/e007096f2804d0f733626e8626c1b81069d942adf00540b0a6d92ec71fb6206c.webp)

~ mkdir ai.engineer
~ cd ai.engineer
ai.engineer virtualenv venv
created virtual environment CPython3.11.5.final.0-64 in 229ms
  creator CPython3Posix(dest=/Users/niklasqnielsen/ai.engineer/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/niklasqnielsen/Library/Application Support/virtualenv)
    added seed packages: pip==23.2.1, setuptools==68.1.2, wheel==0.41.2
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
ai.engineer . ./venv/bin/activate
(venv) ai.engineer

Interface text:
Code  File  Edit  Selection  View  Go  Run  Terminal  Window  Help
Search
TESTING  DEBUG CONSOLE  PORTS  OUTPUT
zsh
76%

### Initialize llmeval

After installing llmeval, Nielsen runs llmeval init to copy example files into the project. The terminal shows commands for running an evaluation and generating a report.

[2:49](https://www.youtube.com/watch?v=fiXjTif1nS4&t=169s) [VS Code terminal shows package installation output and llmeval initialization instructions.](https://assets.ai.engineer/sites/aiecode2025/sha256/360d29c333a2f0d71a49fbdf904439db28ec4e491e8034972e2e3fb97255abee.webp)

Using cached googleapis_common_protos-1.60.0-py2.py3-none-any.whl (227 kB)
Downloading grpcio_status-1.59.0-py3-none-any.whl (14 kB)
Using cached httpcore-0.18.0-py3-none-any.whl (76 kB)
Using cached huggingface_hub-0.16.4-py3-none-any.whl (268 kB)
Using cached jsonpointer-2.4-py2.py3-none-any.whl (7.8 kB)
Using cached marshmallow-3.20.1-py3-none-any.whl (49 kB)
Using cached soupsieve-2.5-py3-none-any.whl (36 kB)
Using cached typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)
Using cached cachetools-5.3.1-py3-none-any.whl (9.3 kB)
Using cached filelock-3.12.4-py3-none-any.whl (11 kB)
Using cached fsspec-2023.9.2-py3-none-any.whl (173 kB)
Installing collected packages: ptyprocess, faiss-cpu, antlr4-python3-runtime, urllib3, typing-extensions, tqdm, tenacity, soupsieve, sniffio, six, semver, regex, PyYAML, python-dotenv, pyasn1, protobuf, pexpect, packaging, numpy, mypy-extensions, multidict, jsonpointer, idna, h11, grpcio, google-crc32c, fsspec, frozenlist, filelock, distro, click, charset-normalizer, certifi, cachetools, backoff, attrs, async-timeout, annotated-types, yarl, typing-inspect, SQLAlchemy, rsa, requests, python-dateutil, pydantic-core, pyasn1-modules, proto-plus, omegaconf, numexpr, marshmallow, jsonpatch, googleapis-common-protos, google-resumable-media, beautifulsoup4, anyio, aiosignal, wikipedia, tiktoken, pydantic, hydra-core, huggingface_hub, httpcore, grpcio-status, google-search-results, google-auth, faker, dataclasses-json, aiohttp, tokenizers, openai, langsmith, httpx, google-api-core, langchain, google-cloud-core, anthropic, google-cloud-bigquery, log10-io, llmeval
Successfully installed PyYAML-6.0.1 SQLAlchemy-2.0.21 aiohttp-3.8.5 aiosignal-1.3.1 annotated-types-0.5.0 anthropic-0.3.11 antlr4-python3-runtime-4.9.3 anyio-3.7.1 async-timeout-4.0.3 attrs-23.1.0 backoff-2.2.1 beautifulsoup4-4.12.2 cachetools-5.3.1 certifi-2023.7.22 charset-normalizer-3.3.0 click-8.1.7 dataclasses-json-0.6.1 distro-1.8.0 faiss-cpu-1.7.4 faker-19.6.2 filelock-3.12.4 frozenlist-1.4.0 fsspec-2023.9.2 google-api-core-2.12.0 google-auth-2.23.2 google-cloud-bigquery-3.11.4 google-cloud-core-2.3.3 google-crc32c-1.5.0 google-resumable-media-2.6.0 google-search-results-2.4.2 googleapis-common-protos-1.60.0 grpcio-1.59.0 grpcio-status-1.59.0 h11-0.14.0 httpcore-0.18.0 httpx-0.25.0 huggingface_hub-0.16.4 hydra-core-1.3.2 idna-3.4 jsonpatch-1.33 jsonpointer-2.4 langchain-0.0.303 langsmith-0.0.41 llmeval-0.4.0 log10-io-0.4.1 marshmallow-3.20.1 multidict-6.0.4 mypy-extensions-1.0.0 numexpr-2.8.7 numpy-1.26.0 omegaconf-2.3.0 openai-0.28.1 packaging-23.1 pexpect-4.8.0 proto-plus-1.22.3 protobuf-4.24.3 ptyprocess-0.7.0 pyasn1-0.5.0 pyasn1-modules-0.3.0 pydantic-2.4.2 pydantic-core-2.10.1 python-dateutil-2.8.2 python-dotenv-1.0.0 regex-2023.8.8 requests-2.31.0 rsa-4.9 semver-3.0.1 six-1.16.0 sniffio-1.3.0 soupsieve-2.5 tenacity-8.2.3 tiktoken-0.4.0 tokenizers-0.14.0 tqdm-4.66.1 typing-extensions-4.8.0 typing-inspect-0.9.0 urllib3-2.0.5 wikipedia-1.4.0 yarl-1.9.2

(venv) ➜ ai.engineer llmeval init
Copied example files to /Users/niklasqnielsen/ai.engineer.
1. Try out evaluating our model with the example config files in /Users/niklasqnielsen/ai.engineer with:
   llmeval run
[illegible] Generate a report stored under /Users/niklasqnielsen/ai.engineer/multirun/ once the evaluation is completed with:
   llmeval report
(venv) ➜ ai.engineer

Interface text:
Code  File  Edit  Selection  View  Go  Run  Terminal  Window  Help
Search
TESTING  DEBUG CONSOLE  PORTS  OUTPUT
zsh
76%

### Exploring the llmeval project and math test

The math test compares the model’s output with “9” for inputs 4 and 5. It strips leading whitespace before the exact-match comparison and records the result as pass or fail.

[2:57](https://www.youtube.com/watch?v=fiXjTif1nS4&t=177s) [VS Code Explorer displays a prompts folder with example YAML files and a tests folder.](https://assets.ai.engineer/sites/aiecode2025/sha256/83316e4065ad18e240f9458ecbfc3cf4979d2d96c2daa133dc51b3ac116e169c.webp)

Interface text:
EXPLORER
AI.ENGINEER
  prompts
    tests
    capitals.yaml
    code_metrics_rollup.yaml
    code.yaml
    grading.yaml
    math.yaml
    noop.yaml
    ping.yaml
    summary.yaml
  venv
  llmeval.yaml
PROBLEMS
Filter (e.g. text, **/*.ts, !**/node...)
No problems have been detected in the workspace.
Show All Commands ⇧ ⌘ P
Go to File ⌘ P
Find in Files ⇧ ⌘ F
Toggle Full Screen ⌃ ⌘ F
Show Settings ⌘ ,

[3:02](https://www.youtube.com/watch?v=fiXjTif1nS4&t=182s) [YAML test compares the model output with the expected answer 9 for inputs 4 and 5.](https://assets.ai.engineer/sites/aiecode2025/sha256/4e75ee92c3421a3dfea2021f8413b9b558300b64e3a37a548b01c1be5eede816.webp)

# Copyright Log10, Inc 2023

metrics:
  - name: exact_match
    code: |
      metric = actual.lstrip() == expected
      result = metric
references:
  - input:
      a: 4
      b: 5
    expected: "9"

Interface text:
EXPLORER
AI.ENGINEER
  prompts
    tests
      capitals.yaml
      code_metrics_rollup.yaml
      code.yaml
      grading.yaml
      math.yaml
      noop.yaml
      ping.yaml
      summary.yaml
    capitals.yaml
    code_metrics_rollup.yaml
    code.yaml
    grading.yaml
    math.yaml
    noop.yaml
PROBLEMS
Filter (e.g. text, **/*.ts, !**/node...)
No problems have been detected in the workspace.
prompts > tests > math.yaml > references > 0
Ln 13, Col 1  Spaces: 2  UTF-8  LF  YAML  No JSON Schema

### Run the math prompt

The math prompt asks GPT-4-16k to add two variables. The demo prepares a command that selects the math example and overrides the default five samples with one.

[4:01](https://www.youtube.com/watch?v=fiXjTif1nS4&t=241s) [A math prompt asks for the sum of two variables; the terminal contains a one-try evaluation command.](https://assets.ai.engineer/sites/aiecode2025/sha256/92b9c39cd56052aff9cb9b97a9f8050341877f5a6ba50e8e3c60bdf83cbc2b91.webp)

# Copyright Log10, Inc 2023

defaults:
  - tests/math
name: math
model: gpt-4-16k
temperature: 0.0
variables:
  - name: a
  - name: b
messages:
  - role: "human"
    content: "What is {a} + {b}?"

(venv) ➜ ai.engineer llmeval run prompts=math n_tries=1

Interface text:
EXPLORER
AI.ENGINEER
summary.yaml
capitals.yaml
code_metrics_rollup.yaml
code.yaml
grading.yaml
math.yaml
noop.yaml
ping.yaml
summary.yaml
venv
llmeval.yaml
PROBLEMS
math.yaml .../tests
math.yaml prompts
prompts > math.yaml > messages > 0 > content
TESTING
DEBUG CONSOLE
PORTS
OUTPUT
zsh
Ln 13, Col 33
Spaces: 2
UTF-8
LF
YAML
No JSON Schema
77%

### Evaluation configuration

The math evaluation overrides the default five tries with one sample per test and runs across GPT-3.5, GPT-4, and Claude 2.

[4:51](https://www.youtube.com/watch?v=fiXjTif1nS4&t=291s) [The evaluation configuration sets five default tries and a sweep across three models while an evaluation runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/681be6c17036fdfacba2417d1b1547d72756d4434e6d8145122fa150666fe116.webp)

version: 0.1.0
n_tries: 5

hydra:
  job:
    chdir: True
  mode: MULTIRUN
  sweeper:
    params:
      prompts: glob(*)
      prompts.model: choice("gpt-3.5-turbo", "gpt-4", "claude-2")

Terminal:
(venv) ➜ ai.engineer llmeval run prompts=math n_tries=1
Running evaluation...

Interface text:
EXPLORER
AI.ENGINEER
summary.yaml
capitals.yaml
code_metrics_rollup.yaml
code.yaml
grading.yaml
math.yaml
noop.yaml
ping.yaml
summary.yaml
venv
llmeval.yaml
PROBLEMS
math.yaml .../tests
math.yaml prompts
llmeval.yaml
TESTING
DEBUG CONSOLE
PORTS
OUTPUT
zsh
Ln 1, Col 1
Spaces: 2
UTF-8
LF
YAML
No JSON Schema
77%

### Generated evaluation report

The report flags GPT-3.5’s answer as an exact-match failure: it returned “4 + 5 equals 9.” when the expected text was “9”.

[5:10](https://www.youtube.com/watch?v=fiXjTif1nS4&t=310s) [The report shows that an explanatory response failed the exact-match test, and the terminal reports that some tests failed.](https://assets.ai.engineer/sites/aiecode2025/sha256/9d205450c610795265604b56a80079d6a28c15b5203f479a4c12ab8bef3d9e92.webp)

a: 4
b: 5
expected: '9'
actual: 4 + 5 equals 9.
metrics:
  exact_match:
    metric: false
    result: fail
    skipped: false
name: math
model: gpt-3.5-turbo
temperature: 0.0
variables:
- name: a
- name: b
messages:

(venv) ➜ ai.engineer llmeval run prompts=math n_tries=1
Running evaluation...
Ran evaluation successfully.
[2023-09-30 09:39:05,958][HYDRA] Launching 3 jobs locally
[2023-09-30 09:39:05,958][HYDRA] #0 : prompts=math prompts.model=gpt-3.5-turbo n_tries=1
[2023-09-30 09:39:06,611][HYDRA] #1 : prompts=math prompts.model=gpt-4 n_tries=1
[2023-09-30 09:39:07,707][HYDRA] #2 : prompts=math prompts.model=claude-2 n_tries=1
[2023-09-30 09:39:09,067][httpx][INFO] - HTTP Request: POST https://api.anthropic.com/v1/complete "HTTP/1.1 200 OK"
(venv) ➜ ai.engineer llmeval report
Generating report for: multirun/2023-09-30/09-39-05
Processing job at date: 2023-09-30 time: 09-39-05 job id: 0 run id: 0...
Processing job at date: 2023-09-30 time: 09-39-05 job id: 1 run id: 0...
Processing job at date: 2023-09-30 time: 09-39-05 job id: 2 run id: 0...
Report written to multirun/2023-09-30/09-39-05/report.md
Some tests failed

Interface text:
EXPLORER
AI.ENGINEER
multirun/2023-09-30/09-39-05
0
1
2
multirun.yaml
report.md
prompts
venv
llmeval.yaml
math.yaml .../tests
math.yaml prompts
llmeval.yaml
report-0.yaml
multirun > 2023-09-30 > 09-39-05 > 0 > report-0.yaml > prompts
PROBLEMS
TESTING  DEBUG CONSOLE  PORTS  OUTPUT
zsh
Ln 1, Col 1  Spaces: 2  UTF-8  LF  YAML  No JSON Schema  78%

### Refining the math evaluation

The test compares the response with “9” after stripping leading whitespace. After a verbose answer failed, Nielsen restores the instruction to return only the answer and reruns the evaluation.

[5:16](https://www.youtube.com/watch?v=fiXjTif1nS4&t=316s) [The test definition is open again above the terminal's report-generation output.](https://assets.ai.engineer/sites/aiecode2025/sha256/ee431fcef4ae1a8e1004d685156ce4c231c6b38f2c3ad028148544104b54b0b4.webp)

# Copyright Log10, Inc 2023

metrics:
  - name: exact_match
    code: |
      metric = actual.lstrip() == expected
      result = metric
references:
  - input:
      a: 4
      b: 5
    expected: "9"

Interface text:
EXPLORER
AI.ENGINEER
multirun/2023-09-30/09-39-05
0
1
2
multirun.yaml
report.md
prompts
venv
llmeval.yaml
math.yaml …/tests
math.yaml prompts
llmeval.yaml
report-0.yaml
report.md
prompts > tests > math.yaml > references > 0
PROBLEMS
TESTING  DEBUG CONSOLE  PORTS  OUTPUT
[2023-09-30 09:39:07,707] [HYDRA] #2 : prompts=math prompts.model=claude-2 n_tries=1
[2023-09-30 09:39:09,067] [httpx] [INFO] - HTTP Request: POST https://api.anthropic.com/v1/complete "HTTP/1.1 200 OK"
(venv) ➜ ai.engineer llmeval report
Generating report for: multirun/2023-09-30/09-39-05
Processing job at date: 2023-09-30 time: 09-39-05 job id: 0 run id: 0...
Processing job at date: 2023-09-30 time: 09-39-05 job id: 1 run id: 0...
Processing job at date: 2023-09-30 time: 09-39-05 job id: 2 run id: 0...
Report written to multirun/2023-09-30/09-39-05/report.md
Some tests failed
(venv) ➜ ai.engineer
Ln 13, Col 1  Spaces: 2  UTF-8  LF  YAML  78%

[5:33](https://www.youtube.com/watch?v=fiXjTif1nS4&t=333s) [YAML prompt asks for the sum of two variables and only the answer.](https://assets.ai.engineer/sites/aiecode2025/sha256/3fd034744467bcd03a3eeded745b48e58d8baa679e2d44d61d54983af9fcda44.webp)

# Copyright Log10, Inc 2023

defaults:
  - tests/math
name: math
model: gpt-4-16k
temperature: 0.0
variables:
  - name: a
  - name: b
messages:
  - role: "human"
    content: "What is {a} + {b}? Only return the answer without any explanation"

Interface text:
EXPLORER
AI.ENGINEER
multirun/2023-09-30/09-39-05
0
1
2
multirun.yaml
report.md
prompts
tests
capitals.yaml
code_metrics_rollup.yaml
code.yaml
grading.yaml
math.yaml
noop.yaml
ping.yaml
summary.yaml
capitals.yaml
code_metrics_rollup.yaml
code.yaml
grading.yaml
math.yaml
noop.yaml
ping.yaml
summary.yaml
math.yaml …/tests
math.yaml prompts
llmeval.yaml
report-0.yaml
report.md
prompts > math.yaml > messages > 0 > content
PROBLEMS
TESTING  DEBUG CONSOLE  PORTS  OUTPUT
(venv) ➜ ai.engineer llmeval report
Generating report for: multirun/2023-09-30/09-39-05
Processing job at date: 2023-09-30 time: 09-39-05 job id: 0 run id: 0...
Processing job at date: 2023-09-30 time: 09-39-05 job id: 1 run id: 0...
Processing job at date: 2023-09-30 time: 09-39-05 job id: 2 run id: 0...
Report written to multirun/2023-09-30/09-39-05/report.md
Some tests failed
[illegible] ai.engineer llmeval run prompts=math n_tries=1
[illegible]ning evaluation...
Ln 13, Col 80 (47 selected)  Spaces: 2  UTF-8  LF  YAML  No JSON Schema  78%

### Running the math evaluation

The presenter restores the instruction to return only the answer, then reruns the math evaluation across three models.

[5:40](https://www.youtube.com/watch?v=fiXjTif1nS4&t=340s) [The math prompt appears above terminal output launching evaluations for three models.](https://assets.ai.engineer/sites/aiecode2025/sha256/b763b965e158b2d6beef441e7548e00bcad8a569286bb05df864f4bfdb38ba18.webp)

# Copyright Log10, Inc 2023

defaults:
  - tests/math
name: math
model: gpt-4-16k
temperature: 0.0
variables:
  - name: a
  - name: b
messages:
  - role: "human"
    content: "What is {a} + {b}? Only return the answer without any explanation"

Terminal:
🔴🔴🔴 Some tests failed 🔴🔴🔴
(venv) ➜ ai.engineer llmeval run prompts=math n_tries=1
Running evaluation...
Ran evaluation successfully.
[2023-09-30 09:40:14,870][HYDRA] Launching 3 jobs locally
[2023-09-30 09:40:14,870][HYDRA] #0 : prompts=math prompts.model=gpt-3.5-turbo n_tries=1
[illegible] 09:40:15,304][HYDRA] #1 : prompts=math prompts.model=gpt-4 n_tries=1
[illegible] 09:40:16,129][HYDRA] #2 : prompts=math prompts.model=claude-2 n_tries=1
[illegible] 09:40:17,280][httpx][INFO] - HTTP Request: POST https://api.anthropic.com/v1/complete "HTTP/1.1 200 OK"
(venv) ➜ ai.engineer

Interface text:
EXPLORER
AI.ENGINEER
multirun
prompts
  tests
    capitals.yaml
    code_metrics_rollup.yaml
    code.yaml
    grading.yaml
    math.yaml
    noop.yaml
    ping.yaml
    summary.yaml
  capitals.yaml
  code_metrics_rollup.yaml
  code.yaml
  grading.yaml
  math.yaml
  noop.yaml
  ping.yaml
  summary.yaml
venv
llmeval.yaml
Tabs: math.yaml …/tests; math.yaml prompts; llmeval.yaml; report-0.yaml; report.md
prompts > math.yaml > messages > 0 > content
PROBLEMS
TESTING  DEBUG CONSOLE  PORTS  OUTPUT
zsh
Ln 13, Col 80 (47 selected)  Spaces: 2  UTF-8  LF  YAML  No JSON Schema

### Math evaluation results

All three models return 9 for 4 + 5 and pass exact match. The terminal also reports failures across the generated reports, even though this run passed.

[5:49](https://www.youtube.com/watch?v=fiXjTif1nS4&t=349s) [A report shows three models returning 9 for 4 plus 5 and passing exact-match checks.](https://assets.ai.engineer/sites/aiecode2025/sha256/0f7cf7a3d7818b36ecbab23ef6c2b47bc66d4b517d679a4065203df77eb61e86.webp)

✅ llmeval report

Date 2023-09-30 09:40:14

✅ math

[{'role': 'human', 'content': 'What is {a} + {b}? Only return the answer without any explanation'}]

model | temperature | a | b | Expected | Actual #0 | exact_match (metric) #0 | exact_match (pass / fail) #0
✅ gpt-3.5-turbo | 0.0 | 4 | 5 | 9 | 9 | True | ✅
✅ gpt-4 | 0.0 | 4 | 5 | 9 | 9 | True | ✅
✅ claude-2 | 0.0 | 4 | 5 | 9 | 9 | True | ✅

Create an account on log10.io to view the logs of this test when you re-run it with passing log10 credentials.

Interface text:
EXPLORER
AI.ENGINEER
multirun/2023-09-30
09-39-05
09-40-14
0
1
2
multirun.yaml
report.md
prompts
tests
capitals.yaml
code_metrics_rollup.yaml
code.yaml
grading.yaml
math.yaml
noop.yaml
ping.yaml
summary.yaml
capitals.yaml
code_metrics_rollup.yaml
code.yaml
grading.yaml
math.yaml
noop.yaml
PROBLEMS

Tabs: math.yaml .../tests; math.yaml prompts; report.md .../09-40-14; llmeval.yaml; report-0.yaml; report.md .../09-39-05
Breadcrumb: multirun > 2023-09-30 > 09-40-14 > report.md

TESTING  DEBUG CONSOLE  PORTS  OUTPUT
Processing job at date: 2023-09-30 time: 09-40-14 job id: 1 run id: 0...
Processing job at date: 2023-09-30 time: 09-40-14 job id: 2 run id: 0...
Report written to multirun/2023-09-30/09-40-14/report.md
Generating report for: multirun/2023-09-30/09-39-05
Processing job at date: 2023-09-30 time: 09-39-05 job id: 0 run id: 0...
Processing job at date: 2023-09-30 time: 09-39-05 job id: 1 run id: 0...
Processing job at date: 2023-09-30 time: 09-39-05 job id: 2 run id: 0...
Report written to multirun/2023-09-30/09-39-05/report.md
🔴🔴🔴 Some tests failed 🔴🔴🔴
(venv) ➜ ai.engineer

### Python code generation prompt

The prompt asks for Python code without surrounding explanation. The demo then checks whether the output can be parsed as a Python program.

[6:03](https://www.youtube.com/watch?v=fiXjTif1nS4&t=363s) [YAML prompt requests a Python function and specifies code-only output.](https://assets.ai.engineer/sites/aiecode2025/sha256/fb69972d28d77d055ecbeddacb0a648b48e4d0dea3a32619304a356d7a279162.webp)

# Copyright Log10, Inc 2023

defaults:
  - tests/code
name: code
model: gpt-4-16k
temperature: 0.0
variables:
  - name: task
messages:
  - role: "human"
    content: |
      Generate a python function which {task}.
      Do not include any surrounding explanation or context.
      Strictly only python code which can be passed to a python interpreter.
      Do not prepend "python".

Interface text:
EXPLORER
AI.ENGINEER
multirun
prompts
  tests
    capitals.yaml
    code_metrics_rollup.yaml
    code.yaml
    grading.yaml
    math.yaml
    noop.yaml
    ping.yaml
    summary.yaml
  capitals.yaml
  code_metrics_rollup.yaml
  code.yaml
  grading.yaml
  math.yaml
  noop.yaml
  ping.yaml
  summary.yaml
venv
llmeval.yaml

Tabs: math.yaml …/tests; code.yaml; math.yaml prompts; report.md …/09-40-14; llmeval.yaml; report.md …/09-39-05
prompts > code.yaml > …
PROBLEMS
TESTING  DEBUG CONSOLE  PORTS  OUTPUT
Processing job at date: 2023-09-30 time: 09-39-05 job id: 1 run id: 0...
Processing job at date: 2023-09-30 time: 09-39-05 job id: 2 run id: 0...
Report written to multirun/2023-09-30/09-39-05/report.md
Generating report for: multirun/2023-09-30/09-41-54
Processing job at date: 2023-09-30 time: 09-41-54 job id: 0 run id: 0...
Processing job at date: 2023-09-30 time: 09-41-54 job id: 1 run id: 0...
Processing job at date: 2023-09-30 time: 09-41-54 job id: 2 run id: 0...
Report written to multirun/2023-09-30/09-41-54/report.md
🔴 Some tests failed 🔴🔴🔴
(venv) ➜ ai.engineer
Ln 1, Col 1  Spaces: 2  UTF-8  LF  YAML  No JSON Schema  79%

### Code evaluation results

The report shows passing checks for addition and inverse-square-root code from GPT-3.5 Turbo, GPT-4, and Claude 2. The test checks whether the response is a parseable Python program; the terminal also reports some failed tests.

[6:43](https://www.youtube.com/watch?v=fiXjTif1nS4&t=403s) [A report lists passing results for addition and inverse-square-root code tasks across three models.](https://assets.ai.engineer/sites/aiecode2025/sha256/e6c8b7c81dfebec405f4c0f74a2c1e87a1cc4cdfcb08b9cefc6159e153a6fe4e.webp)

✅ gpt-3.5-turbo | 0.0 | adds two numbers | N/A | def add_numbers(a, b): return a + b | True | ✅
✅ gpt-3.5-turbo | 0.0 | finds the inverse square root of a number | N/A | def inverse_square_root(x): return 1 / (x ** 0... | True | ✅
✅ gpt-4 | 0.0 | adds two numbers | N/A | def add_two_numbers(num1, num2): return num1 +... | True | ✅
✅ gpt-4 | 0.0 | finds the inverse square root of a number | N/A | def inverse_square_root(number): return 1 / (n... | True | ✅
✅ claude-2 | 0.0 | adds two numbers | N/A | def add(a, b): return a + b | True | ✅
✅ claude-2 | 0.0 | finds the inverse square root of a number | N/A | def invsqrt(x): return 1/math.sqrt(x) | True | ✅

Create an account on log10.io to view the logs of this test when you re-run it with passing log10 credentials.

Interface text:
EXPLORER
AI.ENGINEER
multirun/2023-09-30
prompts
  tests
    capitals.yaml
    code_metrics_rollup.yaml
    code.yaml
    grading.yaml
    math.yaml
    noop.yaml
    ping.yaml
    summary.yaml
  capitals.yaml
  code_metrics_rollup.yaml
  code.yaml
  grading.yaml
  math.yaml
  noop.yaml
  ping.yaml
  summary.yaml
venv
llmeval.yaml

multirun > 2023-09-30 > 09-41-54 > report.md
PROBLEMS
TESTING  DEBUG CONSOLE  PORTS  OUTPUT
Processing job at date: 2023-09-30 time: 09-39-05 job id: 1 run id: 0...
Processing job at date: 2023-09-30 time: 09-39-05 job id: 2 run id: 0...
Report written to multirun/2023-09-30/09-39-05/report.md
Generating report for: multirun/2023-09-30/09-41-54
Processing job at date: 2023-09-30 time: 09-41-54 job id: 0 run id: 0...
Processing job at date: 2023-09-30 time: 09-41-54 job id: 1 run id: 0...
Processing job at date: 2023-09-30 time: 09-41-54 job id: 2 run id: 0...
Report written to multirun/2023-09-30/09-41-54/report.md
🔴🔴 Some tests failed 🔴🔴🔴
[illegible] ai.engineer

### Model-based diagram grading criteria

The demo defines a 1–5 rubric for another model to grade Mermaid diagrams and return a reason. Lower scores flag incorrect or inconsistent diagrams; higher scores indicate fewer changes are needed.

[7:00](https://www.youtube.com/watch?v=fiXjTif1nS4&t=420s) [Code editor showing a one-to-five rubric for evaluating Mermaid diagrams.](https://assets.ai.engineer/sites/aiecode2025/sha256/ee6370e796fac3a37bbcbbc83c98c8553bafa266c7702845707fd008641cca29.webp)

content="""
You are a critic of diagrams in mermaid.js.

You will be given a user prompt asking about something that you know about or asking to represent custom data in a diagram.

Where:

1 means the diagram looks very wrong, or you think it might not compile with mermaid.js.

2 means its inconsistent with your knowledge and is too far from the correct form that it should just be discarded.

3 means its okay, but needs some adjustments to better answer the user. In this case output an improvement suggestion.

4 means its good. Give an optional suggestion.

5 means its perfect. You wouldn't change anything about it.
    """,
  ),
  HumanMessage(content="""
The output should be formed strictly in json.
Examples of output:

## Example 1

{
  "grade": 5
  "reason": "This is a great diagram because it clearly communicates the meaning of the system."
}

## Example 2

Interface text:
EXPLORER
AI.ENGINEER
multirun/2023-09-30
prompts
  tests
    capitals.yaml
    code_metrics_rollup.yaml
    code.yaml
    grading.yaml
    math.yaml
    noop.yaml
    ping.yaml
    summary.yaml
  capitals.yaml
  code_metrics_rollup.yaml
  code.yaml
  grading.yaml
  math.yaml
  noop.yaml
  ping.yaml
  summary.yaml
venv
llmeval.yaml

prompts > tests > grading.yaml > ...
PROBLEMS
TESTING  DEBUG CONSOLE  PORTS  OUTPUT
Processing job at date: 2023-09-30 time: 09-39-05 job id: 1 run id: 0...
Processing job at date: 2023-09-30 time: 09-39-05 job id: 2 run id: 0...
Report written to multirun/2023-09-30/09-39-05/report.md
Generating report for: multirun/2023-09-30/09-41-54
Processing job at date: 2023-09-30 time: 09-41-54 job id: 0 run id: 0...
Processing job at date: 2023-09-30 time: 09-41-54 job id: 1 run id: 0...
Processing job at date: 2023-09-30 time: 09-41-54 job id: 2 run id: 0...
Report written to multirun/2023-09-30/09-41-54/report.md
Some tests failed
(venv) ➜ ai.engineer
Ln 1, Col 1  Spaces: 2  UTF-8  LF  YAML  No JSON Schema  79%

### Diagram grading examples

The grading prompt shows example scores and reasons for Mermaid diagrams. The code sends the prompt to a model, parses its JSON response, and extracts the grade.

[7:07](https://www.youtube.com/watch?v=fiXjTif1nS4&t=427s) [Code editor showing example grades, grading instructions, and a model call.](https://assets.ai.engineer/sites/aiecode2025/sha256/34859f2a4f07cfb18594ee0c4debf8ab2eeef0e11c2df1a97b098dd1f8eb7845.webp)

## Example 1

{
  "grade": 5
  "reason": "This is a great diagram because it clearly communicates the meaning of the system."
}

## Example 2

{
  "grade": 1
  "reason": "This diagram is not very good, as it is convoluted and isn't well formed mermaid.js code."
}

# Instructions

Please grade the following diagram code:

###

{code}

###

Based on the 1-5 criteria defined above.
    """)
]
output = llm(messages)
output = json.loads(output.content)

metric = output.get("grade", "N/A")

Interface text:
EXPLORER
AI.ENGINEER
multirun/2023-09-30
prompts
tests
capitals.yaml
code_metrics_rollup.yaml
code.yaml
grading.yaml
math.yaml
noop.yaml
ping.yaml
summary.yaml
capitals.yaml
code_metrics_rollup.yaml
code.yaml
grading.yaml
math.yaml
noop.yaml
ping.yaml
summary.yaml
venv
llmeval.yaml
PROBLEMS
TESTING  DEBUG CONSOLE  PORTS  OUTPUT
Processing job at date: 2023-09-30 time: 09-39-05 job id: 1 run id: 0...
Processing job at date: 2023-09-30 time: 09-39-05 job id: 2 run id: 0...
Report written to multirun/2023-09-30/09-39-05/report.md
Generating report for: multirun/2023-09-30/09-41-54
Processing job at date: 2023-09-30 time: 09-41-54 job id: 0 run id: 0...
Processing job at date: 2023-09-30 time: 09-41-54 job id: 1 run id: 0...
Processing job at date: 2023-09-30 time: 09-41-54 job id: 2 run id: 0...
Report written to multirun/2023-09-30/09-41-54/report.md
Some tests failed
Ln 1, Col 1  Spaces: 2  UTF-8  LF  YAML  No JSON Schema

### Thanks and questions

The closing slide points to the llmeval installation docs and invites questions.

[9:04](https://www.youtube.com/watch?v=fiXjTif1nS4&t=544s) [Closing slide with a documentation reference, QR code, and speaker contact details.](https://assets.ai.engineer/sites/aiecode2025/sha256/f0b3136722c56fbb6ddf145bde85ca0dc9fd4344faf979284835a997e979253c.webp)

Thanks!
Any questions?

https://log10.io/docs/evaluation/installation

Introducing llmeval today so your team can ship reliable LLM products

You can find me at:
@NiklasQuarfot on X
[redacted]

LOG10

## Full transcript

[0:00](https://www.youtube.com/watch?v=fiXjTif1nS4&t=0s) [upbeat music] Hi, this is Niklas.

[0:17](https://www.youtube.com/watch?v=fiXjTif1nS4&t=17s) I'm CTO and co-founder of Log10, and we want to talk about how you can scale reliability of L- LLM applications using, uh, a new tool that we've built. During this year, I think we all can agree that there's been, like, this kind of craze in the industry, and we've been rolling out a ton of intelligence features based

[0:37](https://www.youtube.com/watch?v=fiXjTif1nS4&t=37s) on GPT. And we're now kind of finding ourselves in a now what moment because without knowing what good means in a generative setting, it's really hard and risky to evolve your applications, like changing your prompts, configurations, let alone considering going from one model provider to another to more advanced use cases like self-hosting or fine-tuning.

[1:02](https://www.youtube.com/watch?v=fiXjTif1nS4&t=62s) We want to introduce a new tool today called llmeval that enables, uh, teams to ship reliable LLM products.

[1:13](https://www.youtube.com/watch?v=fiXjTif1nS4&t=73s) It, it is command line tool that you can run locally, and with these four, uh, lines of code, uh, you should be good to go. Um, the initialization creates a folder structure, um, and best practices for storing prompts and, and tests.

[1:33](https://www.youtube.com/watch?v=fiXjTif1nS4&t=93s) And then this is based on a super configurable system from Meta called Hydra. So you could basically extend it to your heart's desire, and the metrics that we have wired up are in Python, so they could be any logic.

[1:50](https://www.youtube.com/watch?v=fiXjTif1nS4&t=110s) Could be call out to all the LLMs, whatever you want. And after these evaluations have been run, you generate some reports that basically gives you, like, a brief overview of how the entire app and all the tests are looking, but still supports flexible test criteria because, like, these models are very fussy.

[2:11](https://www.youtube.com/watch?v=fiXjTif1nS4&t=131s) It's very hard to say with a guarantee that it's gonna be one or the other, but it's fairly safe to say that the majority cases, or say three out of five, should pass.

[2:22](https://www.youtube.com/watch?v=fiXjTif1nS4&t=142s) And we're gonna jump into command line and taking a look.

[2:28](https://www.youtube.com/watch?v=fiXjTif1nS4&t=148s) We're just gonna create a directory for today.

[2:35](https://www.youtube.com/watch?v=fiXjTif1nS4&t=155s) And go into this directory and create ourselves a virtual environment.

[2:42](https://www.youtube.com/watch?v=fiXjTif1nS4&t=162s) From here, we're gonna install llmeval and initialize the folder structure. What we should be able to see here is

[2:58](https://www.youtube.com/watch?v=fiXjTif1nS4&t=178s) a directory structure where we have our prompts. Let's say a simple case could be this, where we have this message template saying like, "What is A plus B?" Only return the answer without any explanation.

[3:11](https://www.youtube.com/watch?v=fiXjTif1nS4&t=191s) So in this case, we know that we have to prompt engineer further in order to get an exact output. 'Cause let's take a look at how the test looks like.

[3:21](https://www.youtube.com/watch?v=fiXjTif1nS4&t=201s) In this case, we're taking, like, the actual output from the LLM and comparing it with the expected, and this is like a strict comparison. [clears throat]

[3:31](https://www.youtube.com/watch?v=fiXjTif1nS4&t=211s) What we had taken the liberty to do is to strip any spaces that might be... come from, from the left, and that's because some models, in this case, Claude, tends to prepend spaces.

[3:44](https://www.youtube.com/watch?v=fiXjTif1nS4&t=224s) And so it's things like that that you have to watch out for. Then we have the metric, which could be any metric that you wanna surface in the report, and then the result, which is then a pass or fail.

[3:56](https://www.youtube.com/watch?v=fiXjTif1nS4&t=236s) And in this case, we wanna add four and five, and we expect it to be nine.

[4:01](https://www.youtube.com/watch?v=fiXjTif1nS4&t=241s) And I'm just gonna try to run this test here and try to re-revert some of the prompt engineering that we did earlier. So I'm gonna re-remove

[4:12](https://www.youtube.com/watch?v=fiXjTif1nS4&t=252s) only return the answer without any explanation. And the way you can start it is llmeval run.

[4:24](https://www.youtube.com/watch?v=fiXjTif1nS4&t=264s) But if you want to override anything-- If you just do llmeval run, it runs everything. But if you do like prompts=math, then it's only gonna run the math example.

[4:34](https://www.youtube.com/watch?v=fiXjTif1nS4&t=274s) If you do n tries one, then it's just gonna do one sample. By default, we do five samples, so, so we get, like, a better read on the stability of, of each test, but it might be too much for you.

[4:49](https://www.youtube.com/watch?v=fiXjTif1nS4&t=289s) But you can override anything. You can find these default settings here in the llmeval YAML. And but let's try to run this and see what happens. And so this ran across Claude, GPT-4, and, and GPT-3.5 once.

[5:07](https://www.youtube.com/watch?v=fiXjTif1nS4&t=307s) So we can go in and generate a report

[5:12](https://www.youtube.com/watch?v=fiXjTif1nS4&t=312s) and see, like, actually something failed. What was it that failed? So let's take a look at the output here. And in this case, because we've removed our prompt engineering, GPT-3.5 starts being a bit chatty and says like, "Four plus five equals nine."

[5:27](https://www.youtube.com/watch?v=fiXjTif1nS4&t=327s) Claude does something similar. So it kind of writes out the... writes out the equation. And now I'm gonna try to revert

[5:36](https://www.youtube.com/watch?v=fiXjTif1nS4&t=336s) and see... Let's, let's get this in. And we try to run one more time.

[5:45](https://www.youtube.com/watch?v=fiXjTif1nS4&t=345s) Great. Now, when we generate the report, it could say some tests failed, but the most recent test that ran passed. So when you do the report, it's gonna generate a summary.

[5:55](https://www.youtube.com/watch?v=fiXjTif1nS4&t=355s) It could generate a report per, per run, but then also say, overall, was there anything that, that failed out of these reports?

[6:04](https://www.youtube.com/watch?v=fiXjTif1nS4&t=364s) If you wanna go a bit more advanced, let's say you wanna use tools, y- we c- we have an example here where we are generating some Python code, and again, we had to add a number of different, um, clauses to make sure that it only outputs Python.

[6:19](https://www.youtube.com/watch?v=fiXjTif1nS4&t=379s) It tends to be very happy generating, um, s- surrounding explanations.

[6:24](https://www.youtube.com/watch?v=fiXjTif1nS4&t=384s) Uh, so in this case, we are gonna see whether or not, um, it returns an actual Python program that could be, that could be parsed. So let's try to run that.

[6:37](https://www.youtube.com/watch?v=fiXjTif1nS4&t=397s) If you go in and take a look at this report,

[6:41](https://www.youtube.com/watch?v=fiXjTif1nS4&t=401s) you can see that these tests actually end up, end up passing our tool use. And to, to round up,

[6:48](https://www.youtube.com/watch?v=fiXjTif1nS4&t=408s) we have model-based evaluation as well, where you can test using other models. And so in this case, say with grading, we can go in and define, like, a full set of criteria.

[7:02](https://www.youtube.com/watch?v=fiXjTif1nS4&t=422s) Here, we're about evaluating Mermaid diagrams, giving a score between one and five, and the reason, and that, that is also supported in llmeval. One thing about the previous approach is that it takes quite an amount, amount of work to set up these tests and gather your test cases.

[7:21](https://www.youtube.com/watch?v=fiXjTif1nS4&t=441s) And one really compelling answer to evaluation has been model-based evaluation, and it's, uh, it's a setting where you have, well, typically a larger model discriminate or kinda grade or be a judge over the output from another LLM.

[7:36](https://www.youtube.com/watch?v=fiXjTif1nS4&t=456s) And that makes it so you can get more nuanced output like pass/fail or a grade from one to five or preferences between different options and its reasoning behind it.

[7:47](https://www.youtube.com/watch?v=fiXjTif1nS4&t=467s) There's a number of pitfalls, unfortunately, around this approach around biases towards the output from the model itself. If you're sweeping different models, they tend to prefer their own, own output.

[8:00](https://www.youtube.com/watch?v=fiXjTif1nS4&t=480s) They're not very good at giving, uh, point scores, saying anything between zero and one, or larger scores between zer- zero and 100. [clears throat]

[8:09](https://www.youtube.com/watch?v=fiXjTif1nS4&t=489s) But there are different ways where you can start increasing the accuracy of the kind of feedback that's been generated.

[8:17](https://www.youtube.com/watch?v=fiXjTif1nS4&t=497s) And [clears throat] we've been working on this, where you basically start bridging between model-based and human feedback. So instead of removing the human completely from the feedback, you start taking in all feedback that might have been given prior and start modeling it and say, like, if you have all the feedback from John, then we create an Auto John that

[8:38](https://www.youtube.com/watch?v=fiXjTif1nS4&t=518s) will start create-- generating feedback per review, um, for any incoming completions. And so in this case here, we have two pieces of feedback that's been already given by human.

[8:48](https://www.youtube.com/watch?v=fiXjTif1nS4&t=528s) See here, it was all just, like, a score of five, or here, just, like, a bit more nuanced. But here, we are kind of pending feedback. And if you click this, we have AI suggested an answer to, to this.

[9:06](https://www.youtube.com/watch?v=fiXjTif1nS4&t=546s) And that's all I had today. Um, if you want to get started on, um, llmeval, we have our documentation at our usual documentation site. And you can find me at, uh, Niklas Crafoord at, on, uh, X, or formerly, formerly known as Twitter, or shoot me an email at, uh, [REDACTED:email_address].

[9:25](https://www.youtube.com/watch?v=fiXjTif1nS4&t=565s) Thank you. [outro music]
