AI Engineer World's Fair 2025
Practical tactics to build reliable AI apps — Dmitry Kuchin, Multinear
Read the talk
Build AI evaluations around the job users need done
Reliable AI applications need tests that capture user goals, expose specific failures, and make changes to prompts, models, logic, and data safe to compare.
From a talk by Dmitry Kuchin
Before you start: Familiarity with prompts, language models, and basic software testing is helpful; no evaluation framework experience is required.
A working prompt is only the beginning
How do you turn an impressive AI proof of concept into an application that reliably does its job? Dmitry Kuchin approaches that question through experience in startup and enterprise leadership, followed by GenAI projects ranging from proofs of concept to production systems. His focus is the practical work between an initial success and behavior a team can depend on.
The conventional development sequence is familiar: design, develop, test, deploy. With AI, a prompt can produce a convincing result almost immediately. Kuchin illustrates the gap with a POC that works fifty percent of the time; that is an illustrative scenario, not a measured benchmark. Getting the remaining cases right demands continuous experimentation because model behavior is nondeterministic. You try another prompt, another model, or another approach—and every change to code, logic, prompts, models, or source data can affect results unexpectedly. Reliability therefore requires a way to judge each experiment, not just a successful initial demonstration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with the outcome the user needs
Once development becomes experimental, generic quality metrics can seem like the obvious place to start. Groundedness, factuality, and bias scores describe aspects of an answer, but they do not by themselves establish that the application works for its users. A change can improve one of those scores without solving the user's problem.
Kuchin describes asking a former colleague building a support bot at Wix how the team knew it was working well. The discussion began with factuality and similar metrics. As they examined the product goal, the important outcome became the rate at which users escalated from the AI bot to human support. A response could be thoroughly grounded and still fail to provide the answer the user needed. Escalation mattered in that conversation; it was not proposed as a universal measure for every support system.
Work backward from real scenarios to specific evaluation criteria. Identify what users are trying to accomplish, what a successful interaction looks like, and how to reproduce that interaction in a test. Kuchin's objection to universal evaluations is that an average or generic score cannot substitute for these product-specific requirements. The desired business outcome tells you what matters; the individual tests must make that outcome concrete enough to assess.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn a bank FAQ into answer requirements
Consider a bank support bot with FAQ material explaining password resets. Customer support is particularly difficult to evaluate, in Kuchin's experience, because correctness depends on the information needed for each specific question. His method uses a capable model—he mentions o3, after initially naming o1—to work backward from the material: what questions can it answer, and what must a correct answer contain?
In the password-reset example, the answer needs to explain mobile validation through an SMS code. It must also include the fallback: contact support if the user does not have a mobile number. Leaving out either required element makes the answer incomplete under this checklist. The test is more precise than asking whether the response sounds factual: it identifies the information that this particular answer must preserve.
A small TypeScript representation makes those requirements explicit. A judge would assess whether the response satisfies each criterion; the aggregation below applies the strict all-items rule to those judgments.
typescript
type Criterion = {
id: string;
requirement: string;
};
const passwordReset = {
question: "How do I reset my password?",
criteria: [
{
id: "sms-validation",
requirement: "Explain mobile validation using an SMS code.",
},
{
id: "no-mobile-fallback",
requirement: "Explain that users without a mobile number can contact support.",
},
] satisfies Criterion[],
};
function passesChecklist(
criteria: readonly Criterion[],
judgments: Readonly<Record<string, boolean>>,
): boolean {
return criteria.every(({ id }) => judgments[id] === true);
}
const incompleteAnswerJudgments = {
"sms-validation": true,
"no-mobile-fallback": false,
};
const passes = passesChecklist(
passwordReset.criteria,
incompleteAnswerJudgments,
); // false
The two entries capture the requirements discussed here, rather than a complete password-reset policy. A missing judgment also fails this check instead of silently counting as a pass.
Build many such cases from the reference material, then vary how users ask the questions. Give the model enough context about the personas it should represent: different people can express the same underlying request very differently. The wording changes, but the required answer content should remain stable.
The Multinear demonstration brings the test's input, output, and question-specific criteria into one view. Kuchin presents the platform as open source and emphasizes the method over the tool. For the password-reset question, he proposes generating fifty wording variants and checking each against the same required answer elements. This is a proposed test design, not a reported success rate. The all-items rule is the correctness target in this example, not a claim that every platform configuration enforces it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Diagnose failures before changing the application
Evaluations belong beside the first POC, not at the end of development. Build the initial implementation, define an initial set of tests, and run them. Some cases will succeed and others will fail. An average can summarize that run, but it does not explain what to change. The useful evidence lives in the individual input, output, and failed criterion.
A failed evaluation has two possible sources to investigate:
- The test is wrong. Its definition may not correctly describe the required behavior.
- The application is wrong. Its response may fail a valid requirement.
That distinction determines the next experiment. You might repair the evaluation, change the model, revise application logic, edit the prompt, or change the data used to answer the question. Tests should expose enough detail to make that choice an informed one.
A local fix is not yet a reliable improvement. Kuchin repeatedly encounters prompt changes that repair one test while breaking a previously successful case. Existing evaluations make those regressions visible. The workflow is therefore a loop:
- Build the first application version and its initial evaluations.
- Run the evaluations and inspect individual failures.
- Change the implementation, repair faulty tests, or add missing cases.
- Rerun the suite, including cases that previously passed.
- Repeat until the benchmark adequately captures the requirements of this application at this point in time.
The result is a usable baseline: a set of expectations against which subsequent changes can be compared.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the baseline to test simpler alternatives
Once the benchmark is trustworthy enough for the application, optimization becomes a set of testable questions. Could GPT-4o mini preserve the required behavior of GPT-4o? Would a simpler retrieval solution suffice instead of GraphRAG? Does an agentic approach improve the outcome enough to justify its additional time and inference cost? Could the logic be simplified across the application, or only in one portion?
These are proposed comparisons, not demonstrated equivalences or savings. The benchmark supplies a common basis for deciding whether an alternative still does the required job. It makes it possible to investigate cost and complexity without relying only on a few appealing outputs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Match the evaluation method to the task
The experimentation loop transfers across applications; the evaluation itself does not. Different tasks expose correctness in different ways. Kuchin distinguishes three common cases:
| Application | Evaluation method | Expected behavior comes from |
|---|---|---|
| Support bot | LLM-as-a-judge | Question-specific answer criteria |
| Text-to-SQL or graph-database queries | Queries against a mock database | Known data in a representative schema |
| Call-center conversation classifier | Direct category matching | The expected rubric category |
For database applications, the mock database should reproduce the relevant schema and contain data whose answers are known. That gives a specific user question an expected result. For a classifier, a direct comparison with the expected category can be sufficient; it need not inherit the support bot's judging method.
Guardrails belong in the benchmark as well. For the support bot, include questions that should not be answered, questions requiring a different kind of response, and questions whose answers are absent from the available material. Define the expected handling of those cases and evaluate it through the same iterative process. Success includes respecting the boundaries of what the application should do.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make application behavior understandable
Evaluating the application as users actually use it gives frequent testing a purpose: each run checks whether an experiment preserved the behaviors that matter. Kuchin argues that this supports faster progress with fewer regressions because unexpected changes become visible while the team is still iterating.
He describes well-defined evaluations as producing a form of explainable AI. The useful interpretation is behavioral: the team can inspect what the application did on a known input, which requirement it met or missed, and how that changed between versions. This does not establish interpretability of the model's internal reasoning. The understanding comes from explicit expectations and inspectable results.
No particular platform is required. Kuchin says he built Multinear because he wanted support for this evaluation process from beginning to end, but another tool can serve the same purpose. What must remain is the connection between a user's task, its concrete correctness criteria, and the experiments used to improve the application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Open-source evaluation platform with setup instructions, task runners, configurable tests, and regression analysis.
Further reading
- The Strawberry TestArticle
A February 2025 Multinear experiment exploring how question rephrasing affects letter-counting results, token use, latency, and cost.
A graph-based retrieval implementation using knowledge graphs and community summaries, with guides for indexing and querying.
Research on when model reasoning traces omit information that influenced an answer, including the study's experimental limitations.
Updates since the talk
Current documentation for checklists, scoring thresholds, multiple evaluation metrics, and question rephrasing.
Read the complete timestamped transcript
- 0:00
[upbeat music] Welcome, everyone.
- 0:17
I'm going to talk about practical tactics to build, uh, reliable AI applications and why nobody does it this way yet.
- 0:27
Uh, a little bit about myself or why you should trust me. Um, I allowed fifteen years as a startup co-founder and CTO. Uh, I held executive positions for the last five years at, uh, several enterprises.
- 0:41
Uh, but most importantly, I spent last couple of years developing a lot of GenAI projects ranging from POCs to, uh, many production-level, uh, solutions and helped some companies to get it done.
- 0:59
And, uh, I've learned or distilled a way to, uh, make these applications reliable.
- 1:06
And there are quite a lot of, uh, tracks this, uh,
- 1:12
uh, this conference about evals and reliability, but, uh, to my surprise, nobody was talking about the most important things, and, uh, we are going to talk about it right now.
- 1:26
So, uh, standard software development life cycle is, uh, very standard, uh, simple. Uh, you design your solution, you develop it, you test it, and then eventually you deploy it.
- 1:38
And, uh, when people start doing, uh, POC with AI,
- 1:45
it sounds simple, like, uh, you can very easily do some prompt and, uh, models are very capable. But then you start, uh, facing some, uh, unexpected challenges. Uh, actually, like, you can easily do a POC that works fifty percent of the time, uh, but, like, making it do the same reliable work the rest of the fifty percent
- 2:09
is very hard, uh, because models are non-deterministic. And, uh, it starts requiring, uh, a data science approach, uh, continuous experimentation. You need to try this prompt. You need to try that model.
- 2:23
You need to try this approach, et cetera, et cetera. And, uh, everything in your solution, everything that, uh, represents your solution, which is your code, your logic, uh, the prompts that you use, the, the models that you use, the, the data that you base your solution on, changing anything of that impacts your, uh, solution in unexpected ways.
- 2:49
Um, people very often come to this, uh, to, to try solving this with the wrong approach. They start with, uh, data science metrics. They-- like, it sounds reasonable, right?
- 3:04
So it requires data science approach of experimentation, and, uh, people start, uh, measuring groundedness, factuality, bias, and other, uh, metrics that don't really help you to understand, uh, is your solution, uh, working the right way.
- 3:22
Does it, uh, does your latest change improved, uh, your solution in the right way for your users? Uh, for example, I've been talking to an ex-colleague that are building a customer support bot at Wix.
- 3:36
I asked him, "How do you know that your, uh, solution is working well?" He started talking about factuality and other, uh, data science metrics. Uh, that's again, I started to dig deeper, and then we just, uh, together figure out that the most important metric for them is, uh, the rate of, uh, moving from, uh, AI support bot,
- 4:00
like escalation, to a human support. If, uh, your solution, uh, hasn't able to answer the user with all this factuality, like it could be super grounded, but still not provide the right answer that the user expects, and, uh, this is what you actually need to test.
- 4:20
Um, and my experience was to start with real-world scenarios. So basically, you need to reverse engineer your metrics, and your metrics should be very, very specific to what your end goal.
- 4:36
So they should come from a product experience, from business outcomes. Uh, if your solution is customer support bot, you need to figure out what your users want and, uh, how you can mimic it.
- 4:48
And instead of measuring something, um, average or something generic, you need to measure a very specific criterias, uh, 'cause universal evals don't really work.
- 5:03
How do we do it? Uh, so for example, customer support bot, which is, by the way, one of the hardest, uh, things to do evals properly. Uh, let's say I have a, a bank, and a bank has, uh, FAQ materials which contain including, like, how do you reset your password.
- 5:24
Um, so what I usually do when I help my, uh, like, companies that I help them to build, uh, AI solutions, we start with, uh, reverse engineering, like how do we create evals based on that.
- 5:38
So in this case, I use LLM, and in most cases, I use LLM to come up with, uh, right evaluations. So here I can take, say, o1-- uh, o3 now, uh, and just- Reverse engineer what should be the user question, uh, that we know to answer based on these materials, and what should be the specific criteria that,
- 6:01
uh, these materials, uh, provide an answer for. And some of these criteria are quite important. So for example, here it says that, uh, uh, as part of the thing you, you need to receive a mobile validation, so you receive a SMS code, and, uh, it says that if you, uh, don't have a mobile number, then you can
- 6:20
reach, uh, support, et cetera, et cetera. Uh, if some of that information is missing from the answer, the answer would not be correct. But you need to be very specific about what exact information you need to see in the answer, and that is wh-- information is very specific to that specific question.
- 6:40
So you need to build, like, lots of evals, uh, from the materials in this case, uh, that mimic specific user questions that, uh, you need to be able to answer for.
- 6:55
Uh, how do we do it usually? Again, I work with, uh, smart models like o3, uh, and I, uh, provided enough context. I provided which personas are we trying to represent because you can make-- ask the same question in, uh, completely different ways depending on who is the persona asking.
- 7:16
Uh, yet you would expect exactly the same answer, so you need to account for it.
- 7:22
Um, so this is, uh, an example from, uh, the open source platform that we have that, uh, just helps to get it done. So if you look it up, Multinear, I'm not trying to sell you anything.
- 7:35
I'm not trying to, like, vendor lock in or whatever. It's completely open source, and if needed, I can just recreate it in a couple of days now with Cursor.
- 7:44
Uh, the point is in the approach, not in the platform. Uh, so for example, here we see that very same question, um, how do I reset my password? You see the-- what was the input, what was the output, and, uh, that specific criteria that I measured, uh, that specific question, how do I know if the answer is
- 8:08
correct? And now I can just reiterate and pro-- generate, like, fifty different variations of the same question and see if I still get the right answer. The-- if the answer matches all the checklist that I have for that specific answer.
- 8:25
Um, how the process usually works. Um, so contrary to, like, regular approach, you build your evals not at the end of the process, but in the very beginning of the process.
- 8:37
So you s-- just build your first version of the POC. You define the first version of your tests, evaluations. You run them, and you see what's going on. You, you will see that, uh, in some cases it will fail.
- 8:52
Uh, in some cases it will succeed. What's important is to, to look at the details, not just see the average numbers. The average numbers won't tell you anything, uh, won't tell you how to improve it.
- 9:05
If you actually look at the details of each evaluation, you'll see exactly why it's failing. It could be failing, um, because your test is not defined correctly. It could be failing because your, uh, solution is not working as it should be.
- 9:20
And, like, i-in order to do it, you may need to, uh, to do a change in, in-- like, you may change a model. You may change something on-- in your logic.
- 9:29
You may change a prompt or the data that you use in order to, uh, answer a question in our example. And, uh, basically what you do now is experimentation.
- 9:40
So you, you start running your experiment. You change something. You, you need to define these tests in a way that will, uh, help you to make an educated guess on, uh, what you need to change in order to, to do it.
- 9:55
In some cases it will work, in some cases it won't. But even if it works, uh, let's say you change something in your prompt and it fixed this test.
- 10:05
In my experience, in many cases it breaks, uh, something that used to work before. Uh, like, you, you, you have constant regressions, and if you don't have these evaluations, there is no way you'll be able to catch it on time.
- 10:19
So this is hugely important, and what actually happens is that, again, you build your first version. You build your first version of evals. Uh, you match them. You run these evals.
- 10:30
You improve something. You improve your evals or maybe add more evaluations, and then you, like, uh, continuously improve it until you reach some point where you are satisfied with your evals for this specific solution for that specific point of time.
- 10:46
And what actually happened is that you, you, you got your baseline. You got your benchmark that, uh, now you can start optimizing, and, uh, you have the confidence that the tests should be working.
- 11:00
So now you can try another model. Let's say, uh, well, what-- how can I try to see if Foro Mini will work the same way with Foro or not?
- 11:11
Uh, can I use the GraphRAG or can I try a simpler solution? Ca-- uh, should I have, uh, to use the agentic approach that ta-- like, may be better but, uh, requires more time, more, uh, inference cost, et cetera?
- 11:26
Or should I try to simplify the logic? Or maybe I can simplify the logic for a specific portion of the application, et cetera, et cetera. Having this benchmark, uh, allows you to do all these implementations, uh, with confidence.
- 11:41
But again, the, the most important part is, like, how do you reach this benchmark? And, uh, while the approach is, uh, pretty much the same, the evaluations that you need to build and how do you build your evaluations are completely different depending on the solution that you need to build.
- 11:58
Because, uh, the models are super capable right now, uh, so they allow you to build a huge variety of, uh, solutions. But each and every solution is quite, uh, different in terms of how do you, uh, evaluate it.
- 12:12
Uh, for support bot, you usually typically use LLM-as-a-judge, as I, uh, made an example. If you're building Text-to-SQL or text to graph database, then, uh, to my experience, the best way is to create a mock database that represents the, um, whatever, uh, database or databases that you need the-- your solution to work with.
- 12:35
They represent the same schema, and you have the mock data, so you know exactly, uh, what should expect on specific questions. Um, if you need to build some classifier for call center conversations, then your, uh, tests are like simple match whenever this is, this is the right rubric or not.
- 12:54
Uh, and the same appro-- uh, approach applies to guardrails. So, uh, getting back to the support... to, to the, uh, example of a customer support bot, uh, guardrails, you need to cover, uh, questions that should not be answered, or questions that should be answered in different ways, or questions that, uh, uh, the answers are not in the
- 13:14
material. So all of this you can put into your benchmark, just different type of benchmark, but it's pretty much the same approach.
- 13:23
Uh, so just to reiterate, uh, the key takeaways, you need to evaluate your apps the way your users actually use them, um, and, uh, avoid abstract metrics, uh, because these abstract metrics don't really measure anything important.
- 13:40
Uh, and the approach is, uh, through experimentation. So you run these evaluations frequently. You-- That allows you to have rapid progress with, uh, less regressions because testing frequently help you to, to catch these surprises.
- 13:55
Uh, but most importantly, what you get if you devi-- uh, define your evaluations correctly, you get your solution pretty much, uh, as kind of explainable AI because you know exactly what it does, you know exactly how it does it if you test it the right way.
- 14:13
Thank you very much. Uh, take a look at, uh, Multinear. Uh, that's a platform that you can use to, uh, run these evaluations. You can totally use any other platform.
- 14:25
The approach is quite simple. It doesn't require any specific platform. Uh, I've built Multinear just because no other platform helped me to do it this way, to, to help me with the process of evaluation, like end-to-end.
- 14:40
Um, I'm working on a startup that does reliable AI automation right now. Um, and, uh, yeah, thank you very much. [outro jingle]