Practical tactics to build reliable AI apps — Dmitry Kuchin, Multinear

How do you turn an impressive AI proof of concept into an application that reliably does its job? Dmitry Kuchin approaches that question through experience in startup and enterprise leadership, followed by GenAI projects ranging from proofs of concept to production systems. His focus is the practical work between an initial success and behavior a team can depend on.

The conventional development sequence is familiar: design, develop, test, deploy. With AI, a prompt can produce a convincing result almost immediately. Kuchin illustrates the gap with a POC that works fifty percent of the time; that is an illustrative scenario, not a measured benchmark. Getting the remaining cases right demands continuous experimentation because model behavior is nondeterministic. You try another prompt, another model, or another approach—and every change to code, logic, prompts, models, or source data can affect results unexpectedly. Reliability therefore requires a way to judge each experiment, not just a successful initial demonstration.

Once development becomes experimental, generic quality metrics can seem like the obvious place to start. Groundedness, factuality, and bias scores describe aspects of an answer, but they do not by themselves establish that the application works for its users. A change can improve one of those scores without solving the user's problem.

Kuchin describes asking a former colleague building a support bot at Wix how the team knew it was working well. The discussion began with factuality and similar metrics. As they examined the product goal, the important outcome became the rate at which users escalated from the AI bot to human support. A response could be thoroughly grounded and still fail to provide the answer the user needed. Escalation mattered in that conversation; it was not proposed as a universal measure for every support system.

Work backward from real scenarios to specific evaluation criteria. Identify what users are trying to accomplish, what a successful interaction looks like, and how to reproduce that interaction in a test. Kuchin's objection to universal evaluations is that an average or generic score cannot substitute for these product-specific requirements. The desired business outcome tells you what matters; the individual tests must make that outcome concrete enough to assess.

Consider a bank support bot with FAQ material explaining password resets. Customer support is particularly difficult to evaluate, in Kuchin's experience, because correctness depends on the information needed for each specific question. His method uses a capable model—he mentions o3, after initially naming o1—to work backward from the material: what questions can it answer, and what must a correct answer contain?

In the password-reset example, the answer needs to explain mobile validation through an SMS code. It must also include the fallback: contact support if the user does not have a mobile number. Leaving out either required element makes the answer incomplete under this checklist. The test is more precise than asking whether the response sounds factual: it identifies the information that this particular answer must preserve.

A small TypeScript representation makes those requirements explicit. A judge would assess whether the response satisfies each criterion; the aggregation below applies the strict all-items rule to those judgments.

type Criterion = {
  id: string;
  requirement: string;
};

const passwordReset = {
  question: "How do I reset my password?",
  criteria: [
    {
      id: "sms-validation",
      requirement: "Explain mobile validation using an SMS code.",
    },
    {
      id: "no-mobile-fallback",
      requirement: "Explain that users without a mobile number can contact support.",
    },
  ] satisfies Criterion[],
};

function passesChecklist(
  criteria: readonly Criterion[],
  judgments: Readonly<Record<string, boolean>>,
): boolean {
  return criteria.every(({ id }) => judgments[id] === true);
}

const incompleteAnswerJudgments = {
  "sms-validation": true,
  "no-mobile-fallback": false,
};

const passes = passesChecklist(
  passwordReset.criteria,
  incompleteAnswerJudgments,
); // false

The two entries capture the requirements discussed here, rather than a complete password-reset policy. A missing judgment also fails this check instead of silently counting as a pass.

Build many such cases from the reference material, then vary how users ask the questions. Give the model enough context about the personas it should represent: different people can express the same underlying request very differently. The wording changes, but the required answer content should remain stable.

The Multinear demonstration brings the test's input, output, and question-specific criteria into one view. Kuchin presents the platform as open source and emphasizes the method over the tool. For the password-reset question, he proposes generating fifty wording variants and checking each against the same required answer elements. This is a proposed test design, not a reported success rate. The all-items rule is the correctness target in this example, not a claim that every platform configuration enforces it.

Evaluations belong beside the first POC, not at the end of development. Build the initial implementation, define an initial set of tests, and run them. Some cases will succeed and others will fail. An average can summarize that run, but it does not explain what to change. The useful evidence lives in the individual input, output, and failed criterion.

A failed evaluation has two possible sources to investigate:

• The test is wrong. Its definition may not correctly describe the required behavior.
• The application is wrong. Its response may fail a valid requirement.

That distinction determines the next experiment. You might repair the evaluation, change the model, revise application logic, edit the prompt, or change the data used to answer the question. Tests should expose enough detail to make that choice an informed one.

A local fix is not yet a reliable improvement. Kuchin repeatedly encounters prompt changes that repair one test while breaking a previously successful case. Existing evaluations make those regressions visible. The workflow is therefore a loop:

1. Build the first application version and its initial evaluations.
2. Run the evaluations and inspect individual failures.
3. Change the implementation, repair faulty tests, or add missing cases.
4. Rerun the suite, including cases that previously passed.
5. Repeat until the benchmark adequately captures the requirements of this application at this point in time.

The result is a usable baseline: a set of expectations against which subsequent changes can be compared.

Once the benchmark is trustworthy enough for the application, optimization becomes a set of testable questions. Could GPT-4o mini preserve the required behavior of GPT-4o? Would a simpler retrieval solution suffice instead of GraphRAG? Does an agentic approach improve the outcome enough to justify its additional time and inference cost? Could the logic be simplified across the application, or only in one portion?

These are proposed comparisons, not demonstrated equivalences or savings. The benchmark supplies a common basis for deciding whether an alternative still does the required job. It makes it possible to investigate cost and complexity without relying only on a few appealing outputs.

The experimentation loop transfers across applications; the evaluation itself does not. Different tasks expose correctness in different ways. Kuchin distinguishes three common cases:

For database applications, the mock database should reproduce the relevant schema and contain data whose answers are known. That gives a specific user question an expected result. For a classifier, a direct comparison with the expected category can be sufficient; it need not inherit the support bot's judging method.

Guardrails belong in the benchmark as well. For the support bot, include questions that should not be answered, questions requiring a different kind of response, and questions whose answers are absent from the available material. Define the expected handling of those cases and evaluate it through the same iterative process. Success includes respecting the boundaries of what the application should do.

Evaluating the application as users actually use it gives frequent testing a purpose: each run checks whether an experiment preserved the behaviors that matter. Kuchin argues that this supports faster progress with fewer regressions because unexpected changes become visible while the team is still iterating.

He describes well-defined evaluations as producing a form of explainable AI. The useful interpretation is behavioral: the team can inspect what the application did on a known input, which requirement it met or missed, and how that changed between versions. This does not establish interpretability of the model's internal reasoning. The understanding comes from explicit expectations and inspectable results.

No particular platform is required. Kuchin says he built Multinear because he wanted support for this evaluation process from beginning to end, but another tool can serve the same purpose. What must remain is the connection between a user's task, its concrete correctness criteria, and the experiments used to improve the application.