Practical tactics to build reliable AI apps — Dmitry Kuchin, Multinear

Talk transcript

1. 0:00

   [upbeat music] Welcome, everyone.

2. 0:17

   I'm going to talk about practical tactics to build, uh, reliable AI applications and why nobody does it this way yet.

3. 0:27

   Uh, a little bit about myself or why you should trust me. Um, I allowed fifteen years as a startup co-founder and CTO. Uh, I held executive positions for the last five years at, uh, several enterprises.

4. 0:41

   Uh, but most importantly, I spent last couple of years developing a lot of GenAI projects ranging from POCs to, uh, many production-level, uh, solutions and helped some companies to get it done.

5. 0:59

   And, uh, I've learned or distilled a way to, uh, make these applications reliable.

6. 1:06

   And there are quite a lot of, uh, tracks this, uh,

7. 1:12

   uh, this conference about evals and reliability, but, uh, to my surprise, nobody was talking about the most important things, and, uh, we are going to talk about it right now.

8. 1:26

   So, uh, standard software development life cycle is, uh, very standard, uh, simple. Uh, you design your solution, you develop it, you test it, and then eventually you deploy it.

9. 1:38

   And, uh, when people start doing, uh, POC with AI,

10. 1:45

    it sounds simple, like, uh, you can very easily do some prompt and, uh, models are very capable. But then you start, uh, facing some, uh, unexpected challenges. Uh, actually, like, you can easily do a POC that works fifty percent of the time, uh, but, like, making it do the same reliable work the rest of the fifty percent

11. 2:09

    is very hard, uh, because models are non-deterministic. And, uh, it starts requiring, uh, a data science approach, uh, continuous experimentation. You need to try this prompt. You need to try that model.

12. 2:23

    You need to try this approach, et cetera, et cetera. And, uh, everything in your solution, everything that, uh, represents your solution, which is your code, your logic, uh, the prompts that you use, the, the models that you use, the, the data that you base your solution on, changing anything of that impacts your, uh, solution in unexpected ways.

13. 2:49

    Um, people very often come to this, uh, to, to try solving this with the wrong approach. They start with, uh, data science metrics. They-- like, it sounds reasonable, right?

14. 3:04

    So it requires data science approach of experimentation, and, uh, people start, uh, measuring groundedness, factuality, bias, and other, uh, metrics that don't really help you to understand, uh, is your solution, uh, working the right way.

15. 3:22

    Does it, uh, does your latest change improved, uh, your solution in the right way for your users? Uh, for example, I've been talking to an ex-colleague that are building a customer support bot at Wix.

16. 3:36

    I asked him, "How do you know that your, uh, solution is working well?" He started talking about factuality and other, uh, data science metrics. Uh, that's again, I started to dig deeper, and then we just, uh, together figure out that the most important metric for them is, uh, the rate of, uh, moving from, uh, AI support bot,

17. 4:00

    like escalation, to a human support. If, uh, your solution, uh, hasn't able to answer the user with all this factuality, like it could be super grounded, but still not provide the right answer that the user expects, and, uh, this is what you actually need to test.

18. 4:20

    Um, and my experience was to start with real-world scenarios. So basically, you need to reverse engineer your metrics, and your metrics should be very, very specific to what your end goal.

19. 4:36

    So they should come from a product experience, from business outcomes. Uh, if your solution is customer support bot, you need to figure out what your users want and, uh, how you can mimic it.

20. 4:48

    And instead of measuring something, um, average or something generic, you need to measure a very specific criterias, uh, 'cause universal evals don't really work.

21. 5:03

    How do we do it? Uh, so for example, customer support bot, which is, by the way, one of the hardest, uh, things to do evals properly. Uh, let's say I have a, a bank, and a bank has, uh, FAQ materials which contain including, like, how do you reset your password.

22. 5:24

    Um, so what I usually do when I help my, uh, like, companies that I help them to build, uh, AI solutions, we start with, uh, reverse engineering, like how do we create evals based on that.

23. 5:38

    So in this case, I use LLM, and in most cases, I use LLM to come up with, uh, right evaluations. So here I can take, say, o1-- uh, o3 now, uh, and just- Reverse engineer what should be the user question, uh, that we know to answer based on these materials, and what should be the specific criteria that,

24. 6:01

    uh, these materials, uh, provide an answer for. And some of these criteria are quite important. So for example, here it says that, uh, uh, as part of the thing you, you need to receive a mobile validation, so you receive a SMS code, and, uh, it says that if you, uh, don't have a mobile number, then you can

25. 6:20

    reach, uh, support, et cetera, et cetera. Uh, if some of that information is missing from the answer, the answer would not be correct. But you need to be very specific about what exact information you need to see in the answer, and that is wh-- information is very specific to that specific question.

26. 6:40

    So you need to build, like, lots of evals, uh, from the materials in this case, uh, that mimic specific user questions that, uh, you need to be able to answer for.

27. 6:55

    Uh, how do we do it usually? Again, I work with, uh, smart models like o3, uh, and I, uh, provided enough context. I provided which personas are we trying to represent because you can make-- ask the same question in, uh, completely different ways depending on who is the persona asking.

28. 7:16

    Uh, yet you would expect exactly the same answer, so you need to account for it.

29. 7:22

    Um, so this is, uh, an example from, uh, the open source platform that we have that, uh, just helps to get it done. So if you look it up, Multinear, I'm not trying to sell you anything.

30. 7:35

    I'm not trying to, like, vendor lock in or whatever. It's completely open source, and if needed, I can just recreate it in a couple of days now with Cursor.

31. 7:44

    Uh, the point is in the approach, not in the platform. Uh, so for example, here we see that very same question, um, how do I reset my password? You see the-- what was the input, what was the output, and, uh, that specific criteria that I measured, uh, that specific question, how do I know if the answer is

32. 8:08

    correct? And now I can just reiterate and pro-- generate, like, fifty different variations of the same question and see if I still get the right answer. The-- if the answer matches all the checklist that I have for that specific answer.

33. 8:25

    Um, how the process usually works. Um, so contrary to, like, regular approach, you build your evals not at the end of the process, but in the very beginning of the process.

34. 8:37

    So you s-- just build your first version of the POC. You define the first version of your tests, evaluations. You run them, and you see what's going on. You, you will see that, uh, in some cases it will fail.

35. 8:52

    Uh, in some cases it will succeed. What's important is to, to look at the details, not just see the average numbers. The average numbers won't tell you anything, uh, won't tell you how to improve it.

36. 9:05

    If you actually look at the details of each evaluation, you'll see exactly why it's failing. It could be failing, um, because your test is not defined correctly. It could be failing because your, uh, solution is not working as it should be.

37. 9:20

    And, like, i-in order to do it, you may need to, uh, to do a change in, in-- like, you may change a model. You may change something on-- in your logic.

38. 9:29

    You may change a prompt or the data that you use in order to, uh, answer a question in our example. And, uh, basically what you do now is experimentation.

39. 9:40

    So you, you start running your experiment. You change something. You, you need to define these tests in a way that will, uh, help you to make an educated guess on, uh, what you need to change in order to, to do it.

40. 9:55

    In some cases it will work, in some cases it won't. But even if it works, uh, let's say you change something in your prompt and it fixed this test.

41. 10:05

    In my experience, in many cases it breaks, uh, something that used to work before. Uh, like, you, you, you have constant regressions, and if you don't have these evaluations, there is no way you'll be able to catch it on time.

42. 10:19

    So this is hugely important, and what actually happens is that, again, you build your first version. You build your first version of evals. Uh, you match them. You run these evals.

43. 10:30

    You improve something. You improve your evals or maybe add more evaluations, and then you, like, uh, continuously improve it until you reach some point where you are satisfied with your evals for this specific solution for that specific point of time.

44. 10:46

    And what actually happened is that you, you, you got your baseline. You got your benchmark that, uh, now you can start optimizing, and, uh, you have the confidence that the tests should be working.

45. 11:00

    So now you can try another model. Let's say, uh, well, what-- how can I try to see if Foro Mini will work the same way with Foro or not?

46. 11:11

    Uh, can I use the GraphRAG or can I try a simpler solution? Ca-- uh, should I have, uh, to use the agentic approach that ta-- like, may be better but, uh, requires more time, more, uh, inference cost, et cetera?

47. 11:26

    Or should I try to simplify the logic? Or maybe I can simplify the logic for a specific portion of the application, et cetera, et cetera. Having this benchmark, uh, allows you to do all these implementations, uh, with confidence.

48. 11:41

    But again, the, the most important part is, like, how do you reach this benchmark? And, uh, while the approach is, uh, pretty much the same, the evaluations that you need to build and how do you build your evaluations are completely different depending on the solution that you need to build.

49. 11:58

    Because, uh, the models are super capable right now, uh, so they allow you to build a huge variety of, uh, solutions. But each and every solution is quite, uh, different in terms of how do you, uh, evaluate it.

50. 12:12

    Uh, for support bot, you usually typically use LLM-as-a-judge, as I, uh, made an example. If you're building Text-to-SQL or text to graph database, then, uh, to my experience, the best way is to create a mock database that represents the, um, whatever, uh, database or databases that you need the-- your solution to work with.

51. 12:35

    They represent the same schema, and you have the mock data, so you know exactly, uh, what should expect on specific questions. Um, if you need to build some classifier for call center conversations, then your, uh, tests are like simple match whenever this is, this is the right rubric or not.

52. 12:54

    Uh, and the same appro-- uh, approach applies to guardrails. So, uh, getting back to the support... to, to the, uh, example of a customer support bot, uh, guardrails, you need to cover, uh, questions that should not be answered, or questions that should be answered in different ways, or questions that, uh, uh, the answers are not in the

53. 13:14

    material. So all of this you can put into your benchmark, just different type of benchmark, but it's pretty much the same approach.

54. 13:23

    Uh, so just to reiterate, uh, the key takeaways, you need to evaluate your apps the way your users actually use them, um, and, uh, avoid abstract metrics, uh, because these abstract metrics don't really measure anything important.

55. 13:40

    Uh, and the approach is, uh, through experimentation. So you run these evaluations frequently. You-- That allows you to have rapid progress with, uh, less regressions because testing frequently help you to, to catch these surprises.

56. 13:55

    Uh, but most importantly, what you get if you devi-- uh, define your evaluations correctly, you get your solution pretty much, uh, as kind of explainable AI because you know exactly what it does, you know exactly how it does it if you test it the right way.

57. 14:13

    Thank you very much. Uh, take a look at, uh, Multinear. Uh, that's a platform that you can use to, uh, run these evaluations. You can totally use any other platform.

58. 14:25

    The approach is quite simple. It doesn't require any specific platform. Uh, I've built Multinear just because no other platform helped me to do it this way, to, to help me with the process of evaluation, like end-to-end.

59. 14:40

    Um, I'm working on a startup that does reliable AI automation right now. Um, and, uh, yeah, thank you very much. [outro jingle]