BotDojo Launch: Enhancing AI Assistants with Evaluations and Synthetic Data

Talk transcript

1. 0:00

   [on-hold music] So hello, my name is Paul Henry.

2. 0:15

   I'm the founder of BotDojo, and as a previous CTO, I was working with teams deploying LLMs applications for hundreds of thousands of customers. And like many of you guys know, it's super easy to hook up a vector database, um, with an LLM over the weekend, but really hard to get it pr-production ready.

3. 0:33

   And so that's what we do. We are an AI enablement company, and we let companies deploy AI to prod.

4. 0:41

   Live demo time. All right, so today I'm going to show you a, a demo of our product. We're going to take, uh, synthetic data that we're gonna generate, and we're gonna combine it with e-evaluations to see how we can improve the performance of a chatbot.

5. 0:56

   Or at least that's what I hope happens. All right. So I'm gonna open up our template of our, uh, a chatbot, and we have customers live that are using this template.

6. 1:07

   It's kind of battle tested. Um, and so let's test it out.

7. 1:14

   How do I create a vector index in BotDojo?

8. 1:23

   Okay. And as you can see, all the little nodes are lighting up as they execute. Um, we're taking the question, we're looking at the chat history, we're going to the ve-vector database to retrieve the information, and then we're answering it with a AI model.

9. 1:36

   So if I pull this up, you can kind of see in our low-code, uh, editor, this is the prompt that we're sending to the LLM. We're getting the results out here.

10. 1:45

    And we also support, uh, JA- JSON Schema. So if the model, uh, supports JSON output like, um, Groq, um, Claude and all that stuff, then we just conform to that.

11. 1:57

    Um, one key thing is you can pull a trace of each node and see exactly what we sent to the LLM, what came from the retriever, the exact, you know, data, which has been super useful for debugging apps.

12. 2:11

    All right. And cool, we have an image. It's got citations. We should ship it.

13. 2:17

    That was supposed to be a joke, but all right. Um, [laughs]

14. 2:21

    so this is where evaluations come in. So I'm going to demonstrate, um, the evaluations that I previously ran. So we have a, a feature in BotDojo, uh, called Batches, which allow you to run a whole bunch of questions through your chatbot or your AI flow and, um, run evaluations to kind of see how things are doing.

15. 2:40

    So if you can see this, we have a few, uh, five evaluations that we ran. There's a little bit of red. Um, that's because, uh, we don't have enough information from our vector database.

16. 2:49

    Um, it also checks for things like hallucinations. So let's try to fix that. And so I'm going to clone this batch. I'm gonna rename it with generated data. I'm gonna increase the throughput a little bit because of time.

17. 3:06

    And, um, I had... I don't have enough time to generate all the data for this demo, so, um, the previous ran was filtering out the generated data, and so I'm gonna remove the filter that we're passing into the, uh, flow so it, it takes in the generated data.

18. 3:21

    You can also change the model and all that kind of stuff to see how it performs.

19. 3:26

    All right. So while that guy is running, I'm going to open up another flow. And so this is the actual flow that we, uh, generated that, uh, synthetic data.

20. 3:37

    And so let me, uh, let me run this one real quick.

21. 3:43

    And so this particular flow takes in multiple inputs, and so I'm gonna paste in, uh, some JSON from a previous run.

22. 3:52

    And what this is gonna do is, is kind of a trick that's been working well for customers, is where you take, um, you extract questions and answers from support tickets, so these are live agents talking with customers, and you use this as a test data to send it through your chatbot.

23. 4:07

    And, um, we take relevant information from the existing index, and we have it write a document. Um, and so it, it uses the same writing style and it, um, you know...

24. 4:18

    And then we do a inline cite ta- uh, evaluation to where we check to see if the document has enough information to answer the question. And then we also have a code node here where, you know, a lot of times when you're using these low-code editors, there's like situations where you have forty thousand different, um, boxes.

25. 4:34

    And so when you have to do write code, we support, um, TypeScript and, um, and soon, uh, Python. But you can see that, hey, we're getting the information and we're right into the vector index.

26. 4:45

    All right. Running out of time. Okay. Let me go back to the support chatbot. Yeah, moment of truth. So I'm gonna compare, um, the, the batch that we ran before with the new stuff in twenty seconds.

27. 5:01

    Oh, sh- [laughs] You do it, you do it fifteen times, and it doesn't work.

28. 5:10

    Ten, nine... We're also hiring, so if you're an AI engineer- [laughs] [laughs]

29. 5:14

    Help, help us fix this. All right, there it goes. Okay. Phew. All right. One second left. It's all green, so it improved the, uh, you know, [laughs] measurably improved something.

30. 5:23

    So, uh, thank you. Um, BotDojo.com. Check us out. Thanks. [upbeat music]