Building Trust in Enterprise AI: Evaluating Domain-Specific LLMs for Real-World Financial Scenarios

Talk transcript

1. 0:00

   [on-hold music] Hello, everyone.

2. 0:18

   My name is Waseem. I'm one of the co-founder and CTO at Writer.

3. 0:23

   Today, I'm going to just tell you a quick story about actually why we're building at Writer, what we're doing. But before we dive in, I would love just to give you a quick, uh, history of Writer.

4. 0:34

   So Writer's, we started the company in twenty twenty. We love to say the story of Writer is a story of the transformer. We started building those decoder encoder model in the early days, and we start-- we keep building those model, and we built a lot of them.

5. 0:49

   Today, we have a family of models, I believe around sixteen we published. We have another twenty coming in the way, and we keep building those models. And you're going to see from this list, those models come in, in two categories.

6. 1:04

   General model like Palmyra X, Palmyra three, four. We have a Palmyra five's coming soon. And we have a lot of what's called domain-specific model. Creative, financial services, Palmyra Medical.

7. 1:19

   Now, early twenty twenty-four, basically last year almost, we start seeing this trend with all the LLM basically get very high accuracy in general with any benchmark. We see the accuracy moving and just growing, and I believe everyone noticing this accuracy.

8. 1:39

   Today, average accuracy for a good general models between eighty, maybe close to ninety.

9. 1:48

   So that basically make a-- bring a question inside the company saying, "Is it worth it for us to start building and keep building domain-specific models?" If the accuracy today with general model achieving around ninety percent

10. 2:05

    and we have domain-specific model, should we just keep building general models, fine-tune it, maybe go direction with what you call reasoning or thinking models, and that will be more than enough actually.

11. 2:17

    Then we don't need those financial or what's called domain-specific model. Now, to answer these questions, we need data.

12. 2:27

    So whatever we're going to present next actually could be applicable to financial services domain-specific model... Sorry, to medical-specific model, customer support domain-specific model, and all different domain-specific model. Today, I'm going to talk specifically about the financial spec-- uh, what you call it, the financial benchmark for domain-specific model.

13. 2:48

    Uh, we have something similar for medical, but we believe we are-- but we start seeing similar result. Now, let me dive in.

14. 2:59

    Just to remind you, we're trying to answer these questions. General model, domain-specific model, should we keep build them? Where are we actually going from here? We start actually saying, "Great, we don't know the answer.

15. 3:11

    Let's actually do the evaluation. Let's create the data." And we create something called FailSafe. The idea behind it, let's create real-world scenario to evaluate those model, and let's see actually of those new model can really give you the accuracy that we promise or the accuracy that we see today from the benchmarking on domain-specific.

16. 3:35

    We created two type of categories in this evaluation. Something called query failure. In query failure, basically, we introduce three type of subcategories. Something called misspelling queries. You know, when you go ask the LLM questions, but you do some spelling error, segment error, you do some comma, comment typo issues.

17. 3:59

    We introduce that to the eval set. We introduce something in, in, like in, in comp-- what called incomplete queries.

18. 4:07

    You're missing some keyword, some stuff not clear. We introduce what's called out-of-domain queries. If you are not expert in the field, or you decide to copy-paste some general answer, try to answer about something very specific.

19. 4:22

    And also introduce the second category is what we call the context failure. In the context failure, basically, and this gets very interesting, we introduce three subcategories. What's called basically missing context.

20. 4:35

    We basically ask the LLM question about context not exist in the q-- the request itself in the prompt. We introduce what's called OCR error. Today, when we do any kind of OCR or convert physical doc-document to text, we introduce a lot of errors like, you know, character i-issues, distance between them, the

21. 5:00

    word between when you do the OCR could be merged together. So introduce that type of errors. And also we did what's called an irrelevant con-context. Let's say you wanna ask question about specific document, and you end up basically uploading completely wrong document.

22. 5:16

    Does the LLM going to still answer? Is the LLM just actually figure out you have a completely irrelevant context? Now, when you put all this data together in domain spe-- in financial specific, financial service specific, you need some kind of diversity.

23. 5:32

    This is a quick screenshot, just tell you what amount of data, how much token. Something worth mentioning, the white paper, the data, the evaluation set, the leaderboard all actually open source today available in GitHub and Hugging Face.

24. 5:47

    So anyone, please check it out. And we introduce very simple, what you call it, evaluation ma-- key metrics. Basically, we need to look to two things. Either model give the correct answer

25. 6:00

    Can the model actually give good follow to the grounding or context grounding or basically what we call it here the context? This is quick or high-level way of how we do the calculation.

26. 6:15

    So to evaluate, we selected a group of models. Today we can see a lot of chart model and also thinking models. This is basically the two lists we have here.

27. 6:28

    I'm sure you're familiar with this list. And then we run the evaluation and we start seeing very interesting result. I'm going to dive in directly to the result. And basically we start getting something fancy.

28. 6:43

    With all this color, let me switch to the mo- what basically see.

29. 6:49

    What started getting very interesting, we're seeing really good behavior in all thinking models. Actually, they don't refuse to answer. This sound good most of the time. But in reality, when you give something those LLMs wrong context, when you give them wrong data, when you have a completely different grounding, those model actually fail, fail to follow this part

30. 7:14

    and they still give you an answer. And that basically get you way higher hallucination.

31. 7:21

    If you start focusing on the answer itself, can the model give me answer or not, you can see basically almost every model from the domain-specific to general model, they give you some kind of answer, all of them close to each other.

32. 7:34

    Actually reasoning or thinking model, they get you even higher score a little bit from there. But we get to the grounding and concept grounding, this is when stuff get more interesting.

33. 7:47

    You can see specifically in task like text generation or question answering, it's just not performing well. Now all this chart look great. What I prefer is the numbers.

34. 8:00

    This is the same data we used to generate the chart. We can go through this really quick. And if you look at this number here, especially for example like the o1 or o3 or Palmyra Fin, you can start noticing the stuff those model doing amazingly.

35. 8:15

    And basically when you ask was it misspelled, when you got stuff incomplete out of, uh, domain, the numbers look amazing. The model can take a query with misspelling, wrong grammars or even out of domain and still can give you the answer.

36. 8:31

    But when you start going to grounding, this is when the stuff get very interesting. I'm going to hold this slide for a second here. If you-- do you notice something different?

37. 8:45

    Smaller is better.

38. 8:47

    Yep. And also those bigger more thinking give you the worst result. You're getting almost seventy-- fi-fifty percent to sixty percent, uh, worse in the grounding. Meaning the model is just not following.

39. 9:03

    You're attaching context, you ask the questions and the answers exist outside the context completely. Same thing coming to stuff around it, another context. So you can look at the data and see smaller model actually performing better than all this model overthinking at that side.

40. 9:22

    And this is basically will get us about is this thinking or just a chain of thought? You know, this could be a lot of argument. At least from the data we have in domain-specific task, those model not thinking at that stage.

41. 9:39

    Meaning hallucination is really high causing a lot of, a lot of issues especially in this fina- if, uh, in this resi- uh, benchmark we run here in fina- uh, financial use cases.

42. 9:52

    Also we can see there is a huge gap between what you call robustness and the hallucination and getting the answer correct. So definitely we still have a lot work to do to build those model and better performance.

43. 10:08

    But also that get me to, you know, to the main idea. If you go back real quick here, even with the best model between all the slide, we're still not getting between robustness and context grounding more than eighty-one percent.

44. 10:24

    Sounds a great number. If you think in reality, you're saying every hundred requests, twenty of them is just completely wrong. So that basically what we start seeing believe at least today with the technology we have, with the current model we have, until we have something completely different, we're seeing you need full stack.

45. 10:47

    You need the RAG system, you need the,

46. 10:50

    uh, grounding, you need everything from guardrails and the build around the system itself to actually have something reliable utilized today. In the same time,

47. 11:03

    I would love to go back and answer the first questions. And our first question here,

48. 11:10

    do you still need to build the models? At least today from the data we have from running those benchmark, the answer simply yes. We still need to build and continue domain-specific model at least with the today implementation.

49. 11:24

    Even accuracy is keep growing but the grounding, the context, following all the context correctly, it's still way, way, way behind from everything we see today in the market. [outro music]

50. 11:38

    Thank you so much guys. [applause] [outro music]