How Intuit uses LLMs to explain taxes to millions of taxpayers

Talk transcript

1. 0:00

   [on-hold music] Hi, I'm Jaspreet.

2. 0:16

   I'm a senior staff engineer at Intuit. I work on GenAI for TurboTax, and today we'll be talking about how we use LLMs at Intuit to, well, help you understand your taxes better.

3. 0:28

   So I think, uh, to... just to understand the scale, right? Uh, Intuit TurboTax successfully processed forty-four million tax returns for tax year '23,

4. 0:40

   and that's really the scale we are going for. We want everybody to be-- have high confidence in how their taxes are filed and understand them, that they're getting the best deductions, uh, that they can.

5. 0:52

   So, so this is the experience that we work on. So, uh, you go into TurboTax, you, uh, enter your information, then you go through what credits you are eligible for and so on.

6. 1:10

   And we basically help you expa- uh, expand onto how you are getting the tax breaks that you are, help you understand them better, uh, and so on. And,

7. 1:25

   and this is another example. This is basically the overall tax outcome, like what is your overall refund for this year?

8. 1:37

   Now, Intuit's GenAI experiences are built on top of our proprietary GenOS. That's the generative OS that we have built, uh, as a platform capability, and it has a lot of different pieces, uh, that you see over here.

9. 1:51

   Uh, the key goal is that we found that a lot of the GenOS tooling that comes out of the box is not supporting all our use cases. We want to-- Most prominently working in tax, we are in the regulatory business, uh, safety, security, uh, is very, very important, so we wanna focus on that.

10. 2:10

    At the same time, we wanna build a piece that a company at the scale of Intuit can use end-to-end at a really large scale. So that's where GenOS comes in.

11. 2:19

    We have different pieces. There's on the UI side, which is the GenUX, and then there's Orchestrator. That's basically the piece where different teams are working on different components, different pieces, different LLM solutions.

12. 2:30

    How do you find the right solution to answer the right question? And, uh, Intuit calls, uh, the entire experience that we power through this Intuit Assist. So I'm gonna deep dive into specific pieces that our team used to build out, uh, the experience for TurboTax.

13. 2:50

    So as I said earlier, right, we have millions and millions of customers who are coming in, so we're trying to build a scalable solution that can work end-to-end. So on the slide here, I'm basically going to talk about different pieces that are powering the experience.

14. 3:03

    Uh, of course, to begin with, the first iteration was the prompt tooling. Uh, basically a prompt-based solution to try and go through, uh, what's your tax situation going on.

15. 3:12

    Let's take an example of what I was showing earlier, which was your tax refund. So your tax refund has many constituents. These are your deductions, these are your credits, standard deduction, W-2 withholding, and so on.

16. 3:24

    So we wanna make sure that you understand all of that. So we built a prompt-based solution around it and worked from there. The production model that we went with is Claude, uh, for this use case.

17. 3:37

    Uh, Intuit is one of the, uh, biggest users of Claude. Uh, we had a multi-million dollar contract for this year as well. And, uh, you'll also see OpenAI over there.

18. 3:46

    So OpenAI is where we used for other question and answering. So you'll see on the slide we're talking about static and dynamic type of queries. So, uh, static queries would be, you know, what I was showing earlier that we know you are looking at your summary, you want to see what happened, uh, overall, so that would be

19. 4:04

    a static prompt. Think of it like a prepared statement. Uh, however, the additional information that we're gathering is the tax info when the user comes in. Now, uh, dynamic query would be users have questions about their tax situation.

20. 4:18

    You know, "Can I deduct my dog?" Well, you can't, but, uh, you can try. So things like that, that's what we are trying to, uh, answer more dynamically. Um, OpenAI's GPT-4 Mini had been the model of choice for until a few months ago.

21. 4:31

    We are now iterating on the newer versions. Of course, models change every year, uh, every month, I should say. Uh, so we're trying to focus on that. Um, same for the dynamic piece again.

22. 4:43

    Another important, uh, aspect is, you know, tax information. IRS changes forms every year. Uh, Intuit has proprietary tax, uh, information, tax engines that we wanna use. So we have, uh, RAG-based and of course GraphRAG-based solutions around it as well.

23. 4:59

    So they help us, uh, answer users' questions much better. And, uh, one thing that we also piloted recently was actually having a fine-tuned LLM. So, uh, we went with Claude because that's the primary one we are using there, and we stuck to static queries, and we tested it out.

24. 5:16

    And, uh, it does well. Uh, it definitely does well. Uh, quality is there. Uh, it takes effort to fine-tune the model. Uh, however, we found that it was a little too specialized in the specific use case.

25. 5:29

    And, uh, one thing I want to highlight, I'll deep dive further on, is evals. So you wanna make sure that we evaluate everything we do. Um, you wanna make sure what's happening in production.

26. 5:40

    You want to make sure in the development life cycle, you're doing everything you need to do to make sure that you have the best prompts out there. Uh, and with that, moving on to the next slide.

27. 5:52

    So to summarize a little bit, you know, these are the key pillars that we have. I already spoke about some of them before. I want to highlight here that- At the bottom part in this slide, actually.

28. 6:02

    The human domain expert. So, uh, Intuit has a lot of tax analysts that we work with, uh, of course, that are on our-- uh, that work with us, uh, decoding IRS changes year over year, making changes and so on.

29. 6:14

    So they are the experts that provide us the information, uh, make sure the evaluations are correctly done. So we have a phased evaluation system. We have manual evaluations initially in the development life cycle.

30. 6:27

    Um, and another thing that we have done is actually using the tax analysts as the prompt engineers. So that allows us, the folks in data science and ML world, to actually focus on the quality, defining the metrics, uh, making sure we have a nice data set that we can iterate on and test on, uh, as we go

31. 6:45

    along. As I said, models change. We wanna try out different models. We wanna see da-- uh, the laws change in the IRS, say tax year '23 to '24, what happened, uh, so those changes, we focus on that.

32. 6:58

    Uh, and human experts bring their expertise and are able to both help with prompt engineering and get the initial evaluations done. That then becomes the basis for automated evaluations.

33. 7:11

    Um, LLM-as-a-judge is what we use as well. Uh, I'm gonna talk a little bit more about that. Uh, I'm gonna take, uh, going back then to what I was telling earlier about the Claude 3 Haiku and fine-tuning.

34. 7:25

    So, uh, fine-tuning, as part of GenOS, we built out a lot of tool sets. Uh, one more thing that we want to do is support fine-tuning. So for our use case, we actually stuck to just fine-tuning on Claude 3 Haiku, powered by AWS Bedrock.

35. 7:41

    And the goal there was that we wanted to see if we can actually improve, uh, the quality of responses. Uh, biggest driver, of course, is, uh, fewer instructions, uh, needed once you have fine-tuned a model.

36. 7:54

    We wanna make su-- uh, latencies are a big concern, so we wanna see if we can squeeze down the prompt size and at the same time keep the quality, uh, that we need and keep going there.

37. 8:05

    So this is roughly what it looks like. We build out, uh, we have different test AWS accounts, different environments, uh, that are provided by the, uh, platform teams that we work with.

38. 8:16

    We look at the data and, uh, brief not to regulations, uh, seven to six-- uh, 16 regulations. So we only use consented data from users, uh, make sure, uh, we're on the right.

39. 8:29

    And, uh, just to double down on the evaluation part, right? You want to evaluate everything. So the key pillars are accuracy, relevancy, and coherence. So we have both manual and automated systems.

40. 8:41

    We also have broad monitoring, uh, automated systems. Basically look at sample data, uh, on what the LLM is basically giving real users in real time. And, uh, for this tooling that we've built out, uh, here, LLM-as-a-judge comes in, in the auto eval side.

41. 8:58

    We've also developed some tooling, uh, in-house, uh, to basically do some automated prompt engineering, and that actually really helps to update our LLM-as-a-judge. Basically, LLM-as-a-judge operates on top of a prompt.

42. 9:14

    Uh, it needs different information. It needs some manual samples, which are the, like, golden dataset. We use AWS Ground Truth for that, uh, and take on that. Uh, one more thing that I want to highlight here is, uh, models.

43. 9:27

    So we made the move from, uh, uh, Anthropic Claude Instant to Anthropic Claude Haiku for the next year, uh, for, uh, tax year '24. And that takes some effort, and the only way it's possible is because we have clear evals in place so that we can test out, uh, whatever we are changing.

44. 9:47

    And, uh, model changes are not, uh, as smooth as you would think.

45. 9:54

    These are some more details on what we're talking about on the automated evals. Uh,

46. 10:01

    as you can see, the key output is we wanna make sure it stacks accurate. That's the main thing we want to aim for and focus on that. I'm gonna move on here.

47. 10:09

    So let's talk about some major learnings that we have. So, uh, the contracts are really expensive, and the only way they are slightly cheaper, if you have long-term contracts.

48. 10:20

    So, uh, you are tied into the vendor, so, uh, it helps to have strong partners on the vendor side who work with you, uh, to help iterate, help improve.

49. 10:31

    And, uh, I think I was in this conference last year, and this was one thing called out then as well that, uh, essentially vendors are a form of, uh, lock-in.

50. 10:40

    The prompts are a form of lock-in. It's not easy. And we found out it's not even easy to upgrade this model from the same vendor going to the next year.

51. 10:48

    So we want to focus on that. Uh, another thing I really wanna highlight here is the latency. So, uh, LLM models, of course, they don't have the SLAs of back-end services.

52. 11:00

    We're not looking at, you know, hundred millisecond, two hundred milliseconds. We're talking about three seconds, five seconds, ten seconds. So as the user's tax info-- uh, tax information comes in, maybe they have a complicated situation like me that, you know, they own a home, they have maybe something in stocks, and they're trying to file.

53. 11:18

    They have-- Their spouse have their jobs as well. A lot of things going on. So the prompts really balloon up, uh, if you're trying to figure out the outcome.

54. 11:26

    And, uh, as you go into, you know, tax day, everybody's trying to file on tax day, right? April 15th. So, uh, latency really is, uh, shooting through the roof.

55. 11:37

    So we design a product around that. We wanna make sure we have the right, uh, fallback mechanisms, the right, uh, user design, uh, product design to make sure that the user experience is seamless and, uh, useful.

56. 11:52

    Uh, we wanna make sure that the explanations are helpful more than anything else. And, uh, I think I covered all the other places, but once again, I cannot say that enough, evals are a must to launch.

57. 12:03

    Uh, focus on evals. Make sure you have clear guidelines on what you're building. Uh, have clear golden dataset. I've heard that from other talks as well. That's really a key point.

58. 12:16

    Uh, that's all. I'm gonna pause here for questions.

59. 12:20

    Uh, if you're gonna be asking questions, please come to one of the microphones so that we can capture the audio. Thanks.

60. 12:30

    Yeah, hi. Um, you said, uh, evaluate everything, right?

61. 12:34

    Right.

62. 12:34

    But, uh, with GenAI systems there could be, you know, very small changes, right?

63. 12:38

    Yes.

64. 12:38

    You make a small change to a prompt, and evaluations can get very expensive or slow down your whole sort of development process, right? So maybe could you dive a little bit deeper into, like, when do you bring in different types of evaluations?

65. 12:53

    Are there, are there anything that you just say, "Uh, we ran some aggression tests and it looks fine," so you launch? Or do you always go kind of with a expert-

66. 13:02

    Sure. Sure

67. 13:02

    ... opinion approach?

68. 13:02

    Uh, thank you for the question. So just to reiterate. So the evaluations are different types. I would say when we are in the initial phase of development, we are looking more on the, uh, manual evaluations with tax experts so we can get a baseline in place.

69. 13:15

    Then as we are tweaking different things in the prompts, that's where auto-evaluation comes in. So we basically take the input from the, uh, uh, tax experts and use that to train a judge prompt for the LLM.

70. 13:30

    So that LLM is once again expensive. Uh, we go for the GPT-4 series until recently on that one. And, uh, then minor iterations we can do with auto-eval. So we have clear understanding with product.

71. 13:42

    We wanna make sure that the quality is there. And maybe once we have major changes, for example, we went from tax year '23 to tax year '24, then we definitely reiterate.

72. 13:51

    Uh, if the prompt changes a lot, we would, uh, go for manual evaluations.

73. 13:57

    Um, thank you for the technical deep dive. I was more interested in the product side of it.

74. 14:02

    Sure.

75. 14:03

    We, we also do taxes, so I was curious, what are the kind of, um, LLM interactions that the users are having? Like, what are the kind of questions they're asking?

76. 14:12

    Is it, is it more like critical parts of the workflow or more like, um-

77. 14:17

    Sure

78. 14:17

    ... what are my taxes?

79. 14:18

    So, uh, we have question answering for all types of questions. That includes both the product question, as in, you know, "How do I do this in TurboTax?" Uh, or also their tax situation.

80. 14:29

    So for example, uh, "I paid the tuition for my grandchild. Can I claim that on my taxes?" So things like that. So our goal is, we have different teams going after different pieces.

81. 14:39

    Our goal is we want to answer all of these questions. And, uh, accordingly, different types of questions need different solutions, and that's where maybe I would reiterate, go back to

82. 14:52

    here. So... There. So this piece here, Planner. So essentially, this is where it comes in. We wanna make sure when the query comes in, we understand what the user is trying to ask, and then we have different kind of solutions for different kind of questions and go through that.

83. 15:12

    Thank you.

84. 15:13

    Uh, yeah, hi. So you mentioned about the evaluation. So one quick question. Like, so TurboTax, I'm sure it involves a lot of numbers, the answers.

85. 15:21

    Right. Right. Absolutely.

86. 15:21

    So how do you verify those numbers in terms of the evaluation? Let's say, uh, the actual tax number is eleven [REDACTED:generic_id] thirty-five.

87. 15:27

    Right. So-

88. 15:28

    And if it's something like eleven thousand one hundred. So it's quite difficult to catch this with a manual evaluation-

89. 15:34

    Yes. Yes

90. 15:34

    ... or with an AI. So yeah.

91. 15:35

    Yes. Uh, thank you for the question. So that's a key thing that we work on. So TurboTax, of course, has a tax knowledge engine that we have bui- built proprietary in-house, managed over the years, built and developed, and that's really what's providing these numbers.

92. 15:47

    The tax profile information is all coming from these numbers. We are not having LLMs do the calculations at all. We're basically using the ground truth that is already existing in our systems as the numbers that we see.

93. 15:58

    And we have safety guardrails, uh, maybe this piece here, I would probably call out. We have a lot of safety guardrails on what's the raw LLM response. Make sure, you know, we are not hallucinating numbers before we send to the user.

94. 16:13

    Got it. So, uh, the data is coming from the tax engine itself.

95. 16:17

    Yeah. Correct.

96. 16:17

    But when you formulate the final explanation, the answer itself, so how do you make sure that, uh, the numbers that are actually in the final answer are, you know-

97. 16:25

    Right. Right

98. 16:26

    ... same as that's coming from data?

99. 16:27

    So, so basically we have ML models that are working under the hood as part of the, uh, security aspect that you see here that basically make sure we did not hallucinate any numbers that we built on.

100. 16:37

     Got it. Yeah. Thank you.

101. 16:42

     Yeah.

102. 16:42

     Um, could you give an overview of how you use both just a traditional RAG and GraphRAG, like a hybrid, in your workflow?

103. 16:52

     Sure, sure. So, uh-

104. 16:53

     And, and sorry, one more question is, now with the new model Claude 4 coming out, do you think the fine-tuning might be getting easier where it needs needed?

105. 17:01

     I'll take the first one. Uh, so a G- uh, GraphRAG, we've definitely seen better response, uh, better response quality with GraphRAG. Uh, even more than that though, I think for end user helpfulness, me- getting personalized answer is the key piece, I would say.

106. 17:18

     GraphRAG definitely outperforms, uh, uh, regular RAG. Uh, and what even more outperforms is personalizing the answers. And to your second question, uh, we are constantly evaluating the models. Uh, this is really the time that, you know, April is just behind us.

107. 17:34

     We are trying to look at what new things we can do. We also have some, uh, in-house models that Intuit trains and develops. So we are constantly evaluating and, uh, I don't have an answer now what we'll do for the next tax year, but yes, we keep working on that.

108. 17:49

     Uh, you mentioned, uh, you have different situations, tax situations, and you come up with an answer. So if I describe my situation, uh, it's complicated, and it comes up with an answer.

109. 18:00

     Is that answer being generated using the LLM, or is it going back to the tax engine? And how do you explain h-how you came up with that answer? And I, I assume there's going to be a lot of legal challenges to-

110. 18:15

     Right. Right. Right

111. 18:15

     ... wrong answers.

112. 18:16

     Right. Absolutely. I mean, uh, Intuit focuses heavily on legal, legal and privacy, uh, controls. So the solution for this one, right, what we worked on here, this is specific.

113. 18:27

     This is more of the static variety of questions. So once again, what I was saying earlier, the inherent numbers are coming in from Tax Knowledge Engine, and we have tax experts who actually crafted these prompts.

114. 18:39

     So they are specifically tested for each piece that you see here. So that's basically when we do the evals, we make sure it doesn't happen, what you're suggesting.

115. 18:49

     Okay, great.

116. 18:50

     Uh, thanks.

117. 18:51

     Uh, thank you so much. What a great talk. [upbeat music]