How we scaled 500m AI agents in production with 2 engineers

Talk transcript

1. 0:00

   [on-hold music] Hey, everybody.

2. 0:17

   Uh, yep, I'm Kyle Corbitt from OpenPipe, and I'm here with Mustafa Ali from Method.

3. 0:22

   Hello.

4. 0:22

   We're gonna be talking about how Method has scaled in production to over five hundred million agents, uh, and basically all the, the tricks they use to, to make that actually work.

5. 0:31

   So a little bit about Method is that we essentially collect and centralize liability data from across hundreds of different data sources. This includes tapping into the credit bureaus, uh, connecting with the card networks like Visa and Mastercard, um, and just direct connections with the financial institutions and various other third-party sources.

6. 0:48

   And, you know, we, uh, sort of aggregate and enhance this data and serve it to our customers, who are typically other fintechs, banks, or lenders, and they use this enhanced data to, um, anything really to do with debt management, so refinancing, loan consolidation, liability payments, or just personal finance management.

7. 1:07

   Um...

8. 1:09

   Yeah. And at OpenPipe, what we do is we help you build, uh, train, and deploy open-source models, um, for actual usage. We also let you use in production your signals you get from users, from the environment to improve your model continuously over time, and that's some of the things we'll be talking about, uh, what we did with

9. 1:26

   Method.

10. 1:27

    Nice. So one of the early challenges that we faced at Method while coming up with this, you know, aggregation pipeline, uh, was that some of our customers basically came to us and said, "You know, it's really nice that you can give us the balance and payment information on a specific liability for their end consumers.

11. 1:46

    But you know what would be really nice is if you could also give us some of these liability-specific data points, like the payoff amount on an auto loan or the escrow balance for a mortgage."

12. 1:56

    And, you know, we said, "Okay, let's do some research." So we go back to some of our data partners and basically ask them, you know, "Is there anything, you know, we can plug into to get these kinds of data points?"

13. 2:05

    And what we found was there's really no central API that we could get access to that would allow us to get some of these data points. And, of course, ideally, we would want to work with, uh, directly with the banks, but, you know, having already worked with banks before and just from initial conversations, we realized that it

14. 2:21

    would easily take up to at least a couple of years before getting anything solid done. And, you know, we, we're an early-stage company, so we wanna build for the customer fast.

15. 2:29

    Um, and so that's really what we're trying to come up with, a solution that we can just, you know, uh, push into production tomorrow.

16. 2:37

    And so just to get a better understanding of how some of these companies are operating today, uh, the services that they're providing today, how are they doing that in the first place, right?

17. 2:44

    They, they must be getting that data somehow. So we go back to some of these customers and basically ask them, you know, "How are you guys operating?" And what they tell us is, is kind of interesting.

18. 2:53

    So a lot of these companies, they basically hire offshore teams of contractors and, you know, they, uh, uh, these teams are basically responsible for calling these banks, um, on behalf of the company and the end consumer.

19. 3:06

    Uh, they authenticate with the banks, gather the necessary information, somebody has to proof-check it, it gets sent back, um, and then it gets integrated into the financial platforms. Um, and it gets surfaced to the user, is used for underwriting, stuff like that.

20. 3:19

    And so that's the status quo that we're dealing with here. And when you think about it, that's a very inefficient manual process, right? It's, it's-- When you try to think about scaling, it doesn't really scale.

21. 3:30

    It's a very, um-- It has a lot of problems. You know, it's expensive because one person can only do one thing at a time, right? So if you want to scale, uh, you basically have to hire more people.

22. 3:41

    And for the same reason, because it's so synchronous, it's also really slow. Um, and the main-- I guess the, the biggest problem with that is also that it's, there's a lot of human error involved and, um, you, you need to hire a team to fact-check it, uh, to, to proof-check it.

23. 3:55

    And, um, uh, it's the, the be- the, the worst thing that you're gonna end up with is to surface basically inaccurate financial information. And so conceptually though, if you think about it, it's kind of like an API, right?

24. 4:06

    You have the request component, you have the authentication component, you have the response, validation, and all that stuff. Uh, so essentially, when you drill this problem down into the core problem, that's really just trying to make sense of, um, unstructured data, right?

25. 4:21

    So if only there was this magic tool or software that we could use that was really good at parsing unstructured data.

26. 4:29

    And, you know, lucky for us, around the time that we were trying to solve this problem, OpenAI announced GPT-4. And, you know, as people like to call it, there was this Cambrian explosion of AI or LLM-enabled applications all around us, and the results were just mind-blowing.

27. 4:45

    Um, and we thought to ourselves, "You know, this, this, this is the perfect thing for us. This is like a godsend." Uh, so we tried to like, you know, we, we tried to see if there's anything there that we could use.

28. 4:54

    And if there's one thing that we all know in this room is that, uh, advanced LLMs, especially post-GPT-4, are really good with, um, with parsing unstructured data. So tasks like summarization or classification, they're really good with that kind of thing.

29. 5:08

    So we wanted to test that theory out and see what that can get us.

30. 5:14

    And so we put our heads down, hacked together this agentic workflow using GPT-4, and as expected, you know, it worked really well. So we tried to like expand some of our use cases because it, you know, the API costs are high, so we wanted to get as much as we could from a single API call and, you

31. 5:30

    know, it turned out to be really good at that. So we tried to-- Obviously, this was in a very controlled manner, um, but this was in production, and so we were testing out, uh, different, uh, extractions basically, and, um, you know, everything was re- going really good.

32. 5:45

    Uh, but as soon as we started to increase a little bit of, uh, traffic, uh, what we found was, you know, the bill had to come due. And, um, [lip smack] it was a lot.

33. 5:55

    So seventy thousand dollars for our first month in production with GPT-4. And, you know, this was-- this made leadership really unhappy and, you know. But, um, but it was something, it was something they were, they were fine with because the value that we were getting out of GPT-4 was so immense.

34. 6:10

    Um, and so we actually kept this thing in production for at least a couple more months as we tried to work around this kind of cost problem.

35. 6:18

    And, you know, cost wasn't the only thing that we were concerned with. Um, as we started to scale some of these use cases, we quickly ran into a wall with prompt engineering.

36. 6:26

    It only takes you so far. Um, one thing we realized that even though GPT is really smart, it's not a financial expert. So you had to give it really detailed instructions and examples, uh, to really make it work with all kinds of use cases that we were trying to target.

37. 6:40

    Um, so it's hard to generalize those kinds of prompts. They become really long, convoluted. It's always a cat and mouse chase with you fix it for a certain scenario, and it breaks for another one.

38. 6:49

    You fix it for that one, it breaks for the previous one. And so you're always going back and forth. We didn't have any prompt versioning, so we had to figure out a better way to make this work for all of our use cases.

39. 7:02

    And so the TLDR here is that, you know, we, we didn't want to adopt that initial solution that I just talked about earlier in the slides because of its scaling challenges and just because it was so inefficient.

40. 7:12

    But we kind of ran into the same scaling challenges with GPT, where it was expensive because we couldn't really optimize for caching because of the variability in responses and the prompt tweaks we were making all the time.

41. 7:25

    And the baseline latency that we were finding was actually really slow, so we couldn't... You know, it was overall, we couldn't scale concurrently. And similar to human errors that were kind of, uh, in a different nature, we had AI errors, which were just hallucinations that were hard to catch.

42. 7:39

    Um, and we just couldn't scale with this kind of system. But we still kept it in production because for specific use cases, it was actually really, really good.

43. 7:49

    And so now the problem shifted from solving that core problem of trying to make sense of unstructured data. That was solved with GPT. Now, the problem shifted to how do we scale this system?

44. 7:59

    How do we build a robust, uh, you know, agentic workflow that can handle this kind of volume reliably? And so some of the ballpark figures that we came up with, you know, is that we-- we're gonna be at least making sixteen million requests per day.

45. 8:12

    Uh, we're gonna have at least one hundred K concurrent load, and, you know, we need minimal latency to, um, handle this kind of real-time agentic workflow, so sub two hundred milliseconds.

46. 8:22

    And, you know, so the natural next step for us was, like, we thought to ourselves, do we buy more GPUs? Do we host our own model? Like, what do we do at this point?

47. 8:30

    Um, so at that, at that point, OpenPipe comes in.

48. 8:34

    Yeah. So about a year ago, we started working with Method on solving these issues that, uh, Mustafa just listed. And we actually found that the, the-- those three issues he listed, right, which are quality, cost, um, and latency, are very common.

49. 8:47

    Um, these are things that, you know, across almost everyone we work with, uh, at least some subset of those are really top of mind. Um, and so with, uh, Method specifically, we were working on, okay, how do we, how do we solve those problems in a way that makes this, uh, you know, a, a viable business for

50. 9:02

    you? So, uh, the first thing we did was start measuring error rates. Um, you know, like, like he mentioned, uh, even AI models are not perfect. Uh, these are all probabilistic systems.

51. 9:12

    Getting to a zero percent error rate was not really feasible, but we were able to see different models had different, uh, had different performance characteristics there. So on modern models on the tasks they're doing, these are the rates we're seeing.

52. 9:24

    On GPT-4o, um, we're at about an eleven percent error rate, uh, and with o3-mini, it's much better. It's around a four percent error rate. Um, the way you measure that is going to be specific to your business, and that, that's actually true to some extent for all three of these things we'll talk about.

53. 9:37

    Uh, in the case of Method, this is actually relatively easy to measure, luckily, because they have this agentic workflow, but, like, ultimately, what the agent is trying to do is, is fill out, um, you know, extract all this information he was talking about, bank balances, things like that.

54. 9:50

    And so you can, you can have a human go through the flow and figure out what the real number should be, and then you can compare an agentic system's final outputs to that and see if it was successful or not, um, which, which made this part relatively easy to calculate.

55. 10:02

    Uh, so these, these are kinda the error rates we're getting. Um, on the latency point of view, uh, we see that GPT-4o is around a second, uh, to respond.

56. 10:12

    Uh, and then o3-mini takes about five seconds for their specific task. Again, this is somewhat task dependent, uh, depending on how much, uh, you know, for example, o3 has to think.

57. 10:20

    Uh, as you're measuring this, you also wanna make sure that you're using real production conditions, that you're actually doing, um, you know, like a, a real diversity of tasks, uh, that, that match what you're actually doing and at a reasonable concurrency level that matches your production.

58. 10:32

    Um, and we also measured the cost. Um, so again, cost, uh, this is something that is going to obviously be specific, and how much it matters is also very specific to your use case as well.

59. 10:40

    Um, interestingly, o3-mini, even though it has a much lower per token cost than GPT-4o, uh, if you just look at, like, the pricing page on the API, for their specific use case, uh, we found it was a little bit more expensive because it has-- it generates many more reasoning tokens, so it has much longer outputs.

60. 10:56

    Um, again, though, uh, this is somewhat task dependent, so I just recommend, um... Actually, just, just as an aside, I would recommend once you get to the point that you're trying to optimize, that you have sort of that initial proof of concept with, with some model, something that works, I think it's really worthwhile to-- It can be

61. 11:11

    as simple as, like, literally just writing, like, you know, three different py-Python scripts that, like, are able to categorize each of these for a different model. Um, and then as new models come out, you'll be able to quickly tell how they're doing.

62. 11:22

    Um, okay. Once you've done, or in this case, once we've done this, this sort of, um, benchmarking of where the models are, the next question is, all right, what is-- w-where do we need these models to be?

63. 11:32

    Where do we need to get to? Um, and so again, this is very task dependent. Uh, in the case of Method, uh, they do have spec-- like, they have, um, extra checks that happen after this, where they look and see, okay, are the numbers that came out plausible?

64. 11:45

    Do they match, you know, the types of things we've seen before? All the, all these different kinds of checks they're doing. And so they didn't need to get all the way down to a zero percent error rate, but of course, those checks are still fallible.

65. 11:53

    And so, um, if it's over a certain point, then, uh, then some fraction of those errors are gonna get through, and that's gonna be bad. And so we found around a nine percent error rate was, was able to get them what they needed.

66. 12:03

    Um, from a latency point of view, so the way their agent works is a real-time system. Uh, it, it needs to be able to respond quickly to, to move, uh, through the, the, basically, like, through the whole flow to get the information it needs.

67. 12:16

    And so they did have a hard latency cutoff. Um, we see a wide variety in this for what it's worth. We have some customers that I talk to who it's like, "Hey, if I get a result back at some point in the next few days, like, that's totally fine.

68. 12:25

    This is a background batch process." Um, we have other customers who are doing real-time voice with a human on the end-- other end of the line, and it's like, "Hey, you know, if I'm over five hundred milliseconds, that's not gonna work for me."

69. 12:35

    And so again, you just have to know for your specific case how much this matters. Same with cost. Um, in their case, because of that very high volume, as Mustafa was mentioning, cost is pretty important to them.

70. 12:45

    Um, again, depending on your use case, usually mostly dependent on how high volume it is, um, will determine how much cost matters to you. But, but it's something you, you should know these numbers for your specific task as you're comparing different models.

71. 12:57

    Okay. So, um, we're looking here at this, uh, of course, as you're looking at this, this slide, you can-- y-you, you may see there's a problem here, which is, um, of the two models we're comparing, at least, none of them actually meet all three of the requirements we need to be able to deploy this in production.

72. 13:13

    And, uh, you know, GPT-4o on both the error rate as well as the cost, we're not quite there. Um, and then o3-mini, uh, on the cost, but especially on the latency, it's just not going to work for what we need.

73. 13:24

    So this is the point at which, uh, Method came, and they talked to us. We're like, "Hey, we're not able to hit what we need here." Um, because again, we're not, uh, yeah, we're-- th-th-these, these models aren't getting us where we need to be.

74. 13:37

    So what we work on at OpenPipe is fine-tuning. We work on building custom models for your specific use case. And so I'm gonna talk about why you would wanna do that and how that helps in this case.

75. 13:46

    Um, first, I would say fine-tuning is a power tool. Uh, it does take more time. It takes more, uh, engineering investment than just prompting a model. Uh, so you don't really want to do that until you have actually benchmarked the production models, just prompting them and seeing whether they work or not.

76. 14:04

    Um, so in this case, in Method's case, and in m-- all of our customers' cases, uh, they, they found that they were not able to hit the numbers they needed.

77. 14:10

    Um, and so that's the time you wanna bring in fine-tuning.

78. 14:13

    Um, so let's look at-- We were able to fine-tune a model and see, uh, how that was able to help, uh, because it can actually really bend that price performance curve a lot.

79. 14:22

    Um, so on the, the error rate, uh, which is basically just the inverse of, of accuracy, if you wanna measure it that way, um, we were able to get to a place where we were doing significantly better than GPT-4o, and importantly better than that threshold they needed.

80. 14:34

    Uh, this used to actually be much harder to achieve. It required a lot of manual, uh, labeling of data and things like that. It's actually become much easier over time because of the existence of models like now o3-mini, um, which allows you to just use your production data.

81. 14:48

    You can, you can use your, uh, basically the inputs you use in production. You can f-- uh, generate outputs for them using a model like o3-mini and train on them.

82. 14:57

    We find, like in this case, that often you're not able to quite get, uh, to the, the performance of the, the teacher model, the model o3-mini in this case that you're using.

83. 15:05

    But you can get quite close to it and usually do much better than, you know, uh, a slightly less good but much, much larger model. Um, you know, in this case, uh, the model we ended up deploying with them is just an eight billion parameter Llama 3.1 model.

84. 15:17

    And, and we find that actually for the majority of our customers, a model that large or smaller is, is good enough and is able to hit the numbers you need from quality.

85. 15:25

    Um, but, uh, yeah, the, the important thing is to be able to benchmark that and to answer that question for yourself.

86. 15:30

    Um, on the latency point of view, because actually, this is, this is sort of the magic of being able to move to that much smaller model. Because we've got this eight billion parameter model, it is way easier to deploy in a low latency way.

87. 15:42

    Um, there's just many-- fewer, fewer calculations, fewer sequential calculations with the number of layers, and so you can get just a much lower latency. You can even-- and we, we didn't actually have to do this in Method's case, um, but something you can do is you can train this model.

88. 15:53

    You can deploy it within your own infrastructure, co-locate it with the application code that's using it, um, and even completely eliminate the, the network latency.

89. 16:01

    Uh, and then finally, uh, on the cost front, again, just because this is such a smaller model, um, you end up with a much, much lower cost. Uh, and so that, for many of our customers, is a big-- is, is incredibly important, is to be able to get that performance number you need, um, while still maintaining a

90. 16:17

    relatively low cost. Um, in, in Method's case, we were actually able to far exceed the sort of cost thresholds that they were looking for to make this viable, um, which means that they don't have to worry about this from sort of a unit economics point of view, uh, in, in, in the way that they did when they

91. 16:30

    were using the larger models. Um, so just, um, to sort of reiterate what I started with before, um, this is a power tool. Uh, the fine-tuning, uh, is-- it does take a fair amount of work, um, not an extreme amount of work, but significantly more work than you do for prompt engineering.

92. 16:48

    However, if you're not able to get to the reliability numbers you need, uh, through just prompt engineering with the models that exist out there without tuning, it is a viable way to very strongly bend that price performance curve and get to a much better place, uh, which, uh, which, which can help you get to a very large

93. 17:05

    scale in production, just like Method did.

94. 17:10

    Nice. Um, so yeah, just to wrap up here, uh, one thing that, or at least a couple, couple points that we wanna highlight is that, you know, the reason we put two engineers in the title is also because it's not that, it's not that complicated, right?

95. 17:24

    You can get away with using-- We identified a specific use case, and we got away with just using the cheapest model that was out there. Uh, we fine-tuned it.

96. 17:31

    We already had the data from GPT in production, so we already had the data. We didn't have to go digging around for the data in the first place. Uh, so we already used that, and we used the cheapest model that gave us the fastest performance.

97. 17:42

    And, you know, you don't need to buy your own GPUs. Um, and the, the other thing that we realized is that productionizing AI agents actually requires a little bit of, uh, some level of openness, uh, and patience from the engineering team, from the leadership team.

98. 17:55

    It's because when you write code, we're always writing code that just works. You push out a feature and never breaks because you're not changing anything. But with AI agents, you-- it takes some time to get to a point where it's, like, production ready and actually gives you the responses that you're looking for.

99. 18:09

    Um, and, you know, I, I feel compelled to say something about as-- to mark the top of the traditional software engineering job. So I'll leave you with these last few words.

100. 18:18

     If you're in SWE, pivot to AIE. [laughs] [upbeat music] Thank you.

101. 18:23

     Thanks, everyone. [clapping] [upbeat music]