AI Engineer Summit 2025
How we scaled 500m AI agents in production with 2 engineers
Read the talk
Making financial-data agents affordable enough to scale
Method’s missing financial fields led from manual bank calls to GPT-4, then to a smaller model trained on production tasks to meet quality, latency, and cost requirements.
From a talk by Mustafa Ali and Kyle Corbitt
Before you start: Familiarity with LLM API calls, prompts, and the basic purpose of fine-tuning will help you follow the model-selection decisions.
Getting financial data that has no central API
How do you deliver financial information when the data your customers need is still difficult to access? Method aggregates liability data from hundreds of sources, including credit bureaus, Visa and Mastercard, financial institutions, and third parties. Its fintech, bank, and lender customers use that information for refinancing, loan consolidation, liability payments, and personal finance management. Kyle Corbitt introduces the case as Method scaling to more than 500 million agents, without defining the counting unit or time window. His company, OpenPipe, helps build, train, and deploy open-source models, using production signals to improve them over time.
The immediate problem was specific: customers already received balances and payment information, but also wanted an auto loan’s payoff amount or a mortgage’s escrow balance. Mustafa Ali and the Method team asked their data partners how to obtain those fields. They found no accessible central API. Direct bank integrations were an option, but Method expected them to take at least a couple of years. An early-stage company needed a route it could put into production much sooner.
Customers were already solving the problem through people. Offshore contractors called banks on behalf of the company and end consumer, authenticated, gathered the information, and sent it back for checking. The checked results then entered financial platforms, where they could support underwriting or appear directly to users.
That workflow scaled with headcount: one person could handle only one task at a time. Its synchronous nature made it slow, and human errors created another staffing requirement for checking results. The consequential failure was inaccurate financial information reaching a customer. Yet the process had recognizable API components—request, authentication, response, and validation. The missing interface was a way to turn unstructured information into reliable financial fields.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
GPT-4 makes extraction work—and exposes the next bottleneck
The launch of GPT-4 gave Method a promising tool for parsing unstructured data. Its strength at tasks such as summarization and classification suggested that the information-gathering workflow could be automated. Method built an agentic workflow around it, then expanded the extraction tasks performed within a single API call to get more value from each expensive request. Controlled production tests worked well.
As traffic increased, the economics became harder to ignore. Mustafa reports a $70,000 bill for Method’s first month of GPT-4 production usage. Leadership was unhappy with the expense, but the value was substantial enough to keep the system running for at least a couple more months while the team searched for a cost solution.
Prompting also became a maintenance problem. A generally capable model still needed detailed financial instructions and examples. Prompts grew long and convoluted; fixing one scenario could break another, then fixing that regression could undo the first improvement. Method had no prompt versioning to help manage the changes.
The automated system began to reproduce the scaling problems of the manual one. Method attributed limited caching opportunities to variable responses and frequent prompt changes. That should be distinguished from provider prompt caching, which reuses shared input prefixes: variable outputs alone do not prevent prefix reuse, and the talk does not identify Method’s caching implementation. Slow baseline responses constrained concurrent operation, while hallucinations introduced errors that were difficult to catch. The team retained the workflow for the specific use cases where it worked well.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure the task before choosing the model
The problem had shifted from making extraction work to making the workflow reliable at volume. Method’s planning requirements were at least 16 million requests per day, at least 100,000 concurrent load, and latency below 200 milliseconds. These were targets for the proposed system, not measurements of simultaneously achieved performance. Buying GPUs and hosting a model were possibilities under consideration. OpenPipe joined the effort about a year before the talk, framing the decision around quality, latency, and cost.
The first step was to measure errors. Corbitt reports approximately 11% error for GPT-4o and 4% for o3-mini on Method’s tasks. Method could establish reference answers by having a human complete the information-gathering flow and identify the correct financial values. Comparing the agent’s final outputs with those values gave the team an outcome-based evaluation, rather than an assessment of whether an intermediate response sounded convincing. Zero error was not considered a feasible expectation.
Corbitt reports response latency of roughly one second for GPT-4o and five seconds for o3-mini on Method’s task. Reasoning duration depends on the work being done, so the useful measurement is one taken across the task diversity and concurrency expected in production. A fast response to a single convenient prompt does not establish the latency of the deployed workflow.
Cost introduced another surprise: although o3-mini had lower per-token pricing, Corbitt says it was slightly more expensive for Method because it generated more reasoning tokens and longer outputs. The comparison therefore looked like this:
| Dimension | GPT-4o | o3-mini |
|---|---|---|
| Extraction quality | More errors | Fewer errors |
| Response latency | Faster | Slower |
| Cost for Method’s task | Lower | Slightly higher |
Per-task cost matters more than the price of an individual token. A model can charge less per token and still cost more to complete the same job.
Once a proof of concept works, Corbitt recommends keeping simple Python scripts for evaluating quality, latency, and cost as new models arrive. For example, a summary over recorded runs can keep those dimensions separate. Here, each row represents a completed extraction with a human-checked expected result, measured latency, and recorded cost:
python
from statistics import mean
def summarize(runs):
if not runs:
raise ValueError("At least one completed run is required")
errors = sum(
run["actual"] != run["expected"]
for run in runs
)
return {
"runs": len(runs),
"error_rate": errors / len(runs),
"mean_latency_ms": mean(
run["latency_ms"] for run in runs
),
"cost_per_1000_calls_usd": 1000 * mean(
run["cost_usd"] for run in runs
),
}
This example uses exact equality for the final extracted result. The application must define what constitutes a correct result and capture representative runs; the summary function only aggregates those observations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Set thresholds for the whole application
With the baseline measured, the next question was what performance the application actually required. Method already applied downstream checks to extracted numbers: were they plausible, and did they resemble values seen before? Corbitt reports that around 9% model error met Method’s needs when combined with those downstream checks. That is a model-stage tolerance, not a claim that delivering incorrect financial data at that rate was acceptable. The checks themselves were fallible, so increasing the incoming error rate would allow more mistakes through.
Latency requirements came from the interaction. Method’s agent had to respond quickly enough to move through a real-time information-gathering flow, creating a hard cutoff. Corbitt contrasts that with two other workloads:
- Background batches: Some customers can wait days for a result.
- Human-facing voice: Some customers cannot tolerate response latency above 500 milliseconds.
High request volume made cost especially consequential for Method. Quality, timing, and price therefore needed application-specific thresholds, rather than a single model ranking detached from how the result would be used.
Neither candidate met all three requirements. Corbitt identifies GPT-4o’s error rate and cost as problems; o3-mini’s cost and especially its latency also prevented deployment under the desired conditions. That was the point at which a custom model became worth considering. Fine-tuning requires more time and engineering investment than prompting, so benchmark prompted models first, then tune when the measured gap justifies the work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Train a smaller model on production inputs
The tuned model improved on GPT-4o’s error rate and cleared Method’s quality threshold. Producing the training data did not require manually labeling every example. A stronger reasoning model could generate answers for inputs already encountered in production, reducing the manual work that had previously made this approach harder. The procedure was straightforward:
- Collect inputs from the production workload.
- Generate outputs for those inputs with o3-mini.
- Train a smaller model on the resulting input-output pairs.
This moves the more expensive model’s work into training-data generation, allowing the smaller model to learn the specific task.
The deployed model was Llama 3.1 8B, with eight billion parameters; the talk does not specify its starting checkpoint. Its quality did not quite match the o3-mini teacher, but it came close enough to outperform GPT-4o on this workload and satisfy Method’s needs. Corbitt says models of that size or smaller meet the quality requirements of most OpenPipe customers, while emphasizing that each application still needs its own benchmark.
The smaller model also made low-latency serving easier. Corbitt attributes that to fewer calculations, including fewer sequential calculations through model layers. A further option is to deploy the model alongside the application that calls it, removing the remote network round trip. Method did not need that co-location step.
The same reduction in model size lowered serving cost. Corbitt reports that the tuned model comfortably beat Method’s cost requirement, easing the unit-economics concern created by the larger models. Fine-tuning still added engineering work, but it changed the available trade-off: the team could obtain the required task quality from a model that was cheaper and easier to serve quickly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why two engineers could do it
Mustafa explains the title’s two-engineer claim through the scope of the work. The team identified a specific use case and fine-tuned an inexpensive model that delivered the speed it needed. Existing GPT production data supplied the training material, so there was no separate hunt for a dataset. The earlier production system had created both customer value and the examples needed to improve its economics.
The team did not need to buy its own GPUs. It did need openness and patience from engineering and leadership while the agent’s responses improved enough for production. Deploying an agent involves iteration on behavior, not just shipping a feature and assuming its responses will already meet the application’s needs. That organizational willingness to keep refining the system was part of making the technical approach work.
Mustafa closes with a career joke that fits the work the team had just described: “If you're in SWE, pivot to AIE.”
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Specifications, licensing and runnable inference examples for the instruction-tuned 8B member of the Llama 3.1 family.
Historical introduction to o3-mini, its reasoning-effort settings and OpenAI's launch evaluations.
Further reading
Code and setup instructions for request capture, dataset management, fine-tuning and evaluation. The README currently notes paused development of the open-source version.
Explains input-prefix reuse, original model eligibility and cache-usage reporting at the October 2024 launch.
Updates since the talk
- Method's current platform overviewDocumentation
Explains Method's liability discovery, account data, payments and financial application use cases.
Read the complete timestamped transcript
- 0:00
[on-hold music] Hey, everybody.
- 0:17
Uh, yep, I'm Kyle Corbitt from OpenPipe, and I'm here with Mustafa Ali from Method.
- 0:22
Hello.
- 0:22
We're gonna be talking about how Method has scaled in production to over five hundred million agents, uh, and basically all the, the tricks they use to, to make that actually work.
- 0:31
So a little bit about Method is that we essentially collect and centralize liability data from across hundreds of different data sources. This includes tapping into the credit bureaus, uh, connecting with the card networks like Visa and Mastercard, um, and just direct connections with the financial institutions and various other third-party sources.
- 0:48
And, you know, we, uh, sort of aggregate and enhance this data and serve it to our customers, who are typically other fintechs, banks, or lenders, and they use this enhanced data to, um, anything really to do with debt management, so refinancing, loan consolidation, liability payments, or just personal finance management.
- 1:07
Um...
- 1:09
Yeah. And at OpenPipe, what we do is we help you build, uh, train, and deploy open-source models, um, for actual usage. We also let you use in production your signals you get from users, from the environment to improve your model continuously over time, and that's some of the things we'll be talking about, uh, what we did with
- 1:26
Method.
- 1:27
Nice. So one of the early challenges that we faced at Method while coming up with this, you know, aggregation pipeline, uh, was that some of our customers basically came to us and said, "You know, it's really nice that you can give us the balance and payment information on a specific liability for their end consumers.
- 1:46
But you know what would be really nice is if you could also give us some of these liability-specific data points, like the payoff amount on an auto loan or the escrow balance for a mortgage."
- 1:56
And, you know, we said, "Okay, let's do some research." So we go back to some of our data partners and basically ask them, you know, "Is there anything, you know, we can plug into to get these kinds of data points?"
- 2:05
And what we found was there's really no central API that we could get access to that would allow us to get some of these data points. And, of course, ideally, we would want to work with, uh, directly with the banks, but, you know, having already worked with banks before and just from initial conversations, we realized that it
- 2:21
would easily take up to at least a couple of years before getting anything solid done. And, you know, we, we're an early-stage company, so we wanna build for the customer fast.
- 2:29
Um, and so that's really what we're trying to come up with, a solution that we can just, you know, uh, push into production tomorrow.
- 2:37
And so just to get a better understanding of how some of these companies are operating today, uh, the services that they're providing today, how are they doing that in the first place, right?
- 2:44
They, they must be getting that data somehow. So we go back to some of these customers and basically ask them, you know, "How are you guys operating?" And what they tell us is, is kind of interesting.
- 2:53
So a lot of these companies, they basically hire offshore teams of contractors and, you know, they, uh, uh, these teams are basically responsible for calling these banks, um, on behalf of the company and the end consumer.
- 3:06
Uh, they authenticate with the banks, gather the necessary information, somebody has to proof-check it, it gets sent back, um, and then it gets integrated into the financial platforms. Um, and it gets surfaced to the user, is used for underwriting, stuff like that.
- 3:19
And so that's the status quo that we're dealing with here. And when you think about it, that's a very inefficient manual process, right? It's, it's-- When you try to think about scaling, it doesn't really scale.
- 3:30
It's a very, um-- It has a lot of problems. You know, it's expensive because one person can only do one thing at a time, right? So if you want to scale, uh, you basically have to hire more people.
- 3:41
And for the same reason, because it's so synchronous, it's also really slow. Um, and the main-- I guess the, the biggest problem with that is also that it's, there's a lot of human error involved and, um, you, you need to hire a team to fact-check it, uh, to, to proof-check it.
- 3:55
And, um, uh, it's the, the be- the, the worst thing that you're gonna end up with is to surface basically inaccurate financial information. And so conceptually though, if you think about it, it's kind of like an API, right?
- 4:06
You have the request component, you have the authentication component, you have the response, validation, and all that stuff. Uh, so essentially, when you drill this problem down into the core problem, that's really just trying to make sense of, um, unstructured data, right?
- 4:21
So if only there was this magic tool or software that we could use that was really good at parsing unstructured data.
- 4:29
And, you know, lucky for us, around the time that we were trying to solve this problem, OpenAI announced GPT-4. And, you know, as people like to call it, there was this Cambrian explosion of AI or LLM-enabled applications all around us, and the results were just mind-blowing.
- 4:45
Um, and we thought to ourselves, "You know, this, this, this is the perfect thing for us. This is like a godsend." Uh, so we tried to like, you know, we, we tried to see if there's anything there that we could use.
- 4:54
And if there's one thing that we all know in this room is that, uh, advanced LLMs, especially post-GPT-4, are really good with, um, with parsing unstructured data. So tasks like summarization or classification, they're really good with that kind of thing.
- 5:08
So we wanted to test that theory out and see what that can get us.
- 5:14
And so we put our heads down, hacked together this agentic workflow using GPT-4, and as expected, you know, it worked really well. So we tried to like expand some of our use cases because it, you know, the API costs are high, so we wanted to get as much as we could from a single API call and, you
- 5:30
know, it turned out to be really good at that. So we tried to-- Obviously, this was in a very controlled manner, um, but this was in production, and so we were testing out, uh, different, uh, extractions basically, and, um, you know, everything was re- going really good.
- 5:45
Uh, but as soon as we started to increase a little bit of, uh, traffic, uh, what we found was, you know, the bill had to come due. And, um, [lip smack] it was a lot.
- 5:55
So seventy thousand dollars for our first month in production with GPT-4. And, you know, this was-- this made leadership really unhappy and, you know. But, um, but it was something, it was something they were, they were fine with because the value that we were getting out of GPT-4 was so immense.
- 6:10
Um, and so we actually kept this thing in production for at least a couple more months as we tried to work around this kind of cost problem.
- 6:18
And, you know, cost wasn't the only thing that we were concerned with. Um, as we started to scale some of these use cases, we quickly ran into a wall with prompt engineering.
- 6:26
It only takes you so far. Um, one thing we realized that even though GPT is really smart, it's not a financial expert. So you had to give it really detailed instructions and examples, uh, to really make it work with all kinds of use cases that we were trying to target.
- 6:40
Um, so it's hard to generalize those kinds of prompts. They become really long, convoluted. It's always a cat and mouse chase with you fix it for a certain scenario, and it breaks for another one.
- 6:49
You fix it for that one, it breaks for the previous one. And so you're always going back and forth. We didn't have any prompt versioning, so we had to figure out a better way to make this work for all of our use cases.
- 7:02
And so the TLDR here is that, you know, we, we didn't want to adopt that initial solution that I just talked about earlier in the slides because of its scaling challenges and just because it was so inefficient.
- 7:12
But we kind of ran into the same scaling challenges with GPT, where it was expensive because we couldn't really optimize for caching because of the variability in responses and the prompt tweaks we were making all the time.
- 7:25
And the baseline latency that we were finding was actually really slow, so we couldn't... You know, it was overall, we couldn't scale concurrently. And similar to human errors that were kind of, uh, in a different nature, we had AI errors, which were just hallucinations that were hard to catch.
- 7:39
Um, and we just couldn't scale with this kind of system. But we still kept it in production because for specific use cases, it was actually really, really good.
- 7:49
And so now the problem shifted from solving that core problem of trying to make sense of unstructured data. That was solved with GPT. Now, the problem shifted to how do we scale this system?
- 7:59
How do we build a robust, uh, you know, agentic workflow that can handle this kind of volume reliably? And so some of the ballpark figures that we came up with, you know, is that we-- we're gonna be at least making sixteen million requests per day.
- 8:12
Uh, we're gonna have at least one hundred K concurrent load, and, you know, we need minimal latency to, um, handle this kind of real-time agentic workflow, so sub two hundred milliseconds.
- 8:22
And, you know, so the natural next step for us was, like, we thought to ourselves, do we buy more GPUs? Do we host our own model? Like, what do we do at this point?
- 8:30
Um, so at that, at that point, OpenPipe comes in.
- 8:34
Yeah. So about a year ago, we started working with Method on solving these issues that, uh, Mustafa just listed. And we actually found that the, the-- those three issues he listed, right, which are quality, cost, um, and latency, are very common.
- 8:47
Um, these are things that, you know, across almost everyone we work with, uh, at least some subset of those are really top of mind. Um, and so with, uh, Method specifically, we were working on, okay, how do we, how do we solve those problems in a way that makes this, uh, you know, a, a viable business for
- 9:02
you? So, uh, the first thing we did was start measuring error rates. Um, you know, like, like he mentioned, uh, even AI models are not perfect. Uh, these are all probabilistic systems.
- 9:12
Getting to a zero percent error rate was not really feasible, but we were able to see different models had different, uh, had different performance characteristics there. So on modern models on the tasks they're doing, these are the rates we're seeing.
- 9:24
On GPT-4o, um, we're at about an eleven percent error rate, uh, and with o3-mini, it's much better. It's around a four percent error rate. Um, the way you measure that is going to be specific to your business, and that, that's actually true to some extent for all three of these things we'll talk about.
- 9:37
Uh, in the case of Method, this is actually relatively easy to measure, luckily, because they have this agentic workflow, but, like, ultimately, what the agent is trying to do is, is fill out, um, you know, extract all this information he was talking about, bank balances, things like that.
- 9:50
And so you can, you can have a human go through the flow and figure out what the real number should be, and then you can compare an agentic system's final outputs to that and see if it was successful or not, um, which, which made this part relatively easy to calculate.
- 10:02
Uh, so these, these are kinda the error rates we're getting. Um, on the latency point of view, uh, we see that GPT-4o is around a second, uh, to respond.
- 10:12
Uh, and then o3-mini takes about five seconds for their specific task. Again, this is somewhat task dependent, uh, depending on how much, uh, you know, for example, o3 has to think.
- 10:20
Uh, as you're measuring this, you also wanna make sure that you're using real production conditions, that you're actually doing, um, you know, like a, a real diversity of tasks, uh, that, that match what you're actually doing and at a reasonable concurrency level that matches your production.
- 10:32
Um, and we also measured the cost. Um, so again, cost, uh, this is something that is going to obviously be specific, and how much it matters is also very specific to your use case as well.
- 10:40
Um, interestingly, o3-mini, even though it has a much lower per token cost than GPT-4o, uh, if you just look at, like, the pricing page on the API, for their specific use case, uh, we found it was a little bit more expensive because it has-- it generates many more reasoning tokens, so it has much longer outputs.
- 10:56
Um, again, though, uh, this is somewhat task dependent, so I just recommend, um... Actually, just, just as an aside, I would recommend once you get to the point that you're trying to optimize, that you have sort of that initial proof of concept with, with some model, something that works, I think it's really worthwhile to-- It can be
- 11:11
as simple as, like, literally just writing, like, you know, three different py-Python scripts that, like, are able to categorize each of these for a different model. Um, and then as new models come out, you'll be able to quickly tell how they're doing.
- 11:22
Um, okay. Once you've done, or in this case, once we've done this, this sort of, um, benchmarking of where the models are, the next question is, all right, what is-- w-where do we need these models to be?
- 11:32
Where do we need to get to? Um, and so again, this is very task dependent. Uh, in the case of Method, uh, they do have spec-- like, they have, um, extra checks that happen after this, where they look and see, okay, are the numbers that came out plausible?
- 11:45
Do they match, you know, the types of things we've seen before? All the, all these different kinds of checks they're doing. And so they didn't need to get all the way down to a zero percent error rate, but of course, those checks are still fallible.
- 11:53
And so, um, if it's over a certain point, then, uh, then some fraction of those errors are gonna get through, and that's gonna be bad. And so we found around a nine percent error rate was, was able to get them what they needed.
- 12:03
Um, from a latency point of view, so the way their agent works is a real-time system. Uh, it, it needs to be able to respond quickly to, to move, uh, through the, the, basically, like, through the whole flow to get the information it needs.
- 12:16
And so they did have a hard latency cutoff. Um, we see a wide variety in this for what it's worth. We have some customers that I talk to who it's like, "Hey, if I get a result back at some point in the next few days, like, that's totally fine.
- 12:25
This is a background batch process." Um, we have other customers who are doing real-time voice with a human on the end-- other end of the line, and it's like, "Hey, you know, if I'm over five hundred milliseconds, that's not gonna work for me."
- 12:35
And so again, you just have to know for your specific case how much this matters. Same with cost. Um, in their case, because of that very high volume, as Mustafa was mentioning, cost is pretty important to them.
- 12:45
Um, again, depending on your use case, usually mostly dependent on how high volume it is, um, will determine how much cost matters to you. But, but it's something you, you should know these numbers for your specific task as you're comparing different models.
- 12:57
Okay. So, um, we're looking here at this, uh, of course, as you're looking at this, this slide, you can-- y-you, you may see there's a problem here, which is, um, of the two models we're comparing, at least, none of them actually meet all three of the requirements we need to be able to deploy this in production.
- 13:13
And, uh, you know, GPT-4o on both the error rate as well as the cost, we're not quite there. Um, and then o3-mini, uh, on the cost, but especially on the latency, it's just not going to work for what we need.
- 13:24
So this is the point at which, uh, Method came, and they talked to us. We're like, "Hey, we're not able to hit what we need here." Um, because again, we're not, uh, yeah, we're-- th-th-these, these models aren't getting us where we need to be.
- 13:37
So what we work on at OpenPipe is fine-tuning. We work on building custom models for your specific use case. And so I'm gonna talk about why you would wanna do that and how that helps in this case.
- 13:46
Um, first, I would say fine-tuning is a power tool. Uh, it does take more time. It takes more, uh, engineering investment than just prompting a model. Uh, so you don't really want to do that until you have actually benchmarked the production models, just prompting them and seeing whether they work or not.
- 14:04
Um, so in this case, in Method's case, and in m-- all of our customers' cases, uh, they, they found that they were not able to hit the numbers they needed.
- 14:10
Um, and so that's the time you wanna bring in fine-tuning.
- 14:13
Um, so let's look at-- We were able to fine-tune a model and see, uh, how that was able to help, uh, because it can actually really bend that price performance curve a lot.
- 14:22
Um, so on the, the error rate, uh, which is basically just the inverse of, of accuracy, if you wanna measure it that way, um, we were able to get to a place where we were doing significantly better than GPT-4o, and importantly better than that threshold they needed.
- 14:34
Uh, this used to actually be much harder to achieve. It required a lot of manual, uh, labeling of data and things like that. It's actually become much easier over time because of the existence of models like now o3-mini, um, which allows you to just use your production data.
- 14:48
You can, you can use your, uh, basically the inputs you use in production. You can f-- uh, generate outputs for them using a model like o3-mini and train on them.
- 14:57
We find, like in this case, that often you're not able to quite get, uh, to the, the performance of the, the teacher model, the model o3-mini in this case that you're using.
- 15:05
But you can get quite close to it and usually do much better than, you know, uh, a slightly less good but much, much larger model. Um, you know, in this case, uh, the model we ended up deploying with them is just an eight billion parameter Llama 3.1 model.
- 15:17
And, and we find that actually for the majority of our customers, a model that large or smaller is, is good enough and is able to hit the numbers you need from quality.
- 15:25
Um, but, uh, yeah, the, the important thing is to be able to benchmark that and to answer that question for yourself.
- 15:30
Um, on the latency point of view, because actually, this is, this is sort of the magic of being able to move to that much smaller model. Because we've got this eight billion parameter model, it is way easier to deploy in a low latency way.
- 15:42
Um, there's just many-- fewer, fewer calculations, fewer sequential calculations with the number of layers, and so you can get just a much lower latency. You can even-- and we, we didn't actually have to do this in Method's case, um, but something you can do is you can train this model.
- 15:53
You can deploy it within your own infrastructure, co-locate it with the application code that's using it, um, and even completely eliminate the, the network latency.
- 16:01
Uh, and then finally, uh, on the cost front, again, just because this is such a smaller model, um, you end up with a much, much lower cost. Uh, and so that, for many of our customers, is a big-- is, is incredibly important, is to be able to get that performance number you need, um, while still maintaining a
- 16:17
relatively low cost. Um, in, in Method's case, we were actually able to far exceed the sort of cost thresholds that they were looking for to make this viable, um, which means that they don't have to worry about this from sort of a unit economics point of view, uh, in, in, in the way that they did when they
- 16:30
were using the larger models. Um, so just, um, to sort of reiterate what I started with before, um, this is a power tool. Uh, the fine-tuning, uh, is-- it does take a fair amount of work, um, not an extreme amount of work, but significantly more work than you do for prompt engineering.
- 16:48
However, if you're not able to get to the reliability numbers you need, uh, through just prompt engineering with the models that exist out there without tuning, it is a viable way to very strongly bend that price performance curve and get to a much better place, uh, which, uh, which, which can help you get to a very large
- 17:05
scale in production, just like Method did.
- 17:10
Nice. Um, so yeah, just to wrap up here, uh, one thing that, or at least a couple, couple points that we wanna highlight is that, you know, the reason we put two engineers in the title is also because it's not that, it's not that complicated, right?
- 17:24
You can get away with using-- We identified a specific use case, and we got away with just using the cheapest model that was out there. Uh, we fine-tuned it.
- 17:31
We already had the data from GPT in production, so we already had the data. We didn't have to go digging around for the data in the first place. Uh, so we already used that, and we used the cheapest model that gave us the fastest performance.
- 17:42
And, you know, you don't need to buy your own GPUs. Um, and the, the other thing that we realized is that productionizing AI agents actually requires a little bit of, uh, some level of openness, uh, and patience from the engineering team, from the leadership team.
- 17:55
It's because when you write code, we're always writing code that just works. You push out a feature and never breaks because you're not changing anything. But with AI agents, you-- it takes some time to get to a point where it's, like, production ready and actually gives you the responses that you're looking for.
- 18:09
Um, and, you know, I, I feel compelled to say something about as-- to mark the top of the traditional software engineering job. So I'll leave you with these last few words.
- 18:18
If you're in SWE, pivot to AIE. [laughs] [upbeat music] Thank you.
- 18:23
Thanks, everyone. [clapping] [upbeat music]