# How we scaled 500m AI agents in production with 2 engineers

Mustafa Ali, Method Financial · Kyle Corbitt, OpenPipe

AI Engineer Summit 2025

[Talk page](https://ai.engineer/talks/zM9RYqCcioM-we-scaled-500m-ai-agents-in-production) · [Video](https://www.youtube.com/watch?v=zM9RYqCcioM)

## How we scaled 500m AI agents in production with 2 engineers

Method’s missing financial fields led from manual bank calls to GPT-4, then to a smaller model trained on production tasks to meet quality, latency, and cost requirements.

Familiarity with LLM API calls, prompts, and the basic purpose of fine-tuning will help you follow the model-selection decisions.

### Getting financial data that has no central API

[0:22](https://www.youtube.com/watch?v=zM9RYqCcioM&t=22s)

How do you deliver financial information when the data your customers need is still difficult to access? Method aggregates liability data from hundreds of sources, including credit bureaus, Visa and Mastercard, financial institutions, and third parties. Its fintech, bank, and lender customers use that information for refinancing, loan consolidation, liability payments, and personal finance management. Kyle Corbitt introduces the case as Method scaling to more than 500 million agents, without defining the counting unit or time window. His company, OpenPipe, helps build, train, and deploy open-source models, using production signals to improve them over time.

The immediate problem was specific: customers already received balances and payment information, but also wanted an auto loan’s payoff amount or a mortgage’s escrow balance. Mustafa Ali and the Method team asked their data partners how to obtain those fields. They found no accessible central API. Direct bank integrations were an option, but Method expected them to take at least a couple of years. An early-stage company needed a route it could put into production much sooner.

![Slide titled “Opaque Financial Data Access” mentions payoff amounts and escrow balances, lists missing central access and lengthy bank integrations, and asks “How do we go to market fast?”](https://ai.engineer/api/talk-essay-media/647b0c67db95f160ec0744cc77f4378d7d2aca6451a43ac1342846f685323461)

Opaque financial data access: no central API, and bank integrations can take years.

Customers were already solving the problem through people. Offshore contractors called banks on behalf of the company and end consumer, authenticated, gathered the information, and sent it back for checking. The checked results then entered financial platforms, where they could support underwriting or appear directly to users.

That workflow scaled with headcount: one person could handle only one task at a time. Its synchronous nature made it slow, and human errors created another staffing requirement for checking results. The consequential failure was inaccurate financial information reaching a customer. Yet the process had recognizable API components—request, authentication, response, and validation. **The missing interface was a way to turn unstructured information into reliable financial fields.**

### GPT-4 makes extraction work—and exposes the next bottleneck

[4:29](https://www.youtube.com/watch?v=zM9RYqCcioM&t=269s)

The launch of **GPT-4** gave Method a promising tool for parsing unstructured data. Its strength at tasks such as summarization and classification suggested that the information-gathering workflow could be automated. Method built an agentic workflow around it, then expanded the extraction tasks performed within a single API call to get more value from each expensive request. Controlled production tests worked well.

As traffic increased, the economics became harder to ignore. Mustafa reports a $70,000 bill for Method’s first month of GPT-4 production usage. Leadership was unhappy with the expense, but the value was substantial enough to keep the system running for at least a couple more months while the team searched for a cost solution.

Prompting also became a maintenance problem. A generally capable model still needed detailed financial instructions and examples. Prompts grew long and convoluted; fixing one scenario could break another, then fixing that regression could undo the first improvement. Method had no prompt versioning to help manage the changes.

The automated system began to reproduce the scaling problems of the manual one. Method attributed limited caching opportunities to variable responses and frequent prompt changes. That should be distinguished from provider [prompt caching](https://openai.com/index/api-prompt-caching/), which reuses shared input prefixes: variable outputs alone do not prevent prefix reuse, and the talk does not identify Method’s caching implementation. Slow baseline responses constrained concurrent operation, while hallucinations introduced errors that were difficult to catch. The team retained the workflow for the specific use cases where it worked well.

### Measure the task before choosing the model

[7:49](https://www.youtube.com/watch?v=zM9RYqCcioM&t=469s)

The problem had shifted from making extraction work to making the workflow reliable at volume. Method’s planning requirements were at least 16 million requests per day, at least 100,000 concurrent load, and latency below 200 milliseconds. These were targets for the proposed system, not measurements of simultaneously achieved performance. Buying GPUs and hosting a model were possibilities under consideration. OpenPipe joined the effort about a year before the talk, framing the decision around **quality, latency, and cost**.

The first step was to measure errors. Corbitt reports approximately 11% error for **GPT-4o** and 4% for **[o3-mini](https://openai.com/index/openai-o3-mini/)** on Method’s tasks. Method could establish reference answers by having a human complete the information-gathering flow and identify the correct financial values. Comparing the agent’s final outputs with those values gave the team an outcome-based evaluation, rather than an assessment of whether an intermediate response sounded convincing. Zero error was not considered a feasible expectation.

Corbitt reports response latency of roughly one second for GPT-4o and five seconds for o3-mini on Method’s task. Reasoning duration depends on the work being done, so the useful measurement is one taken across the task diversity and concurrency expected in production. A fast response to a single convenient prompt does not establish the latency of the deployed workflow.

Cost introduced another surprise: although o3-mini had lower per-token pricing, Corbitt says it was slightly more expensive for Method because it generated more reasoning tokens and longer outputs. The comparison therefore looked like this:

| Dimension | GPT-4o | o3-mini |
|---|---|---|
| Extraction quality | More errors | Fewer errors |
| Response latency | Faster | Slower |
| Cost for Method’s task | Lower | Slightly higher |

**Per-task cost matters more than the price of an individual token.** A model can charge less per token and still cost more to complete the same job.

![Three bar charts compare o3-mini and GPT-4o. o3-mini has lower error rate but higher average latency and cost per 1K calls; all charts indicate lower is better.](https://ai.engineer/api/talk-essay-media/d405f2a41cc9ca9222d248c8dd96ed566a1cb41e5ae9df53187edd276193b278)

The Inference Trifecta compares error rate, average latency, and cost per 1K calls.

Once a proof of concept works, Corbitt recommends keeping simple Python scripts for evaluating quality, latency, and cost as new models arrive. For example, a summary over recorded runs can keep those dimensions separate. Here, each row represents a completed extraction with a human-checked expected result, measured latency, and recorded cost:

```python
from statistics import mean


def summarize(runs):
    if not runs:
        raise ValueError("At least one completed run is required")

    errors = sum(
        run["actual"] != run["expected"]
        for run in runs
    )
    return {
        "runs": len(runs),
        "error_rate": errors / len(runs),
        "mean_latency_ms": mean(
            run["latency_ms"] for run in runs
        ),
        "cost_per_1000_calls_usd": 1000 * mean(
            run["cost_usd"] for run in runs
        ),
    }
```

This example uses exact equality for the final extracted result. The application must define what constitutes a correct result and capture representative runs; the summary function only aggregates those observations.

### Set thresholds for the whole application

[11:22](https://www.youtube.com/watch?v=zM9RYqCcioM&t=682s)

With the baseline measured, the next question was what performance the application actually required. Method already applied downstream checks to extracted numbers: were they plausible, and did they resemble values seen before? Corbitt reports that around 9% model error met Method’s needs when combined with those downstream checks. That is a model-stage tolerance, not a claim that delivering incorrect financial data at that rate was acceptable. The checks themselves were fallible, so increasing the incoming error rate would allow more mistakes through.

Latency requirements came from the interaction. Method’s agent had to respond quickly enough to move through a real-time information-gathering flow, creating a hard cutoff. Corbitt contrasts that with two other workloads:

- **Background batches:** Some customers can wait days for a result.
- **Human-facing voice:** Some customers cannot tolerate response latency above 500 milliseconds.

High request volume made cost especially consequential for Method. Quality, timing, and price therefore needed application-specific thresholds, rather than a single model ranking detached from how the result would be used.

Neither candidate met all three requirements. Corbitt identifies GPT-4o’s error rate and cost as problems; o3-mini’s cost and especially its latency also prevented deployment under the desired conditions. That was the point at which a custom model became worth considering. Fine-tuning requires more time and engineering investment than prompting, so **benchmark prompted models first, then tune when the measured gap justifies the work**.

### Train a smaller model on production inputs

[14:13](https://www.youtube.com/watch?v=zM9RYqCcioM&t=853s)

The tuned model improved on GPT-4o’s error rate and cleared Method’s quality threshold. Producing the training data did not require manually labeling every example. A stronger reasoning model could generate answers for inputs already encountered in production, reducing the manual work that had previously made this approach harder. The procedure was straightforward:

1. Collect inputs from the production workload.
2. Generate outputs for those inputs with o3-mini.
3. Train a smaller model on the resulting input-output pairs.

This moves the more expensive model’s work into training-data generation, allowing the smaller model to learn the specific task.

The deployed model was **[Llama 3.1 8B](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)**, with eight billion parameters; the talk does not specify its starting checkpoint. Its quality did not quite match the o3-mini teacher, but it came close enough to outperform GPT-4o on this workload and satisfy Method’s needs. Corbitt says models of that size or smaller meet the quality requirements of most OpenPipe customers, while emphasizing that each application still needs its own benchmark.

The smaller model also made low-latency serving easier. Corbitt attributes that to fewer calculations, including fewer sequential calculations through model layers. A further option is to deploy the model alongside the application that calls it, removing the remote network round trip. Method did not need that co-location step.

The same reduction in model size lowered serving cost. Corbitt reports that the tuned model comfortably beat Method’s cost requirement, easing the unit-economics concern created by the larger models. Fine-tuning still added engineering work, but it changed the available trade-off: the team could obtain the required task quality from a model that was cheaper and easier to serve quickly.

### Why two engineers could do it

[17:10](https://www.youtube.com/watch?v=zM9RYqCcioM&t=1030s)

Mustafa explains the title’s two-engineer claim through the scope of the work. The team identified a specific use case and fine-tuned an inexpensive model that delivered the speed it needed. Existing GPT production data supplied the training material, so there was no separate hunt for a dataset. The earlier production system had created both customer value and the examples needed to improve its economics.

The team did not need to buy its own GPUs. It did need openness and patience from engineering and leadership while the agent’s responses improved enough for production. Deploying an agent involves iteration on behavior, not just shipping a feature and assuming its responses will already meet the application’s needs. That organizational willingness to keep refining the system was part of making the technical approach work.

![Slide titled “Lessons Learned at Method” lists “The cheapest model just works,” “Don’t buy your own GPUs,” and the need for openness and patience when productionizing AI agents.](https://ai.engineer/api/talk-essay-media/8287beaca25f560e2526590fb365dc0a81c40bb5e4de10b83c35e4d6d4392933)

Lessons learned at Method: inexpensive models, no owned GPUs, and patience in productionizing agents.

Mustafa closes with a career joke that fits the work the team had just described: “If you're in SWE, pivot to AIE.”

## Resources

- [Meta's Llama 3.1 model card and inference examples](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) — Specifications, licensing and runnable inference examples for the instruction-tuned 8B member of the Llama 3.1 family.
- [OpenAI o3-mini launch](https://openai.com/index/openai-o3-mini/) — Historical introduction to o3-mini, its reasoning-effort settings and OpenAI's launch evaluations.
- [OpenPipe repository](https://github.com/OpenPipe/OpenPipe) — Code and setup instructions for request capture, dataset management, fine-tuning and evaluation. The README currently notes paused development of the open-source version.
- [Method's current platform overview](https://docs.methodfi.com/guides/overview) — Explains Method's liability discovery, account data, payments and financial application use cases.
- [How OpenAI introduced prompt caching](https://openai.com/index/api-prompt-caching/) — Explains input-prefix reuse, original model eligibility and cache-usage reporting at the October 2024 launch.

## Slides

### Scaling to 500m AI agents in production with 2 engineers

Kyle Corbitt of OpenPipe and Mustafa Ali of Method introduce Method’s experience scaling to over 500 million agents in production with two engineers.

[0:16](https://www.youtube.com/watch?v=zM9RYqCcioM&t=16s) [Method and OpenPipe title slide about scaling AI agents in production.](https://assets.ai.engineer/sites/aiecode2025/sha256/9e131be2ede59b0152ccfb6edfbbedf81225d419cd23d48f9d13013595e65b0c.webp)

Method | OpenPipe

Scaling to 500m AI agents in production with 2 engineers

AGENTS AT WORK

### About OpenPipe

OpenPipe helps teams train and deploy open-source models, using production signals to improve them over time. The slide claims 90% fewer production errors, 5 minutes to start collecting training data, and 8x lower cost than GPT-4o.

[1:15](https://www.youtube.com/watch?v=zM9RYqCcioM&t=75s) [OpenPipe describes fine-tuning for production apps and presents claims about errors, setup time, and cost.](https://assets.ai.engineer/sites/aiecode2025/sha256/09407f487fb14135ce61d679658f7c84bc1dd966db4ae461bfa0ef005ce81dd4.webp)

About OpenPipe

Fine-tuning for production apps

Train higher-quality, faster models that continuously improve.

90%
Fewer errors in production

5min
To start collecting training data

8x
Cheaper than GPT-4o

### Opaque Financial Data Access

Method’s customers wanted payoff amounts and escrow balances, but Method found no central API for these data points. Direct bank integrations can take years, raising the question of how to reach market quickly.

[1:50](https://www.youtube.com/watch?v=zM9RYqCcioM&t=110s) [Slide describes demand for payoff amounts and escrow balances, the lack of a central API, and lengthy bank integrations.](https://assets.ai.engineer/sites/aiecode2025/sha256/06305631664a576494787a545015b351e4979caef832cd2e95d7b783095675f0.webp)

Opaque Financial Data Access

Demand for valuable yet inaccessible data points, like payoff amounts, escrow balances, etc.
• No central API or known data source
• Direct integrations with banks can take years

How do we get to market fast?

AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### Opaque Financial Data Access

The company needed payoff amounts and escrow balances, but found no accessible central API. Direct bank integrations could take years, too long for an early-stage company building for customers.

[2:22](https://www.youtube.com/watch?v=zM9RYqCcioM&t=142s) [Slide describing inaccessible financial data, the lack of a central API, and lengthy bank integrations.](https://assets.ai.engineer/sites/aiecode2025/sha256/023a12a3d4f0c1814c5a9f604fac24a8811b579e32eb71edaa406ffdd8a87970.webp)

Opaque Financial Data Access

Demand for valuable yet inaccessible data points, like payoff amounts, escrow balances, etc.
• No central API or known data source
• Direct integrations with banks can take years

How do we get to market fast?

### The Status Quo

Companies hire offshore contractors to call banks, authenticate and gather information. After a proof-check, the data is integrated into financial platforms and surfaced to users.

[2:59](https://www.youtube.com/watch?v=zM9RYqCcioM&t=179s) [Projected slide titled The Status Quo with an illustration of call-center workers.](https://assets.ai.engineer/sites/aiecode2025/sha256/c432c88b606bb6bce1e05e3da3f72628edf40db3e222172d343f098c2d265bb4.webp)

The Status Quo

### The Status Quo

The slide estimates ~160 cycles in 8 hours of constant work at ~3 minutes per cycle. Scaling the manual process requires more people, while human errors require proof-checking.

[3:23](https://www.youtube.com/watch?v=zM9RYqCcioM&t=203s) [Slide describing the cost, slowness and error risks of manual work, alongside a call-center illustration and a throughput estimate.](https://assets.ai.engineer/sites/aiecode2025/sha256/4b5c62bc3d084e991e82ff8b418f61a77f91006632dcc1c2d43adfb1ca39abf3.webp)

The Status Quo

Problems with this approach:
• Expensive
  ◦ Higher concurrency requires more people
• Slow and synchronous process
  ◦ Can only do one task at a time
  ◦ Training time & feedback loop
• Prone to human error
  ◦ Data accuracy issues

Back of the envelope math:
• Avg cycle takes ~3 mins
• Assuming constant cycles for 8 hrs => ~160 cycles

### Enter GPT-4 — A Moment of AGI Optimism

GPT-4 arrived as the team sought to parse unstructured data. The announcement and API invitation capture their early optimism about using it for tasks like summarization and classification.

[4:30](https://www.youtube.com/watch?v=zM9RYqCcioM&t=270s) [Slide pairing the GPT-4 announcement with a screenshot of an API invitation.](https://assets.ai.engineer/sites/aiecode2025/sha256/094254551c2c6499fc370666f1fdde62f8bf82c3f84842a5fc425a9ee0e24923.webp)

Enter GPT-4 — A Moment of AGI Optimism

OpenAI
@OpenAI
Announcing GPT-4, a large multimodal model, with our best-ever results on capabilities and alignment: openai.com/product/gpt-4

Your GPT-4 API invite is here
OpenAI
To: Me
You're invited to use the OpenAI GPT-4 API!
You can now access GPT-4 models with 8K context via the existing OpenAI API.
Get started
As an early customer we'd love to hear about your experience. Feel free to share feedback on our community forum or reach out directly to our team.
Best,
The OpenAI team

Interface text:
Post
0:46 / 0:49
12:00 PM · Mar 14, 2023 · 11.3M Views
jose · VIP · 12:28 PM
Screenshot 2023-03-16 at 1.27.32 PM.png
1:26 pm
4

### GPT-4 in Production

The first month in production cost about $70,000. The October dashboard shows a sharp spending spike around October 20, with GPT-4 accounting for $66,068.20 of the monthly bill.

[5:51](https://www.youtube.com/watch?v=zM9RYqCcioM&t=351s) [Monthly spending dashboard showing a large GPT-4 cost spike.](https://assets.ai.engineer/sites/aiecode2025/sha256/8ed67484dde71059260d1f043d6e0b3daa7198a49c2753b2992779693a1e11cf.webp)

GPT-4 in Production 💸

Monthly Spend: $70,012.63
Y-axis ticks: $0, $5K, $10K, $15K, $20K
Timeline: 01 Oct, 08 Oct, 15 Oct, 22 Oct, 29 Oct

Project's Monthly Bill
Oct 1 - 31
100%
$70,012.63 / $5,000.00 limit

GPT-4: $66068.20
Y-axis ticks: $0, $5K, $10K, $15K, $20K
Timeline: 01 Oct, 07 Oct, 13 Oct, 19 Oct, 25 Oct, 31 Oct

GPT-3.5: $1730.99
Y-axis ticks: $0, $65, $130, $250
Timeline: 01 Oct, 07 Oct, 13 Oct, 19 Oct, 25 Oct, 31 Oct

Interface text:
Increase limit
View monthly bill for the whole organization

### Prompt Engineering Limitations

Financial use cases required detailed instructions, but prompts became hard to generalize across banks and goals. Fixing one scenario could break another, leaving the team chasing edge cases.

[6:21](https://www.youtube.com/watch?v=zM9RYqCcioM&t=381s) [Bullets describe difficulties generalizing financial prompts and handling edge cases, alongside excerpts from team discussions.](https://assets.ai.engineer/sites/aiecode2025/sha256/61cb1b5fcc7f7b1c7b341c44cc9d72d3024b04680d47ac4807d8195c85f22a6c.webp)

Prompt Engineering Limitations

• GPT is not a financial expert, requires detailed instructions and examples
• Hard to generalize prompts due to various types of unstructured data (different goals, different banks, etc)
• Prompts become long & convoluted in trying to handle edge cases and hallucinations

i went absolutely crazy today just prompt engineering
so i might lol
i lost brain cells

Big brain idea, tell gpt to not hallucinate
#542 Tell extractor gpt to not hallucinate, also minor improvements

next level prompt engineering

i think this hacky, sub prompts/problems can get us closer
after that though i think we'll need our own models

Interface text:
Assignees
@josebet1, @chboggs, @jmdelcarmen, @mustafalii
MethodFi/keep  Apr 7th, 2023  Added by GitHub
1 reply
jose  Apr 7th, 2023 at 4:11 PM

### Similar Scaling Challenges

Variable responses and frequent prompt changes made caching difficult, while latency and hallucinations limited scale. The team kept GPT-4 in production because it worked well for specific use cases.

[7:06](https://www.youtube.com/watch?v=zM9RYqCcioM&t=426s) [GPT-4 scaling challenges include expense, slow responses and AI errors; the team kept it in production because it performed well.](https://assets.ai.engineer/sites/aiecode2025/sha256/4d53d0b0e576f45b52ef74648ab7db8ff3f6b9f161d12c140ad0fa104b3cee7a.webp)

TLDR; Similar Scaling Challenges

Challenges with GPT-4:

• Expensive
  ○ Highly variable / nondeterministic responses
• Slow
  ○ Limited concurrency / API rate-limits
  ○ High latency
• AI errors
  ○ GPT-4 is smart but not a domain expert
  ○ Hallucinations, convoluted prompts etc

We still kept it in production because it was so good!

### Solving for Scale – 500m Accounts

GPT-4 solved the unstructured-data problem, shifting the challenge to scaling agents: targets included 16M requests per day, ~100k concurrent load, sub-200ms latency and a 99.999% SLA.

[7:49](https://www.youtube.com/watch?v=zM9RYqCcioM&t=469s) [Slide states GPT-4 solved the core problem and presents scale targets: 99.999% SLA, 16 million daily requests, approximately 100,000 concurrent load, and sub-200-millisecond latency.](https://assets.ai.engineer/sites/aiecode2025/sha256/d6258a4052ac9c5cb1c6832c743b99040f3248bb9cf7d8f8815f979c43c1cc1f.webp)

Solving for Scale – 500m Accounts

✅ GPT-4 solved the core problem

Focusing on delivering 99.999% SLA:
• 16M requests per day
• ~100k concurrent load
• Sub 200ms latency

We solved the core problem but how do we now build robust, scalable, and reliable agents to handle our volume?

AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### The Inference Trifecta: Error Rate and Latency

For this task, o3-mini has a lower error rate than GPT-4o but takes about five seconds to respond, versus around one second. Latency depends on the task.

[10:09](https://www.youtube.com/watch?v=zM9RYqCcioM&t=609s) [Bar charts compare o3-mini and GPT-4o on error rate and average latency; the cost chart is still blank.](https://assets.ai.engineer/sites/aiecode2025/sha256/1340257d8d5ab58d6170cf4fcc8099d714daa6f5f5c73f04351048cd5908874f.webp)

The Inference Trifecta

Error Rate (Lower is Better)
Y-axis ticks: 0.00, 0.02, 0.04, 0.06, 0.08, 0.10
X-axis labels: o3-mini, GPT-4o

Average Latency (Lower is Better)
Y-axis ticks: 0, 1, 2, 3, 4, 5
X-axis labels: o3-mini, GPT-4o

Cost Per 1K Calls (Lower is Better)

AIE
OpenPipe
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### The Inference Trifecta: Three Comparisons

For this task, o3-mini has a lower error rate than GPT-4o, but higher latency and cost per 1K calls. The speaker attributes the higher cost to longer reasoning outputs.

[10:31](https://www.youtube.com/watch?v=zM9RYqCcioM&t=631s) [Three bar charts compare o3-mini and GPT-4o on error rate, average latency, and cost per 1,000 calls.](https://assets.ai.engineer/sites/aiecode2025/sha256/87a43582632ef7c3342810c677a7c59553e5c5dcc7029c3ed53652e6633e098f.webp)

The Inference Trifecta

Error Rate (Lower is Better)
Y-axis ticks: 0.00, 0.02, 0.04, 0.06, 0.08, 0.10
X-axis: o3-mini, GPT-4o

Average Latency (Lower is Better)
Y-axis ticks: 0, 1, 2, 3, 4, 5
X-axis: o3-mini, GPT-4o

Cost Per 1K Calls (Lower is Better)
Y-axis ticks: 0, 1, 2, 3, 4, 5, 6
X-axis: o3-mini, GPT-4o

Interface text:
AIE
OpenPipe
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### The Inference Trifecta

Neither model meets all three production thresholds: GPT-4o exceeds the error and cost limits, while o3-mini exceeds latency and cost. For this use case, o3-mini’s longer reasoning outputs also made it more expensive per 1K calls.

[10:38](https://www.youtube.com/watch?v=zM9RYqCcioM&t=638s) [Full-screen charts compare o3-mini and GPT-4o on error rate, average latency, and cost per 1,000 calls.](https://assets.ai.engineer/sites/aiecode2025/sha256/db387c9e6c42ef92f38ecc9ab7dfc09679abe6a6cd0acd579cae459650e93bee.webp)

The Inference Trifecta

Error Rate (Lower is Better)
Y-axis ticks: 0.00, 0.02, 0.04, 0.06, 0.08, 0.10
Categories: o3-mini, GPT-4o

Average Latency (Lower is Better)
Y-axis ticks: 0, 1, 2, 3, 4, 5
Categories: o3-mini, GPT-4o

Cost Per 1K Calls (Lower is Better)
Y-axis ticks: 0, 1, 2, 3, 4, 5, 6
Categories: o3-mini, GPT-4o

[12:24](https://www.youtube.com/watch?v=zM9RYqCcioM&t=744s) [Bar charts compare o3-mini and GPT-4o on error rate, latency, and cost against maximum acceptable thresholds.](https://assets.ai.engineer/sites/aiecode2025/sha256/3188469317b07ccd1ffc815cb3ad689616ce66b6d7e6fd14c72a117b8e0a9886.webp)

The Inference Trifecta

Error Rate (Lower is Better)
Legend: Max Acceptable
Y-axis ticks: 0.00, 0.02, 0.04, 0.06, 0.08, 0.10
Categories: o3-mini, GPT-4o

Average Latency (Lower is Better)
Legend: Max Acceptable
Y-axis ticks: 0, 1, 2, 3, 4, 5
Categories: o3-mini, GPT-4o

Cost Per 1K Calls (Lower is Better)
Legend: Max Acceptable
Y-axis ticks: 0, 1, 2, 3, 4, 5, 6
Categories: o3-mini, GPT-4o

Event text:
AIE
OpenPipe
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### The Inference Trifecta: Threshold Failures

GPT-4o exceeds the error-rate limit, o3-mini exceeds the latency limit, and both exceed the cost limit. Neither model meets all three requirements.

[13:31](https://www.youtube.com/watch?v=zM9RYqCcioM&t=811s) [Red crosses mark GPT-4o's error rate, o3-mini's latency, and both models' costs as exceeding their thresholds.](https://assets.ai.engineer/sites/aiecode2025/sha256/432a4a5a5f03a82681c88a3301343a313cf7793ae26b9ad7096ac284be80386f.webp)

The Inference Trifecta

Error Rate (Lower is Better)
Legend: Max Acceptable
Y-axis ticks: 0.00, 0.02, 0.04, 0.06, 0.08, 0.10
X-axis: o3-mini, GPT-4o

Average Latency (Lower is Better)
Legend: Max Acceptable
Y-axis ticks: 0, 1, 2, 3, 4, 5
X-axis: o3-mini, GPT-4o

Cost Per 1K Calls (Lower is Better)
Legend: Max Acceptable
Y-axis ticks: 0, 1, 2, 3, 4, 5, 6
X-axis: o3-mini, GPT-4o

AIE
OpenPipe
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### Fine-Tuning: A Power Tool

o3-mini meets the error threshold but misses latency; GPT-4o misses the error threshold. Both exceed the cost limit. The speaker recommends benchmarking prompted models before investing in fine-tuning.

[13:38](https://www.youtube.com/watch?v=zM9RYqCcioM&t=818s) [Three charts compare o3-mini and GPT-4o on error rate, latency, and cost against acceptance thresholds.](https://assets.ai.engineer/sites/aiecode2025/sha256/3514ae8ffd8364cc05dae60b513a6a904b84a337a0eb4cb1f56c1f496d250575.webp)

Fine-Tuning: A Power Tool

Error Rate (Lower is Better)
Y-axis ticks: 0.00, 0.02, 0.04, 0.06, 0.08, 0.10
Models: o3-mini, GPT-4o
Legend: Max Acceptable

Average Latency (Lower is Better)
Y-axis ticks: 0, 1, 2, 3, 4, 5
Models: o3-mini, GPT-4o
Legend: Max Acceptable

Cost Per 1K Calls (Lower is Better)
Y-axis ticks: 0, 1, 2, 3, 4, 5, 6
Models: o3-mini, GPT-4o
Legend: Max Acceptable

AIE
OpenPipe
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### Fine-Tuning: Error Rate

Fine-tuned Llama 3.1 8B falls below the error-rate threshold and GPT-4o, though o3-mini remains lower. The latency and cost charts still compare only o3-mini and GPT-4o.

[14:16](https://www.youtube.com/watch?v=zM9RYqCcioM&t=856s) [The fine-tuned Llama 3.1 8B model has an error rate below the displayed threshold; latency and cost charts still show the two original models.](https://assets.ai.engineer/sites/aiecode2025/sha256/40d2aad257e87a60cf3e059bff195acebad732f123eacd57ef6684c1faad04ac.webp)

Fine-Tuning: A Power Tool

Error Rate (Lower is Better)
Y-axis ticks: 0.00, 0.02, 0.04, 0.06, 0.08, 0.10
Models: o3-mini, GPT-4o, Llama 3.1 8B (FT)
Max Acceptable

Average Latency (Lower is Better)
Y-axis ticks: 0, 1, 2, 3, 4, 5
Models: o3-mini, GPT-4o
Max Acceptable

Cost Per 1K Calls (Lower is Better)
Y-axis ticks: 0, 1, 2, 3, 4, 5, 6
Models: o3-mini, GPT-4o
Max Acceptable

### Fine-Tuning: A Power Tool

Fine-tuned Llama 3.1 8B falls below the error and latency limits. Its error rate is lower than GPT-4o’s, and its latency is lower than both GPT-4o’s and o3-mini’s.

[14:33](https://www.youtube.com/watch?v=zM9RYqCcioM&t=873s) [The error-rate chart includes fine-tuned Llama 3.1 8B alongside o3-mini and GPT-4o, with an acceptance threshold.](https://assets.ai.engineer/sites/aiecode2025/sha256/54f5144f4519f9ca7d3b60998a6589999b70147e3b130b81e69f7982a3702803.webp)

Fine-Tuning: A Power Tool

Error Rate (Lower is Better)
Legend: Max Acceptable
Y-axis ticks: 0.00, 0.02, 0.04, 0.06, 0.08, 0.10
X-axis: o3-mini, GPT-4o, Llama 3.1 8B (FT)

Average Latency (Lower is Better)
Legend: Max Acceptable
Y-axis ticks: 0, 1, 2, 3, 4, 5
X-axis: o3-mini, GPT-4o

Cost Per 1K Calls (Lower is Better)
Legend: Max Acceptable
Y-axis ticks: 0, 1, 2, 3, 4, 5, 6
X-axis: o3-mini, GPT-4o

AIE
OpenPipe
AGENT ENGINEERING
HTTPS://AI.ENGINEER

[15:38](https://www.youtube.com/watch?v=zM9RYqCcioM&t=938s) [Charts compare model error rates, latency, and cost against maximum acceptable thresholds.](https://assets.ai.engineer/sites/aiecode2025/sha256/fba816a2ff256c01a1a2529bfb39de385d395d0734b43d77786651d5061845f3.webp)

Fine-Tuning: A Power Tool

Error Rate (Lower is Better)
Legend: Max Acceptable
Y-axis ticks: 0.00, 0.02, 0.04, 0.06, 0.08, 0.10
X-axis: o3-mini, GPT-4o, Llama 3.1 8B (FT)

Average Latency (Lower is Better)
Legend: Max Acceptable
Y-axis ticks: 0, 1, 2, 3, 4, 5
X-axis: o3-mini, GPT-4o, Llama 3.1 8B (FT)

Cost Per 1K Calls (Lower is Better)
Legend: Max Acceptable
Y-axis ticks: 0, 1, 2, 3, 4, 5, 6
X-axis: o3-mini, GPT-4o

AIE
OpenPipe
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### Fine-Tuning: A Power Tool

The fine-tuned Llama model falls below all three acceptable thresholds; the larger models exceed at least one. Fine-tuning takes more work than prompt engineering, but can improve price and performance at scale.

[16:38](https://www.youtube.com/watch?v=zM9RYqCcioM&t=998s) [Three model comparison charts with red crosses over results exceeding acceptable thresholds.](https://assets.ai.engineer/sites/aiecode2025/sha256/1183a96252fe654bd7f6d684c03f2450a43450913bb1c01574a661ed43d6eb51.webp)

Fine-Tuning: A Power Tool

Error Rate (Lower is Better)
Legend: Max Acceptable
Y-axis ticks: 0.00, 0.02, 0.04, 0.06, 0.08, 0.10
Models: o3-mini, GPT-4o, Llama 3.1 8B (FT)

Average Latency (Lower is Better)
Legend: Max Acceptable
Y-axis ticks: 0, 1, 2, 3, 4, 5
Models: o3-mini, GPT-4o, Llama 3.1 8B (FT)

Cost Per 1K Calls (Lower is Better)
Legend: Max Acceptable
Y-axis ticks: 0, 1, 2, 3, 4, 5, 6
Models: o3-mini, GPT-4o, Llama 3.1 8B (FT)

Stage text: AGENTS AT WORK

### Lessons Learned at Method

Method fine-tuned the cheapest model for its use case using existing GPT production data. The team advises against buying GPUs and recommends openness and patience while agents become production ready.

[17:22](https://www.youtube.com/watch?v=zM9RYqCcioM&t=1042s) [Three lessons: the cheapest model works, avoid buying GPUs, and productionizing AI agents requires openness and patience.](https://assets.ai.engineer/sites/aiecode2025/sha256/7e16b6871157ba4813a89bffa80bbc180d010b3eaec950658586ff223165c945.webp)

Lessons Learned at Method

• The cheapest model just works
• Don’t buy your own GPUs
• Productionizing AI agents requires some level of openness & patience

AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

[17:30](https://www.youtube.com/watch?v=zM9RYqCcioM&t=1050s) [Slide listing three lessons: use the cheapest suitable model, avoid buying GPUs, and bring openness and patience to productionizing AI agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/8e63ba1098d07c03c66e6919e589dac4676ad253f47c7c0d78a241b38d129fbd.webp)

Lessons Learned at Method

• The cheapest model just works
• Don’t buy your own GPUs
• Productionizing AI agents requires some level of openness & patience

AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

## Full transcript

[0:00](https://www.youtube.com/watch?v=zM9RYqCcioM&t=0s) [on-hold music] Hey, everybody.

[0:17](https://www.youtube.com/watch?v=zM9RYqCcioM&t=17s) Uh, yep, I'm Kyle Corbitt from OpenPipe, and I'm here with Mustafa Ali from Method.

[0:22](https://www.youtube.com/watch?v=zM9RYqCcioM&t=22s) Hello.

[0:22](https://www.youtube.com/watch?v=zM9RYqCcioM&t=22s) We're gonna be talking about how Method has scaled in production to over five hundred million agents, uh, and basically all the, the tricks they use to, to make that actually work.

[0:31](https://www.youtube.com/watch?v=zM9RYqCcioM&t=31s) So a little bit about Method is that we essentially collect and centralize liability data from across hundreds of different data sources. This includes tapping into the credit bureaus, uh, connecting with the card networks like Visa and Mastercard, um, and just direct connections with the financial institutions and various other third-party sources.

[0:48](https://www.youtube.com/watch?v=zM9RYqCcioM&t=48s) And, you know, we, uh, sort of aggregate and enhance this data and serve it to our customers, who are typically other fintechs, banks, or lenders, and they use this enhanced data to, um, anything really to do with debt management, so refinancing, loan consolidation, liability payments, or just personal finance management.

[1:07](https://www.youtube.com/watch?v=zM9RYqCcioM&t=67s) Um...

[1:09](https://www.youtube.com/watch?v=zM9RYqCcioM&t=69s) Yeah. And at OpenPipe, what we do is we help you build, uh, train, and deploy open-source models, um, for actual usage. We also let you use in production your signals you get from users, from the environment to improve your model continuously over time, and that's some of the things we'll be talking about, uh, what we did with

[1:26](https://www.youtube.com/watch?v=zM9RYqCcioM&t=86s) Method.

[1:27](https://www.youtube.com/watch?v=zM9RYqCcioM&t=87s) Nice. So one of the early challenges that we faced at Method while coming up with this, you know, aggregation pipeline, uh, was that some of our customers basically came to us and said, "You know, it's really nice that you can give us the balance and payment information on a specific liability for their end consumers.

[1:46](https://www.youtube.com/watch?v=zM9RYqCcioM&t=106s) But you know what would be really nice is if you could also give us some of these liability-specific data points, like the payoff amount on an auto loan or the escrow balance for a mortgage."

[1:56](https://www.youtube.com/watch?v=zM9RYqCcioM&t=116s) And, you know, we said, "Okay, let's do some research." So we go back to some of our data partners and basically ask them, you know, "Is there anything, you know, we can plug into to get these kinds of data points?"

[2:05](https://www.youtube.com/watch?v=zM9RYqCcioM&t=125s) And what we found was there's really no central API that we could get access to that would allow us to get some of these data points. And, of course, ideally, we would want to work with, uh, directly with the banks, but, you know, having already worked with banks before and just from initial conversations, we realized that it

[2:21](https://www.youtube.com/watch?v=zM9RYqCcioM&t=141s) would easily take up to at least a couple of years before getting anything solid done. And, you know, we, we're an early-stage company, so we wanna build for the customer fast.

[2:29](https://www.youtube.com/watch?v=zM9RYqCcioM&t=149s) Um, and so that's really what we're trying to come up with, a solution that we can just, you know, uh, push into production tomorrow.

[2:37](https://www.youtube.com/watch?v=zM9RYqCcioM&t=157s) And so just to get a better understanding of how some of these companies are operating today, uh, the services that they're providing today, how are they doing that in the first place, right?

[2:44](https://www.youtube.com/watch?v=zM9RYqCcioM&t=164s) They, they must be getting that data somehow. So we go back to some of these customers and basically ask them, you know, "How are you guys operating?" And what they tell us is, is kind of interesting.

[2:53](https://www.youtube.com/watch?v=zM9RYqCcioM&t=173s) So a lot of these companies, they basically hire offshore teams of contractors and, you know, they, uh, uh, these teams are basically responsible for calling these banks, um, on behalf of the company and the end consumer.

[3:06](https://www.youtube.com/watch?v=zM9RYqCcioM&t=186s) Uh, they authenticate with the banks, gather the necessary information, somebody has to proof-check it, it gets sent back, um, and then it gets integrated into the financial platforms. Um, and it gets surfaced to the user, is used for underwriting, stuff like that.

[3:19](https://www.youtube.com/watch?v=zM9RYqCcioM&t=199s) And so that's the status quo that we're dealing with here. And when you think about it, that's a very inefficient manual process, right? It's, it's-- When you try to think about scaling, it doesn't really scale.

[3:30](https://www.youtube.com/watch?v=zM9RYqCcioM&t=210s) It's a very, um-- It has a lot of problems. You know, it's expensive because one person can only do one thing at a time, right? So if you want to scale, uh, you basically have to hire more people.

[3:41](https://www.youtube.com/watch?v=zM9RYqCcioM&t=221s) And for the same reason, because it's so synchronous, it's also really slow. Um, and the main-- I guess the, the biggest problem with that is also that it's, there's a lot of human error involved and, um, you, you need to hire a team to fact-check it, uh, to, to proof-check it.

[3:55](https://www.youtube.com/watch?v=zM9RYqCcioM&t=235s) And, um, uh, it's the, the be- the, the worst thing that you're gonna end up with is to surface basically inaccurate financial information. And so conceptually though, if you think about it, it's kind of like an API, right?

[4:06](https://www.youtube.com/watch?v=zM9RYqCcioM&t=246s) You have the request component, you have the authentication component, you have the response, validation, and all that stuff. Uh, so essentially, when you drill this problem down into the core problem, that's really just trying to make sense of, um, unstructured data, right?

[4:21](https://www.youtube.com/watch?v=zM9RYqCcioM&t=261s) So if only there was this magic tool or software that we could use that was really good at parsing unstructured data.

[4:29](https://www.youtube.com/watch?v=zM9RYqCcioM&t=269s) And, you know, lucky for us, around the time that we were trying to solve this problem, OpenAI announced GPT-4. And, you know, as people like to call it, there was this Cambrian explosion of AI or LLM-enabled applications all around us, and the results were just mind-blowing.

[4:45](https://www.youtube.com/watch?v=zM9RYqCcioM&t=285s) Um, and we thought to ourselves, "You know, this, this, this is the perfect thing for us. This is like a godsend." Uh, so we tried to like, you know, we, we tried to see if there's anything there that we could use.

[4:54](https://www.youtube.com/watch?v=zM9RYqCcioM&t=294s) And if there's one thing that we all know in this room is that, uh, advanced LLMs, especially post-GPT-4, are really good with, um, with parsing unstructured data. So tasks like summarization or classification, they're really good with that kind of thing.

[5:08](https://www.youtube.com/watch?v=zM9RYqCcioM&t=308s) So we wanted to test that theory out and see what that can get us.

[5:14](https://www.youtube.com/watch?v=zM9RYqCcioM&t=314s) And so we put our heads down, hacked together this agentic workflow using GPT-4, and as expected, you know, it worked really well. So we tried to like expand some of our use cases because it, you know, the API costs are high, so we wanted to get as much as we could from a single API call and, you

[5:30](https://www.youtube.com/watch?v=zM9RYqCcioM&t=330s) know, it turned out to be really good at that. So we tried to-- Obviously, this was in a very controlled manner, um, but this was in production, and so we were testing out, uh, different, uh, extractions basically, and, um, you know, everything was re- going really good.

[5:45](https://www.youtube.com/watch?v=zM9RYqCcioM&t=345s) Uh, but as soon as we started to increase a little bit of, uh, traffic, uh, what we found was, you know, the bill had to come due. And, um, [lip smack] it was a lot.

[5:55](https://www.youtube.com/watch?v=zM9RYqCcioM&t=355s) So seventy thousand dollars for our first month in production with GPT-4. And, you know, this was-- this made leadership really unhappy and, you know. But, um, but it was something, it was something they were, they were fine with because the value that we were getting out of GPT-4 was so immense.

[6:10](https://www.youtube.com/watch?v=zM9RYqCcioM&t=370s) Um, and so we actually kept this thing in production for at least a couple more months as we tried to work around this kind of cost problem.

[6:18](https://www.youtube.com/watch?v=zM9RYqCcioM&t=378s) And, you know, cost wasn't the only thing that we were concerned with. Um, as we started to scale some of these use cases, we quickly ran into a wall with prompt engineering.

[6:26](https://www.youtube.com/watch?v=zM9RYqCcioM&t=386s) It only takes you so far. Um, one thing we realized that even though GPT is really smart, it's not a financial expert. So you had to give it really detailed instructions and examples, uh, to really make it work with all kinds of use cases that we were trying to target.

[6:40](https://www.youtube.com/watch?v=zM9RYqCcioM&t=400s) Um, so it's hard to generalize those kinds of prompts. They become really long, convoluted. It's always a cat and mouse chase with you fix it for a certain scenario, and it breaks for another one.

[6:49](https://www.youtube.com/watch?v=zM9RYqCcioM&t=409s) You fix it for that one, it breaks for the previous one. And so you're always going back and forth. We didn't have any prompt versioning, so we had to figure out a better way to make this work for all of our use cases.

[7:02](https://www.youtube.com/watch?v=zM9RYqCcioM&t=422s) And so the TLDR here is that, you know, we, we didn't want to adopt that initial solution that I just talked about earlier in the slides because of its scaling challenges and just because it was so inefficient.

[7:12](https://www.youtube.com/watch?v=zM9RYqCcioM&t=432s) But we kind of ran into the same scaling challenges with GPT, where it was expensive because we couldn't really optimize for caching because of the variability in responses and the prompt tweaks we were making all the time.

[7:25](https://www.youtube.com/watch?v=zM9RYqCcioM&t=445s) And the baseline latency that we were finding was actually really slow, so we couldn't... You know, it was overall, we couldn't scale concurrently. And similar to human errors that were kind of, uh, in a different nature, we had AI errors, which were just hallucinations that were hard to catch.

[7:39](https://www.youtube.com/watch?v=zM9RYqCcioM&t=459s) Um, and we just couldn't scale with this kind of system. But we still kept it in production because for specific use cases, it was actually really, really good.

[7:49](https://www.youtube.com/watch?v=zM9RYqCcioM&t=469s) And so now the problem shifted from solving that core problem of trying to make sense of unstructured data. That was solved with GPT. Now, the problem shifted to how do we scale this system?

[7:59](https://www.youtube.com/watch?v=zM9RYqCcioM&t=479s) How do we build a robust, uh, you know, agentic workflow that can handle this kind of volume reliably? And so some of the ballpark figures that we came up with, you know, is that we-- we're gonna be at least making sixteen million requests per day.

[8:12](https://www.youtube.com/watch?v=zM9RYqCcioM&t=492s) Uh, we're gonna have at least one hundred K concurrent load, and, you know, we need minimal latency to, um, handle this kind of real-time agentic workflow, so sub two hundred milliseconds.

[8:22](https://www.youtube.com/watch?v=zM9RYqCcioM&t=502s) And, you know, so the natural next step for us was, like, we thought to ourselves, do we buy more GPUs? Do we host our own model? Like, what do we do at this point?

[8:30](https://www.youtube.com/watch?v=zM9RYqCcioM&t=510s) Um, so at that, at that point, OpenPipe comes in.

[8:34](https://www.youtube.com/watch?v=zM9RYqCcioM&t=514s) Yeah. So about a year ago, we started working with Method on solving these issues that, uh, Mustafa just listed. And we actually found that the, the-- those three issues he listed, right, which are quality, cost, um, and latency, are very common.

[8:47](https://www.youtube.com/watch?v=zM9RYqCcioM&t=527s) Um, these are things that, you know, across almost everyone we work with, uh, at least some subset of those are really top of mind. Um, and so with, uh, Method specifically, we were working on, okay, how do we, how do we solve those problems in a way that makes this, uh, you know, a, a viable business for

[9:02](https://www.youtube.com/watch?v=zM9RYqCcioM&t=542s) you? So, uh, the first thing we did was start measuring error rates. Um, you know, like, like he mentioned, uh, even AI models are not perfect. Uh, these are all probabilistic systems.

[9:12](https://www.youtube.com/watch?v=zM9RYqCcioM&t=552s) Getting to a zero percent error rate was not really feasible, but we were able to see different models had different, uh, had different performance characteristics there. So on modern models on the tasks they're doing, these are the rates we're seeing.

[9:24](https://www.youtube.com/watch?v=zM9RYqCcioM&t=564s) On GPT-4o, um, we're at about an eleven percent error rate, uh, and with o3-mini, it's much better. It's around a four percent error rate. Um, the way you measure that is going to be specific to your business, and that, that's actually true to some extent for all three of these things we'll talk about.

[9:37](https://www.youtube.com/watch?v=zM9RYqCcioM&t=577s) Uh, in the case of Method, this is actually relatively easy to measure, luckily, because they have this agentic workflow, but, like, ultimately, what the agent is trying to do is, is fill out, um, you know, extract all this information he was talking about, bank balances, things like that.

[9:50](https://www.youtube.com/watch?v=zM9RYqCcioM&t=590s) And so you can, you can have a human go through the flow and figure out what the real number should be, and then you can compare an agentic system's final outputs to that and see if it was successful or not, um, which, which made this part relatively easy to calculate.

[10:02](https://www.youtube.com/watch?v=zM9RYqCcioM&t=602s) Uh, so these, these are kinda the error rates we're getting. Um, on the latency point of view, uh, we see that GPT-4o is around a second, uh, to respond.

[10:12](https://www.youtube.com/watch?v=zM9RYqCcioM&t=612s) Uh, and then o3-mini takes about five seconds for their specific task. Again, this is somewhat task dependent, uh, depending on how much, uh, you know, for example, o3 has to think.

[10:20](https://www.youtube.com/watch?v=zM9RYqCcioM&t=620s) Uh, as you're measuring this, you also wanna make sure that you're using real production conditions, that you're actually doing, um, you know, like a, a real diversity of tasks, uh, that, that match what you're actually doing and at a reasonable concurrency level that matches your production.

[10:32](https://www.youtube.com/watch?v=zM9RYqCcioM&t=632s) Um, and we also measured the cost. Um, so again, cost, uh, this is something that is going to obviously be specific, and how much it matters is also very specific to your use case as well.

[10:40](https://www.youtube.com/watch?v=zM9RYqCcioM&t=640s) Um, interestingly, o3-mini, even though it has a much lower per token cost than GPT-4o, uh, if you just look at, like, the pricing page on the API, for their specific use case, uh, we found it was a little bit more expensive because it has-- it generates many more reasoning tokens, so it has much longer outputs.

[10:56](https://www.youtube.com/watch?v=zM9RYqCcioM&t=656s) Um, again, though, uh, this is somewhat task dependent, so I just recommend, um... Actually, just, just as an aside, I would recommend once you get to the point that you're trying to optimize, that you have sort of that initial proof of concept with, with some model, something that works, I think it's really worthwhile to-- It can be

[11:11](https://www.youtube.com/watch?v=zM9RYqCcioM&t=671s) as simple as, like, literally just writing, like, you know, three different py-Python scripts that, like, are able to categorize each of these for a different model. Um, and then as new models come out, you'll be able to quickly tell how they're doing.

[11:22](https://www.youtube.com/watch?v=zM9RYqCcioM&t=682s) Um, okay. Once you've done, or in this case, once we've done this, this sort of, um, benchmarking of where the models are, the next question is, all right, what is-- w-where do we need these models to be?

[11:32](https://www.youtube.com/watch?v=zM9RYqCcioM&t=692s) Where do we need to get to? Um, and so again, this is very task dependent. Uh, in the case of Method, uh, they do have spec-- like, they have, um, extra checks that happen after this, where they look and see, okay, are the numbers that came out plausible?

[11:45](https://www.youtube.com/watch?v=zM9RYqCcioM&t=705s) Do they match, you know, the types of things we've seen before? All the, all these different kinds of checks they're doing. And so they didn't need to get all the way down to a zero percent error rate, but of course, those checks are still fallible.

[11:53](https://www.youtube.com/watch?v=zM9RYqCcioM&t=713s) And so, um, if it's over a certain point, then, uh, then some fraction of those errors are gonna get through, and that's gonna be bad. And so we found around a nine percent error rate was, was able to get them what they needed.

[12:03](https://www.youtube.com/watch?v=zM9RYqCcioM&t=723s) Um, from a latency point of view, so the way their agent works is a real-time system. Uh, it, it needs to be able to respond quickly to, to move, uh, through the, the, basically, like, through the whole flow to get the information it needs.

[12:16](https://www.youtube.com/watch?v=zM9RYqCcioM&t=736s) And so they did have a hard latency cutoff. Um, we see a wide variety in this for what it's worth. We have some customers that I talk to who it's like, "Hey, if I get a result back at some point in the next few days, like, that's totally fine.

[12:25](https://www.youtube.com/watch?v=zM9RYqCcioM&t=745s) This is a background batch process." Um, we have other customers who are doing real-time voice with a human on the end-- other end of the line, and it's like, "Hey, you know, if I'm over five hundred milliseconds, that's not gonna work for me."

[12:35](https://www.youtube.com/watch?v=zM9RYqCcioM&t=755s) And so again, you just have to know for your specific case how much this matters. Same with cost. Um, in their case, because of that very high volume, as Mustafa was mentioning, cost is pretty important to them.

[12:45](https://www.youtube.com/watch?v=zM9RYqCcioM&t=765s) Um, again, depending on your use case, usually mostly dependent on how high volume it is, um, will determine how much cost matters to you. But, but it's something you, you should know these numbers for your specific task as you're comparing different models.

[12:57](https://www.youtube.com/watch?v=zM9RYqCcioM&t=777s) Okay. So, um, we're looking here at this, uh, of course, as you're looking at this, this slide, you can-- y-you, you may see there's a problem here, which is, um, of the two models we're comparing, at least, none of them actually meet all three of the requirements we need to be able to deploy this in production.

[13:13](https://www.youtube.com/watch?v=zM9RYqCcioM&t=793s) And, uh, you know, GPT-4o on both the error rate as well as the cost, we're not quite there. Um, and then o3-mini, uh, on the cost, but especially on the latency, it's just not going to work for what we need.

[13:24](https://www.youtube.com/watch?v=zM9RYqCcioM&t=804s) So this is the point at which, uh, Method came, and they talked to us. We're like, "Hey, we're not able to hit what we need here." Um, because again, we're not, uh, yeah, we're-- th-th-these, these models aren't getting us where we need to be.

[13:37](https://www.youtube.com/watch?v=zM9RYqCcioM&t=817s) So what we work on at OpenPipe is fine-tuning. We work on building custom models for your specific use case. And so I'm gonna talk about why you would wanna do that and how that helps in this case.

[13:46](https://www.youtube.com/watch?v=zM9RYqCcioM&t=826s) Um, first, I would say fine-tuning is a power tool. Uh, it does take more time. It takes more, uh, engineering investment than just prompting a model. Uh, so you don't really want to do that until you have actually benchmarked the production models, just prompting them and seeing whether they work or not.

[14:04](https://www.youtube.com/watch?v=zM9RYqCcioM&t=844s) Um, so in this case, in Method's case, and in m-- all of our customers' cases, uh, they, they found that they were not able to hit the numbers they needed.

[14:10](https://www.youtube.com/watch?v=zM9RYqCcioM&t=850s) Um, and so that's the time you wanna bring in fine-tuning.

[14:13](https://www.youtube.com/watch?v=zM9RYqCcioM&t=853s) Um, so let's look at-- We were able to fine-tune a model and see, uh, how that was able to help, uh, because it can actually really bend that price performance curve a lot.

[14:22](https://www.youtube.com/watch?v=zM9RYqCcioM&t=862s) Um, so on the, the error rate, uh, which is basically just the inverse of, of accuracy, if you wanna measure it that way, um, we were able to get to a place where we were doing significantly better than GPT-4o, and importantly better than that threshold they needed.

[14:34](https://www.youtube.com/watch?v=zM9RYqCcioM&t=874s) Uh, this used to actually be much harder to achieve. It required a lot of manual, uh, labeling of data and things like that. It's actually become much easier over time because of the existence of models like now o3-mini, um, which allows you to just use your production data.

[14:48](https://www.youtube.com/watch?v=zM9RYqCcioM&t=888s) You can, you can use your, uh, basically the inputs you use in production. You can f-- uh, generate outputs for them using a model like o3-mini and train on them.

[14:57](https://www.youtube.com/watch?v=zM9RYqCcioM&t=897s) We find, like in this case, that often you're not able to quite get, uh, to the, the performance of the, the teacher model, the model o3-mini in this case that you're using.

[15:05](https://www.youtube.com/watch?v=zM9RYqCcioM&t=905s) But you can get quite close to it and usually do much better than, you know, uh, a slightly less good but much, much larger model. Um, you know, in this case, uh, the model we ended up deploying with them is just an eight billion parameter Llama 3.1 model.

[15:17](https://www.youtube.com/watch?v=zM9RYqCcioM&t=917s) And, and we find that actually for the majority of our customers, a model that large or smaller is, is good enough and is able to hit the numbers you need from quality.

[15:25](https://www.youtube.com/watch?v=zM9RYqCcioM&t=925s) Um, but, uh, yeah, the, the important thing is to be able to benchmark that and to answer that question for yourself.

[15:30](https://www.youtube.com/watch?v=zM9RYqCcioM&t=930s) Um, on the latency point of view, because actually, this is, this is sort of the magic of being able to move to that much smaller model. Because we've got this eight billion parameter model, it is way easier to deploy in a low latency way.

[15:42](https://www.youtube.com/watch?v=zM9RYqCcioM&t=942s) Um, there's just many-- fewer, fewer calculations, fewer sequential calculations with the number of layers, and so you can get just a much lower latency. You can even-- and we, we didn't actually have to do this in Method's case, um, but something you can do is you can train this model.

[15:53](https://www.youtube.com/watch?v=zM9RYqCcioM&t=953s) You can deploy it within your own infrastructure, co-locate it with the application code that's using it, um, and even completely eliminate the, the network latency.

[16:01](https://www.youtube.com/watch?v=zM9RYqCcioM&t=961s) Uh, and then finally, uh, on the cost front, again, just because this is such a smaller model, um, you end up with a much, much lower cost. Uh, and so that, for many of our customers, is a big-- is, is incredibly important, is to be able to get that performance number you need, um, while still maintaining a

[16:17](https://www.youtube.com/watch?v=zM9RYqCcioM&t=977s) relatively low cost. Um, in, in Method's case, we were actually able to far exceed the sort of cost thresholds that they were looking for to make this viable, um, which means that they don't have to worry about this from sort of a unit economics point of view, uh, in, in, in the way that they did when they

[16:30](https://www.youtube.com/watch?v=zM9RYqCcioM&t=990s) were using the larger models. Um, so just, um, to sort of reiterate what I started with before, um, this is a power tool. Uh, the fine-tuning, uh, is-- it does take a fair amount of work, um, not an extreme amount of work, but significantly more work than you do for prompt engineering.

[16:48](https://www.youtube.com/watch?v=zM9RYqCcioM&t=1008s) However, if you're not able to get to the reliability numbers you need, uh, through just prompt engineering with the models that exist out there without tuning, it is a viable way to very strongly bend that price performance curve and get to a much better place, uh, which, uh, which, which can help you get to a very large

[17:05](https://www.youtube.com/watch?v=zM9RYqCcioM&t=1025s) scale in production, just like Method did.

[17:10](https://www.youtube.com/watch?v=zM9RYqCcioM&t=1030s) Nice. Um, so yeah, just to wrap up here, uh, one thing that, or at least a couple, couple points that we wanna highlight is that, you know, the reason we put two engineers in the title is also because it's not that, it's not that complicated, right?

[17:24](https://www.youtube.com/watch?v=zM9RYqCcioM&t=1044s) You can get away with using-- We identified a specific use case, and we got away with just using the cheapest model that was out there. Uh, we fine-tuned it.

[17:31](https://www.youtube.com/watch?v=zM9RYqCcioM&t=1051s) We already had the data from GPT in production, so we already had the data. We didn't have to go digging around for the data in the first place. Uh, so we already used that, and we used the cheapest model that gave us the fastest performance.

[17:42](https://www.youtube.com/watch?v=zM9RYqCcioM&t=1062s) And, you know, you don't need to buy your own GPUs. Um, and the, the other thing that we realized is that productionizing AI agents actually requires a little bit of, uh, some level of openness, uh, and patience from the engineering team, from the leadership team.

[17:55](https://www.youtube.com/watch?v=zM9RYqCcioM&t=1075s) It's because when you write code, we're always writing code that just works. You push out a feature and never breaks because you're not changing anything. But with AI agents, you-- it takes some time to get to a point where it's, like, production ready and actually gives you the responses that you're looking for.

[18:09](https://www.youtube.com/watch?v=zM9RYqCcioM&t=1089s) Um, and, you know, I, I feel compelled to say something about as-- to mark the top of the traditional software engineering job. So I'll leave you with these last few words.

[18:18](https://www.youtube.com/watch?v=zM9RYqCcioM&t=1098s) If you're in SWE, pivot to AIE. [laughs] [upbeat music] Thank you.

[18:23](https://www.youtube.com/watch?v=zM9RYqCcioM&t=1103s) Thanks, everyone. [clapping] [upbeat music]
