AI Engineer World's Fair 2025
Analyzing 10,000 Sales Calls with AI in 2 Weeks
Read the talk
Analyzing 10,000 Sales Calls in Two Weeks
Charlie Guo’s customer research project at Pulley shows how model selection, traceable extraction, prompt caching, and a reusable interface turn a transcript archive into a working research tool.
From a talk by Charlie Guo
The two-week analysis challenge
How many sales calls can you listen to and take notes on in a day? At 30 minutes per call, an eight-hour day without breaks fits 16 calls. Spend every waking hour listening, with only eight hours reserved for sleep, and that becomes 32 calls a day, or 224 per seven-day week.
Charlie Guo faced a different scale: Pulley’s CEO wanted 10,000 sales calls analyzed in two weeks to refine the company’s ideal customer profile, or ICP. Guo describes a project that a single AI engineer could complete in a fortnight—work that previously would have required a dedicated team over several weeks. The challenge was to turn conversations into trustworthy customer research quickly enough to inform a business decision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a broad ICP to specific customer evidence
Pulley’s existing ICP was venture-backed startups. That was a useful guide for product development, but too broad for marketing. A persona such as CTO of an early-stage, venture-backed crypto startup gives a marketer more to work with than founder. Understanding that specificity requires listening to customers; Pulley already had thousands of hours of sales representatives doing exactly that.
A manual research process would repeat the same sequence for each call:
- Download the transcript and read the conversation.
- Decide whether the conversation matches the target persona.
- Scan hundreds or thousands of lines for useful insights.
- Compile notes and reports while retaining citations for later verification.
At 16 calls per day, 10,000 calls require 625 eight-hour workdays—nearly two years only if someone works every day. Even finishing the reading would leave the synthesis problem: remembering enough of an entire library to write one book report.
The familiar approaches each sacrificed something essential:
| Approach | Strength | Limitation |
|---|---|---|
| Manual analysis | High-quality interpretation | Cannot scale to this workload |
| Keyword analysis | Fast and cheap | Misses context and nuance |
Large language models offered a way to recognize patterns in unstructured conversations without reducing every question to a keyword match. But producing useful research still required solving several connected problems: choosing a model, controlling unsupported claims, preserving evidence, and paying for the resulting workload.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the model against consequential errors
At the time of the project, GPT-4o and Claude 3.5 Sonnet were the most capable models available to Guo’s team, but also the slowest and most expensive. Smaller, cheaper models were attractive until the team examined their classifications.
Two failures exposed the difference between recognizing words and establishing facts:
- Company classification: A sales representative mentioned the platform’s blockchain features, and a smaller model classified the prospect as a crypto company. The feature belonged to the sales pitch; it did not establish the customer’s business.
- Person classification: A model labeled a prospect as a founder despite no supporting evidence in the transcript.
These false positives threatened the purpose of the project. A polished report built from incorrect customer identities would produce a bad analysis. The team chose Claude 3.5 Sonnet because its hallucination rate was acceptable for their needs, although Guo gives no numerical error rate. The cost of an unsupported classification mattered more than the price of an individual request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make extracted findings traceable
Model selection was only the first layer of reliability. Guo’s team started with raw transcripts, enriched them through retrieval-augmented generation using third-party and internal sources, and used prompting techniques, including chain-of-thought prompting, to improve the results. The model therefore had additional context to help interpret the conversation rather than relying on the transcript alone.
The next layer was structured JSON output, used where possible to generate citations. A useful output contract keeps a finding and its evidence together. For example, the earlier blockchain failure can be represented with an explicitly unknown customer classification rather than an unsupported positive label:
json
{
"transcript_id": "call-001",
"company_sector": null,
"evidence": [
{
"line_id": "line-042",
"speaker_role": "sales_rep",
"text": "Our platform includes blockchain features."
}
],
"finding": "The sales representative describes a platform feature; the prospect's sector is not established."
}
This illustrative record makes the evidence boundary concrete: the quoted line supports a statement about the sales pitch, while company_sector remains unknown. The project used application-generated citations from JSON; it should not be read as a demonstration of Anthropic’s later native citations API, whose launch is recorded in the Claude Platform release notes.
The resulting system extracted company details and customer insights with a verifiable trail back to the original transcripts. That trail made it possible to inspect the basis of a finding instead of accepting the model’s answer on authority. But richer analysis and the effort to keep errors low increased the cost substantially.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reduce repeated input work and split outputs
The workflow often reached the 4,000-token output limit Guo encountered with Claude 3.5 Sonnet. Completing one transcript’s analysis could therefore require several requests. Two features that were experimental at the time addressed different sources of repeated work.
Prompt caching addressed repeated input. The team reused the same transcript to extract metadata and then insights, so caching let later requests reuse that transcript content. Guo cites reductions of up to 90% in cost and up to 85% in latency for prompt caching. Those figures match the ceilings advertised in Anthropic’s prompt-caching announcement, rather than a supplied measurement of this project’s overall performance. Cache writes carry a premium, and the announcement’s latency example measures time to first token; the benefit depends on reusing the long prompt.
Extended outputs addressed incomplete responses. An experimental feature flag doubled the available output capacity, allowing complete summaries in a single pass instead of splitting the analysis across multiple turns. This expanded the amount the model could generate, not its input context window. Historically, Anthropic introduced the 8,192-token output beta in July 2024 and removed the beta-header requirement in August; the flag described here belongs to that earlier API period.
Guo reports that the optimizations reduced the analysis cost from $5,000 to $500 and delivered results in days instead of weeks. That is his project-level account, without a billing or runtime breakdown. The engineering mechanisms are distinct: caching reduces repeated processing of the transcript, while longer outputs reduce the need for additional requests to finish a summary.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The research infrastructure reaches other teams
The larger surprise was how widely the work became useful. The initial customer was the executive team, which wanted insights that Guo says would normally require tens, if not hundreds, of hours to produce. Once the underlying transcript workflow existed, other teams could use it for their own work.
- Marketing: Retrieved customer quotes for branding and positioning exercises.
- Sales: Automated transcript downloads using the system. Guo reports that automating transcript downloads saved the sales team dozens of hours every week.
The value extended beyond the generated research: even the surrounding data-access machinery removed recurring work.
Teams also began asking questions they previously would not have considered because manual analysis was too daunting. Making the corpus accessible changed which research questions were practical to pursue, turning an accumulation of unstructured conversations into a shared source of evidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the system around the model
Model capability mattered: in Guo’s experiments, Claude 3.5 Sonnet and GPT-4o handled tasks the other tested models could not. But the final choice combined accuracy with prompt caching. Model fit depended on both the errors the project could tolerate and the economics of repeatedly analyzing long transcripts.
The surrounding software mattered just as much. Structured JSON, good database schemas, and the overall architecture produced significant gains. AI engineering means building an effective system around the model: its outputs need to fit the application’s data structures and workflows, and its evidence needs to remain available to the people using the results. Guo’s lesson is to integrate the model thoughtfully into the system rather than add it as an afterthought.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make customer research a repeatable operation
The project did not end with a report or white paper. The team built an interface around the analysis with search, filters, and exports. Those features let colleagues find relevant material and carry it into their own work, making a simple, flexible tool useful across the company.
The purpose was to augment human analysis by removing the bottlenecks that prevented it. Once downloading, reading, and organizing the corpus no longer consumed the entire research budget, people could investigate questions that had previously been out of reach. Faster processing enabled a broader range of work.
That leaves a practical question for any organization: what customer data is already available but rarely examined? Sales calls, support tickets, product reviews, user feedback, and social media interactions all contain potential evidence. Guo’s closing challenge is to use the available tools and techniques to make those neglected sources accessible—and begin turning the accumulated data into something people can use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
Charlie Guo's earlier account of the sales-call analysis project, including model selection, transcript caching, citations and the interface built for colleagues.
- Prompt caching with ClaudeArticle
Anthropic explains prompt reuse, cache pricing and example cost and time-to-first-token improvements for long inputs.
- Claude Platform release notesDocumentation
Dated API history covering Claude 3.5 Sonnet's longer outputs, prompt caching and subsequent citation support.
Read the complete timestamped transcript
- 0:04
I want to start with a question. How many sales calls can you listen to and take notes on in a single day? If you assume each call is 30 minutes and you work an eight-hour day with no lunch break, that gives you 16 calls.
- 0:18
If you have absolutely zero work-life balance and only stop to sleep for eight hours a day, you might get to 32 calls. And if you do that every single day, you'll get to 224 per week, doing nothing but listening to sales calls over and over again.
- 0:35
Last year, our CEO wanted to analyze 10,000 sales calls in order to do a wide-ranging analysis of our ideal customer profile, and I had two weeks to do it.
- 0:47
Two years ago, this would have been impossible or would have at least required a dedicated team working for several weeks. Today, it's something a single AI engineer can accomplish in about a fortnight, and I want to show you exactly how I did it.
- 1:01
At Pulley, our ideal customer profile, or ICP, was venture-backed startups. It's a useful North Star from a product perspective, but much less helpful when it comes to marketing. We knew we needed to get more specific.
- 1:16
Think CTO of an early-stage venture-backed crypto startup instead of merely founder. And as every founder knows, when you're trying to understand your customers, there's really no substitute for talking to them.
- 1:29
In my case, I had the next best thing: thousands of hours of our sales reps talking to customers directly. The only problem was it was too much data. I want to paint you a picture of what a manual analysis of this sales call database would look like.
- 1:47
It would require downloading each transcript, reading the conversation, deciding if that conversation matches the target persona you wanted to analyze, scanning hundreds or thousands of lines of conversation for key insights, trying to remember everything while writing reports, uh, and compiling notes, as well as citations for future reference in a research report.
- 2:10
And then, of course, doing this 10,000 times in a row. Even if you wanted to do this manually, we're talking about 625 days of continuous work, nearly two years.
- 2:23
The human brain simply isn't wired to process that much information. It's like trying to read an entire library and then write a single book report about it. Before LLMs, traditional approaches to doing this kind of analysis generally fell into two categories:
- 2:39
a manual analysis that was high quality but completely unscalable, and a keyword analysis that was fast and cheap but would often miss context and nuance. This is where modern large language models come in.
- 2:54
This intersection of unstructured data and pattern recognition, it's a sweet spot for AI projects. But here's what I learned. What looks simple in hindsight, just use AI to analyze the sales calls, actually required solving several interconnected technical challenges.
- 3:12
Our first major decision was choosing the right model. At the time, GPT-4o and Claude 3.5 Sonnet were the most intelligent options that we had access to, but they were also the most expensive and the slowest.
- 3:26
The temptation to use smaller and cheaper models was pretty strong, but our experiments showed their limitations rather quickly. They produced an alarming number of false positives. They would classify transcripts as being related to crypto companies because a sales rep mentioned blockchain features on the platform, or they might decide that a prospect was actually the founder of a
- 3:49
company despite zero supporting evidence in the transcript. In our case, the worst outcome was a bad analysis. If we couldn't trust the model's data, the entire project would be pointless.
- 4:02
So we made the choice to shell out for the more expensive models since they produced a hallucination rate that was acceptable for our needs. In the end, we ended up going with Claude 3.5 Sonnet.
- 4:13
I mention the hallucination rate here because it also wasn't as straightforward as feeding the transcript in and asking for answers out. We ended up developing a multi-layered approach to reducing hallucinations in the system.
- 4:26
We started with the raw transcript data, which we then enriched via retrieval-augmented generation, both from third-party sources as well as other internal sources that we had access to. We employed some prompt engineering techniques like Chain-of-thought prompting in order to get the model to produce more reliable results.
- 4:45
And finally, we made sure to get structured JSON outputs where possible in order to generate citations. This, taken together, created a system that could reliably extract both accurate company details and meaningful insights.
- 5:00
And more importantly, it had a verifiable trail back to the original transcripts where it was pulling information from, meaning we could be confident about the final results. The main problem was doing all of this analysis and making sure we had low error rates drove up our costs pretty significantly.
- 5:18
We often hit the 4,000 token output limit for Claude 3.5 Sonnet, which would require multiple requests per transcript analysis. Luckily, we were able to leverage two features that were experimental at the time to dramatically lower our costs.
- 5:33
The first was prompt caching. A lot of the analysis that we did involved reusing the same transcript repeatedly, both to extract metadata and to extract insights. By caching the transcript content, we were able to reduce costs by up to 90% and latency by up to 85%.
- 5:51
The other feature we leveraged was extended outputs. At the time, Claude had an experimental feature flag that could give access to double the original output context.
- 6:03
Using this flag let us generate complete summaries in single passes rather than having to break the analysis into multiple turns and burning multiple rounds of credits. As a result, we turned a $5,000 analysis into a $500 one, and we got results in days instead of weeks.
- 6:22
But what ultimately surprised me the most about this project wasn't the take-- technical capabilities that we discovered or the challenges that we overcame. It was the wide-ranging impact of our analysis.
- 6:35
What started off as a project for the executive team to produce insights that would normally take tens, if not hundreds of hours, ended up being more useful across the organization.
- 6:46
The marketing team, for example, were able to pull customer quotes for branding and positioning exercises that they needed, and the sales team was able to automate transcript downloads on the back of this system, saving them dozens of hours every week.
- 7:02
On top of all this, teams started asking questions that wouldn't have been considered before because doing manual analysis like this would have been too daunting. In the end, we transformed our mountains of unstructured data from a liability into an asset.
- 7:18
So what did we learn from all this? I want to share three key takeaways. First, models matter. Despite the push for open source and smaller, cheaper models, Claude 3.5 and GPT-4o could handle tasks that other models simply couldn't.
- 7:35
We chose Claude for its prompt caching capabilities and its accuracy, but in the end, the right tool isn't always the most powerful one. It's the one that best fits your specific needs.
- 7:46
Two, I want to point out that good engineering still matters. Despite AI's capabilities, we gain significant wins from good old-fashioned software engineering, leveraging JSON structured output, good database schemas, and proper architecture for this system as a whole.
- 8:04
AI engineering is knowing how to build effective systems around large language models. That means that the AI can't just be bolted on or used as an afterthought. It needs to be thoughtfully integrated into existing systems and architectures.
- 8:19
And we learned that we needed to consider additional use cases for our tech. We didn't just stop at a single report or white paper. We built out an entire UX around our AI analysis with features like search, filters, and exports.
- 8:34
And by building a simple yet flexible tool, what could have been a one-off project actually became a company-wide resource. At the end of the day, this project showed how AI can transform seemingly impossible tasks into routine operations.
- 8:49
It's not about replacing human analysis. It's about augmenting it and removing human bottlenecks. To me, that's the real promise of tools like Claude, ChatGPT, Gemini, and all the rest.
- 9:01
Not just doing things faster, but unlocking entirely new possibilities.
- 9:07
So here's my challenge to you. What customer data are you sitting on right now? Sales calls, support tickets, product reviews, user feedback, and social media interactions. These are all valuable sources of insight that go untouched in most companies but are now very much accessible via large language models.
- 9:29
The tools exist today. The techniques work. The only question is, when will you stop ignoring your data and start turning it into gold?
- 9:39
If you like this, you can read much more about it and other projects like it at my blog, Artificial Ignorance. Thanks for watching.