AI Engineer World's Fair 2024
Giving a Voice to AI Agents
Read the talk
Giving Voice Agents the Context Behind the Words
A fast speech-to-text, language-model and text-to-speech loop can still miss what someone means. Scott Stephenson traces how context could connect those stages without sacrificing control.
From a talk by Scott Stephenson
Before you start: Basic familiarity with speech recognition, language models and speech synthesis is helpful; no Deepgram API experience is required.
Hearing through a noisy drive-through
How does a voice agent understand an order while cars idle and an Amazon truck reverses nearby? That is the acoustic environment Scott Stephenson brings into the discussion. Deepgram, which he describes as nine years into building end-to-end deep learning for speech recognition, had also introduced text-to-speech the preceding December. That early release belongs to the product’s preview period, before general availability. Its audio services could run in the cloud or on premises, with models automatically adapted to environments such as the drive-through he had just visited.
Deepgram’s research team investigates fundamental capabilities, and its product team turns those findings into services. Stephenson says many devices already use Deepgram behind the scenes. But deployment alone is not the quality target: the benchmark guiding the company’s conversational systems is what a human can do.
That benchmark becomes a way to formulate research problems. If neural networks learn from examples, what examples and learning task would let a machine acquire the required ability? Stephenson describes internal discussions comparing neural capacity with mice, cats and people—not as a deployment specification, but as part of thinking about what learning systems might become capable of after a decade of progress.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From narrow assistants to a repeating conversation
Earlier voice assistants such as Siri accepted broad questions but answered reliably only within particular domains. Stephenson characterizes this generation as slow, imperfectly accurate and lacking a real-time conversational feel. LLMs expand the possible exchange: they accept text and generate open-ended responses, within the capabilities established by their training.
The resulting conversation follows a repeating sequence:
- Speech-to-text turns incoming audio into words.
- Text-to-text transforms those words into a response.
- Text-to-speech turns the response into outgoing audio.
- The next spoken turn starts the loop again.
The engineering targets are speed, appropriate agent behavior and expressive speech. Calling the middle stage text-to-text keeps its function separate from its implementation: a transformer is one way to perform that transformation, not the definition of the stage itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The first cars in the street
Stephenson places this architecture near the beginning of adoption, comparing it with seeing the first cars in the street around 1910. His forecast is a 100× expansion in voice, text and related AI over the following decade. The analogy answers the suggestion that the industry has already reached its peak of hype: visible early products may represent only a small part of the eventual market.
His broader analogy compresses successive technological revolutions: agriculture over roughly 1,000–2,000 years, industry over 250–300 years and information technology over 75 years. He projects an intelligence revolution lasting 25–30 years and argues that AI companies consequently need to move three times faster than earlier technology companies. These are his historical framing and forecast, rather than measured adoption curves.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fast components can still miss the meaning
The three stages work largely independently, and each has improved. Stephenson describes speech recognition accuracy rising from roughly 75% five to eight years earlier to over 90% at the time of the talk. He also describes real-time speech recognition delay falling from 2–5 seconds to 100–200 milliseconds. No dataset or accuracy definition accompanies these comparisons, and speech recognition delay is a component measure, not the full conversational response time. Running services on premises and colocating them can also shorten the path between stages.
Stephenson cites Daily’s The World’s Fastest Voice Bot as an example using Deepgram speech recognition, Llama for text-to-text and Deepgram speech synthesis. He describes its round trip as below 500 milliseconds; the post more precisely reports a 500-millisecond best case and an achieved 800-millisecond median target, using colocated models, WebRTC and tuned endpointing. Stephenson compares this with a human turn-taking interval of 400–600 milliseconds. The distinction between best-case and median response matters when using the example as a latency target.
Fast text exchange does not preserve all conversational context. In the architecture Stephenson describes, the stages pass words along without carrying the surrounding conversational state. The system can therefore respond promptly while missing what the person means. He illustrates the problem with an estimate of misunderstandings in 10–20% of turns, explicitly refining his initial wording from interactions to turns. That estimate motivates the next design change; it is not presented as an evaluated failure rate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give speech recognition a conversation to work with
The proposed change begins at the recognizer’s input. Instead of accepting only audio, speech-to-text would accept audio plus context. Prompting need not mean a text instruction: the context might include other audio, images, documents or a previous conversational turn.
Stephenson compares conventional recognition to being thrown into a basketball game without knowing anything about the conversation and having to transcribe immediately. A recognizer may have only a few seconds from which to infer the setting. Giving it the conversation so far supplies evidence it otherwise has to reconstruct, which he argues makes transcription more accurate.
Context would also become an output. The recognizer could pass through the original context and add new information expressed as human-readable text, audio, images or vector embeddings. Those outputs would carry conversational state into later stages. Stephenson explicitly presents this as a proposed architecture, not the standard system design at the time of the talk; his expectation that it would become common within the following year is a forecast.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry intent into speech—and delivery into the next turn
Once the text-to-text stage receives context derived from audio, it can respond to more than a transcript. It may know whether someone sounds angry or happy, whether the conversation is moving quickly or slowly, and whether light music is playing in the background. That information helps it choose an appropriate response.
The response can then include delivery instructions for text-to-speech: speak softly, speak slowly, or use an authoritative tone. This separates what to say from how to say it, while keeping both in the same conversational state.
The proposed feedback loop continues after synthesis. Text-to-speech would report how it tried to deliver the response, how it sounded and its assessment of the result. That feedback would become context for the next turn. The following TypeScript expresses a small application-level record for this design; the response text and feedback values are a teaching example. A requested delivery and a report of actual delivery remain separate fields.
typescript
type Delivery = {
volume: "soft" | "normal";
pace: "slow" | "normal";
};
type Turn = {
responseText: string;
requestedDelivery: Delivery;
deliveryReport: {
observedDelivery: Delivery;
assessment: string;
} | null;
};
function recordDelivery(
turn: Turn,
report: NonNullable<Turn["deliveryReport"]>
): Turn {
return { ...turn, deliveryReport: report };
}
const pendingTurn: Turn = {
responseText: "Let’s take this one step at a time.",
requestedDelivery: { volume: "soft", pace: "slow" },
deliveryReport: null,
};
const reportedTurn = recordDelivery(pendingTurn, {
observedDelivery: { volume: "soft", pace: "slow" },
assessment: "Delivery matched the requested volume and pace.",
});
The assignment records supplied feedback; it does not synthesize or evaluate audio. In a working version of the proposed architecture, the synthesis stage would supply that report. Deepgram’s internal name for this exchange is contextual AI.
Stephenson treats speed and recognition accuracy as sufficiently advanced to make context the next major source of humanlike interaction. The change is not just a richer prompt for one model: each stage receives state, contributes to it and passes it onward.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep control where the business needs it
Multimodal or speech-to-speech models could combine some or all of these stages. Stephenson accepts that possibility but identifies reduced controllability as the tradeoff. He describes Deepgram as the largest speech-to-text API, attributing its scale to business applications including Spotify, food ordering and call centers. Those users need to constrain what their systems do.
A bank makes the requirement concrete. An open-ended instruction to a speech-to-speech model gives the bank less control than separately configuring recognition, response generation and vocal delivery. The workload may justify very different investments in each stage:
| Stage | Bank’s requirement | Allocation |
|---|---|---|
| Speech-to-text | Recognize the customer precisely | Substantial compute |
| Text-to-text | Handle a few tasks, such as password resets | Smaller model |
| Text-to-speech | Sound expressive but calm | One controlled voice |
The middle stage need not be especially large if its job is narrow, even when recognition quality and vocal delivery are demanding.
This is also a cost of goods sold, or COGS, decision. Separating the stages lets a business size each component for its own task and choose services that focus on operating cost. The expensive part of one application need not dictate the size of every other part of its voice stack.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An integrated agent with component choices
Deepgram’s platform pitch leaves room to adopt individual components. An application could use its low-latency speech recognition or speech synthesis without taking the entire stack. Stephenson describes TTS as a newer product and promises greater expressiveness in its next version. His subjective assessment places its then-current quality above Amazon’s and Microsoft’s neural voices, but below ElevenLabs.
The forthcoming Voice AI Agent API preview would bring the full stack together while allowing developers to use their own API keys and LLMs. Stephenson gives integrated-stack turn-taking examples of 300, 500 or 600 milliseconds, compared with 800 or 1,500 milliseconds when assembling components independently. He presents these as examples of the integration benefit, without a controlled measurement protocol.
The closing invitation is to apply for preview access using the displayed QR code or link, or speak with the staff at the back. Stephenson also points attendees to Damien, who had led a workshop roughly two days earlier.
He closes by offering $250 in Deepgram credit to try the platform. That is the offer made in the recording, rather than a statement of current signup terms.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Daily explains its 2024 voice bot architecture, including model colocation, WebRTC transport, endpointing, and latency optimization.
Further reading
- Introducing Deepgram AuraArticle
The original Aura general-availability announcement explains its role in Deepgram's speech recognition, language processing, and speech synthesis platform.
Updates since the talk
Configure managed LLM providers or custom model endpoints in the current Deepgram Voice Agent API.
Read the complete timestamped transcript
- 0:00
[on-hold music] Uh, hey, everybody. If you don't know about Deepgram, Deepgram is a company that does, uh, audio AI.
- 0:18
So we've been around for nine years now, um, so ancient in AI years. Um, but, uh, s- we, we brought, uh, end-to-end deep learning to, uh, speech recognition, and, uh, our most recent product is a TTS product that we released back in, uh, December.
- 0:37
And so you can use our API, uh, on-prem, you can use it in the cloud. You can, um, adapt models to a specific acoustic environment. I just came from a drive-through that has all sorts of crazy things happening in it, like cars running and Amazon trucks bre- backing up and all of that, and we, uh, uh, like
- 0:56
automatically adapt models for that kind of thing, et cetera. Um, so anyway, if, if, if you don't know about us, that's what we do. Uh, also, we're a research-led company.
- 1:05
W-- Uh, so what that means is we figure out fundamental things through our research team, and then we bring them to, uh, the market as soon as possible through our product team.
- 1:14
So, uh, yeah, it's, it's, uh, everything is moving at breakneck speed, but a lot of the things that, uh, folks enjoy now with like really fast, accurate, real-time speech recognition, if you have any devices, um, out there, I was just talking with a founder a couple minutes ago, uh, uh, uh, most of them use Deepgram behind the
- 1:34
scenes. So, uh, just, uh, that's, that's who we are, and, uh, we help companies build truly conversational experiences. And the bar that we set for ourselves and we set from the beginning, um, but it takes a long time to get there, is, uh, what can a human do?
- 1:50
And we ask this all the time in our product meetings, et cetera, and our research team. Um, and that helps, uh, guide us toward, uh, what's possible, um, because neural networks really do work in a way that is similar to a human.
- 2:03
They can learn by example, et cetera. And so you have to think about how can we formulate, uh, this problem in a way that, uh, this, this machine can actually learn from it.
- 2:12
And so we spend all of our, all of our time thinking that way. And, uh, so arguments about like how many neurons does it have? Is it equivalent to a mouse or a cat or a person or whatever?
- 2:20
We're talking about that kind of thing all the time. Um, and it's amazing to see, um, the progress that has happened over the last decade, and we're at a, uh, an amazing time right now.
- 2:29
So, um, one thing to keep in mind is that there was like a previous version of voice AI, and, uh, it was kinda, you know, voice AI 1.0. Think Siri, think things like that, where it was slow, it wasn't super accurate, um, there wasn't really a real-time feel to it.
- 2:46
Um, and it, it-- you could try to ask it anything, but it wouldn't really, um... It'd only answer in a very specific domain. So now with the, the next version of voice AI, there's, uh, it's open-ended, and a lot of what's driving that is LLMs.
- 3:01
You can put any text into it, and, uh, depending on how the model is trained, you can output any text. And so if you're having a, uh, voice AI conversation, you can take the audio, turn it, using speech-to-text software, turn it into text, inject it into an LLM model, get text out the other side, and then use
- 3:17
a text-to-speech model in order to, uh, produce audio. And, uh, part of the, this now voice AI 2.0 is how fast can you do that and in how smart of a way can you do that so that the agent actually like does what it's supposed to, and then how expressive does it sound?
- 3:33
And so we'll talk about that a little bit, but, you know, this is, uh, the, the type of process speech... Uh, like speech-to-text to an LLM, I would just internally call that text-to-text.
- 3:43
Who cares if it's a, if it's a transformer model or whatever it is, but, uh, you transform speech to text, and then text to text, then text to speech, and you keep that loop going.
- 3:52
Um, and I would say, uh, what we're, what we're, what we're seeing right now, and we, we might, you know, the folks out there might say there's a big hype cycle.
- 4:02
We're at the peak of it or something. I would say, uh, hang on a second. This is more like 1910. You fee-- You see the first cars in the street or something like that.
- 4:11
Um, really, it's just a tiny thing that's happening right now, and there's gonna be a massive hundred x, uh, explosion in the next like decade for, uh, voice AI, text, et cetera.
- 4:20
It's, uh, AI's gonna be everywhere. The way I would think about this and, you know, encourage others to think about it this way is there's, there was like an agricultural revolution that took like a thousand, two thousand years to happen.
- 4:31
Then there was an industrial revolution that took like maybe two hundred fifty, three hundred years to happen. Then there was an information revolution that took like seventy-five years to happen.
- 4:39
See the trend? [chuckles] The intelligence revolution is going to take like twenty-five, maybe thirty years to happen. And so if you thought tech companies were fast before, AI companies have to move three times faster.
- 4:49
Um, so, uh, anyway, this is how it works today. There's a, um, speech-to-text system. You can see in there, there's an, an LLM or a text-to-text system, and, uh, then a text-to-speech system.
- 5:03
And each of these systems works fairly independently, um, but the state-of-the-art works very quickly, and they have high efficacy or accuracy. So speech-to-text now from, from even just five or, you know, eight years ago, it used to be maybe seventy-five percent accuracy or somewhere around there.
- 5:22
Uh, now it's like over ninety, and that over ninety, uh, really makes a big difference. Also, speech-to-text used to be maybe
- 5:29
two to five-second delay for real-time. Now it's like a hundred milliseconds or maybe two hundred milliseconds, um, with the high accuracy. And also you can run it on-prem, and you can co-locate all these services together.
- 5:42
Uh, actually, the, the founder of Daily, I just saw him roaming around. They just came out with a blog post showing that you can do the entire, uh, voice AI round-trip conversation in le-- in less than five hundred milliseconds using Deepgram for the speech-to-text, Llama for the TTT, and then, uh, Deepgram for the text-to-speech.
- 5:58
Uh, but nevertheless, that's what a human responds in. It's between like four hundred to six hundred milliseconds in turn-taking. So, um, you can, you can do all that here, but there is a piece that's missing, which is, um, if you...
- 6:11
These are all, like I just said, speech-to-text models, text-to-text models, text-to-speech. They're not passing along any context throughout the conversation. So the-- What, what this ends up with is a few, a few spots that you're like, "Hang on a second, um..."
- 6:26
It's not really getting exactly what I'm saying, um, and maybe that only happens ten percent of the interactions or twenty percent of the interactions, or what I really mean is like ten or twenty percent of the turns.
- 6:38
Um, but the way that you combat that is by, uh, adding in context. And so, um, the-- that's, that's what this, this view is right here. Uh, it looks like a really subtle change, but instead of there being speech-to-text models now where it just takes in audio and it puts out text, it will take in audio and,
- 6:56
and context. So think promptable, but not necessarily text promptable. It could be promptable with anything. It could be promptable with other audio. It could be promptable with images. It could be promptable with documents.
- 7:07
Um, it could be promptable with the previous turn of the conversation. Um, but what that gives you is it gives that speech-to-text model context. Um, when you send something to a speech-to-text model right now, it actually has to...
- 7:19
It's kind of amazing what it's able to do. It knows nothing about the conversation, and then it's just thrown into, like, a basketball game, and it has to transcribe everything, you know, quickly and, and with ju- with just, like, a few seconds of context and then do a really good job.
- 7:32
Um, w- what happens when you give that model the entire context of the conversation up until that point, it gets way more accurate. Uh, but it's not just about the accuracy because the next step in that is once you, uh, once you pa- you can pass that context along, you can pass the original input context along, but
- 7:48
you can also have your speech-to-text model output context as well, and that can output text that is human-readable. It can output audio. It could output images. It could output just embeddings, um, which people are now familiar with as, like, just a, a vector embedding.
- 8:03
Um, and so that can carry the state of the conversation, um, throughout the entire thing. I just wanna point out, this is not how systems are built right now.
- 8:10
Um, but in the next year, this is how they're going to be built, and this is when things are going to flip into, like, holy shit, this is-- this feels like human.
- 8:18
And because once that text-to-text is contextual from the audio, it knows it's hearing an angry person or a happy person. It knows that the conversation is flowing quickly or slowly.
- 8:29
It knows that there's, you know, light music playing in the background. It knows all this stuff, right? And so that text-to-text model can now generate the appropriate response. But it's not just about the text that it generates.
- 8:39
It ge- it generates its own context, right? So it can say to the text-to-speech model, "Hey, I need you to say this softly. I need you to say it slowly."
- 8:47
You know, I'm speaking very quickly right now, right? But, "Uh, I need you to say it slowly. I need you to say it in an authoritative tone." You know, that kind of thing.
- 8:56
And, uh, then when the text-to-speech model generates that audio, it will say, "I tr- I tried to generate it this way. I sounded like this. I think I did a good job," et cetera.
- 9:05
That's context that's gonna get passed to the next turn in the conversation. And so all this context is gonna be passed around. We call it contextual AI internally [REDACTED:username].
- 9:14
But, uh, this is what the next generation of these models is going to look like, and this is actually the innovation that's going to make it feel like a human because the speed part is taken care of, the accuracy part is taken care of.
- 9:24
Now it's all about context. Um, I know there's, uh, to preempt any follow-up questions [chuckles] about, uh, like, a mu- a multimodal model or a speech-to-speech model. Sure. Absolutely. It's, uh, you may mold or, like, meld some of these together.
- 9:39
You may put them all together. The problem with putting them all together is it's not as controllable. So we, uh, we are the largest, uh, speech-to-text API in the world now, but that's mostly because of businesses using us to power, like, Spotify, to power food ordering, to power, um, call centers, and that kind of thing,
- 9:59
and they need controllability in, in what they're doing. So if you just give, like, an open-ended prompt to a speech-to-speech model and just say, like, "Go to town," that's not the kind of experience that, like, a bank wants.
- 10:11
You know, [chuckles] they want a little more control. They want to maybe put a whole bunch of compute power in the speech-to-text to make sure that they get everything, uh, precisely right, and then the text-to-text doesn't actually have to be all that big because they're just doing a few...
- 10:25
They're, they're, they're just doing a few things like helping them reset their password or something like that, and then they want the text-to-speech to be, like, really expressive but, like, calm and only a single voice.
- 10:33
And so these are all gonna be kinda compartmentalized, um, because that brings us to the COGS conversation, the cost of goods sold. Um, right now a lot of people probably feel that, you know, AI is kind of expensive, but, uh, it, it doesn't have to be if you use the right tools and you use the right services
- 10:48
that focus on, uh, cost of goods sold, and so, um, and if you, if you choose the right size for each component in the stack.
- 10:56
So, um, uh, the, so in the future, that's what it's going to look like. Um, I'm trying to hurry through this because I, uh, wanna leave room for questions.
- 11:05
It's already been, like, fourteen minutes, um, or maybe twelve minutes or so. But, uh, uh, I'll j- I'll give you just a flash of, like, what the future will look like.
- 11:13
But, um, one thing that we're doing as, you know, a platform [REDACTED:username], um, we think, hey, if you, if you want... If you're doing anything in audio, you should be thinking about Deepgram.
- 11:23
Maybe you don't use this for every piece of it, but you should be thinking, "Hey, if I want low latency or real-time speech-to-text," you should definitely be thinking about Deepgram.
- 11:31
If you wanna be, uh, if you wanna be using low latency text-to-speech, uh, definitely at least talk to us. Um, that's a new product for us, but you know, the next version will be even, even more expressive.
- 11:41
But right now it's, I would say, uh, better than, like, Amazon, uh, Microsoft, et cetera, than their neural models. Not quite as good as Ele- ElevenLabs. Um, but, uh, but anyway, the next product coming out for us, which I want to give everybody the chance to, uh, try out or apply to for our preview, is our voice
- 12:01
AI agent, uh, which is a full, full stack where we put everything together. So y- you could, if you want, you can use your own API keys and use your own, um, LLMs.
- 12:10
Um, but, uh, also you could have it all put together with Deepgram, and this helps with reducing that latency, so you get your, um, turn-taking down to a very short, you know, three hundred milliseconds, five hundred milliseconds, six hundred milliseconds rather than, like, eight hundred or fifteen hundred or something if you tried to piece them together yourself.
- 12:27
Um, and if you wanna get access to this, uh, Voice AI Agent API, we have a QR code here. Um, and we have some folks in the back too.
- 12:36
If you saw a workshop, I think two days ago with Damien, he ga- gave an awesome workshop. He's in the back. You could talk to him about this. Um, but also just feel free to screenshot this or go to the link now or whatever it is.
- 12:48
Um, also, uh, [REDACTED:username], [REDACTED:username], we, um, give out, uh, two hundred and fifty dollars in credit, so anybody can, uh, try it out. Thanks, everyone. [outro music]