Your Voice Agent is Just a Walkie Talkie — Neil Zeghidour, Gradium

Gradium co-founder and CEO Neil Zeghidour frames voice-agent progress around capabilities that often get conflated: open-ended conversation, useful action, natural vocal expression, low latency, and the ability to listen while speaking. Gradium grew from voice research at Kyutai and trains audio foundation models for tasks including speech recognition, synthesis, translation, and spoken dialogue. The history matters because each generation improved a different part of the experience rather than advancing all of them together.

The 2011 Siri demonstration already contained real agency. A person could ask for the weather or the Nasdaq, and the assistant connected that utterance to an application action. Behind the voice, speech recognition produced a transcript; natural-language understanding classified the request, selected the application, and triggered a supported operation. This was a complex, closed-ended pipeline: useful inside its predefined territory, unable to roam far beyond it.

OpenAI’s original voice mode reversed that balance. An LLM replaced much of the hand-built dialogue logic, so a user could request a bedtime story about Larry, a hedgehog with sunflower petals instead of spines, and receive an improvised response. The system gained conversational breadth and more natural speech, but in Zeghidour’s comparison it lacked Siri’s ability to retrieve weather or market information and still took several seconds to answer. Open-ended dialogue and agency turned out to be separate features.

The drive-through demonstration develops the next step through one concrete order. The customer asks for a chicken sandwich, selects the classic option, asks what else is available, and adds mac and cheese. The observable change is persistent task state: after several exchanges, the agent can report that the order contains both items. Zeghidour says payment would follow, although the shown exchange stops before completing it.

The causal chain is straightforward: the system interprets each utterance, consults the menu, records the chosen sandwich, answers a category question, adds the side, and computes the current order. An LLM is no longer merely generating a conversational response; it participates in an agent equipped with tool calling, reasoning, planning, and task state. That recovers agency in a more general form than the original closed assistant.

The remaining weakness sits in the cascade: speech-to-text converts the voice into words, a text model reasons over those words, and text-to-speech generates a new voice. This architecture can use a strong text model and dependable tools, but every component adds delay. More subtly, the transcript flattens information that was present in the input audio—tone, emotion, hesitation, and other nonlinguistic cues. Speech-to-speech promises to preserve that information while reducing latency.

A single speech-to-speech model can absorb the explicit speech-recognition, text-model, and speech-synthesis stages. In the advanced voice example, a person says he is nervous during a live demonstration and receives an immediate spoken response. Direct audio modeling can retain nonlinguistic information and deliver latency that Zeghidour considers already good enough. The cost, in his qualitative assessment, is that these speech-to-speech systems remain less capable than their text-based, cascaded counterparts; the talk does not present a benchmark for the size of that gap.

Speech-to-speech still does not imply full duplex. A half-duplex system divides the exchange into two states: the model speaks, or the model listens. That works until the listener contributes a brief “mm-hmm,” “yeah,” or “sure.” Humans use these backchannels to signal attention without asking for the floor, but a turn-taking system can interpret any incoming speech as an interruption.

The demonstration makes the failure visible. The model starts brainstorming, the user says “mm-hmm,” and the response stops. After the user explains that the acknowledgment was not an interruption, the model resumes—only to stop again at the next “sure.” Even the explicit lesson about backchanneling does not fix the exchange, because the failure is structural rather than instructional. The system must switch turns whenever it detects user speech.

That is the walkie-talkie problem. Faster turn changes can make half duplex feel responsive, but they cannot represent both sides speaking at once. Human calls include acknowledgments, interruptions, overlap, and moments when neither person speaks. Zeghidour says calls with a relative can contain simultaneous speech for up to 20% of the time; the talk supplies no measurement behind that figure, so it serves as an illustration of normal overlap rather than a quantified general rule.

A text language model predicts the next token from earlier tokens. A speech model needs an analogous prediction target for audio, but a raw waveform is far longer than a sentence’s word sequence. Zeghidour’s example takes about three seconds to say and contains eight words. At 24 kHz, those three seconds contain 72,000 waveform samples: 3 × 24,000.

That length is prohibitive for ordinary transformer attention. Under the quadratic-cost relationship used in the talk, multiplying sequence length by 10,000 multiplies attention cost by 100 million. The 10,000 factor is a rounded illustration rather than the exact ratio between 72,000 samples and eight words, but it captures the scale problem: treating every audio sample like a language token makes the sequence astronomically expensive.

A neural codec, also called an audio tokenizer here, changes the representation. An encoder compresses the waveform into a dense sequence of learned audio tokens; the language model predicts over those tokens; a decoder reconstructs high-quality audio. Tokenization makes speech manageable for language-model training, but it does not solve duplex behavior. If user and system tokens are still placed one after another in a single sequence, the conversation remains turn-based.

Full duplex adds a second representation change. Instead of one token stream alternating between user and system, a multi-stream model represents both participants independently. At any instant, both may speak, both may be silent, or either one may speak alone. Overlap is no longer an exceptional event that forces a turn transition; it is an ordinary state in the model’s input and output.

The Moshi demonstration makes that relationship audible through a fictional spacecraft exchange. A person asks for a route, travel time, and mission readiness. The model sometimes anticipates the end of a question and begins answering before the speaker finishes. Crucially, simultaneous speech does not terminate the interaction. The demonstration establishes overlap handling and conversational timing, not actual trajectory calculation or verification of the ship’s supplies.

Full duplex brings its own behavioral problem: the early model interrupted too often. Zeghidour describes that as irritating, even while reporting that the conversation remained continuous through noise and coughing. Two streams make overlap survivable; they do not teach politeness or determine when anticipating a speaker is appropriate.

Natural flow also did not make the model capable enough for serious agent work. Zeghidour bluntly describes the early full-duplex system as less intelligent than speech-to-speech models that were already behind cascaded agents. Casual conversation may tolerate that. An agent entrusted with tool calls, planning, or transactions cannot give up reasoning merely to sound more human.

His explanation is finite model capacity. A text model adapted to speech must understand language, interpret audio, and generate audio using a fixed collection of weights. Those extra modality responsibilities consume capacity that could otherwise support reasoning. This is a proposed explanation for the observed tradeoff, not a measured law quantifying how many reasoning capabilities each audio feature costs.

The first path is to keep the system end to end and scale it: use a larger speech-to-speech model, better pretraining, and stronger post-training until its intelligence becomes sufficient for more tasks. The operational advantage is simplicity—a single model handles the interaction. The development disadvantage is that adapting a frontier text model to understand and produce audio is complex, slow, and expensive. Zeghidour speculates that a major provider follows this path but explicitly says he does not know its internal architecture.

The second path separates the voice interface from the brain. A small, potentially on-device full-duplex model maintains timing, overlap, and natural speech. When the conversation requires reasoning, tools, planning, or external knowledge, it delegates asynchronously to a background text model. A shared backend could serve many small voice interfaces and return text for them to express conversationally.

This hybrid depends on a difficult handoff: the small interface must know when it does not know. The talk identifies that requirement without explaining how uncertainty is detected, how delegation is trained, or how a delayed answer is reconciled with a conversation that may have moved on. Those coordination details determine whether the split feels natural rather than like a cascade wearing a better voice.

Zeghidour favors the split for two practical reasons:

• Cost: Ordinary small talk does not need a gigantic multimodal model capable of solving differential equations. A small interface can handle routine interaction and reserve expensive reasoning for requests that need it.
• Backend choice: Developers can replace the text model as better reasoning systems appear. A unified speech-to-speech product ties conversational behavior and intelligence to one provider and to a costly audio-adaptation cycle.

The closing bet is deliberately more demanding than an impressive demo. A viable voice agent must match the natural overlap of full duplex while remaining economically competitive and capable of the tool use available in today’s cascaded systems. The hybrid architecture is presented as Gradium’s route toward that combination, not as proof that the handoff, economics, and agent reliability have already been solved.