Neil Zeghidour - Voice AI: when is the "Her" moment?

What does it take to reproduce a voice—and how much further is that from building an assistant people can talk to naturally? Gradium supplies the model building blocks: speech recognition, speech synthesis, speech transformation, translation and spoken dialogue. Neil Zeghidour distinguishes that work from orchestration and applications for particular industries. The intended customers are developers assembling voice agents and other voice products.

The opening demonstration makes the synthesis problem tangible. A synthetic voice, which Zeghidour identifies as a Joe Rogan clone, describes recording roughly ten seconds of speech, capturing tone, pitch, accent and vocal quirks, then speaking newly supplied text. That reference length is a claim in the demonstration, not a measured cloning benchmark. The striking result is the separation between vocal identity and content: the recognizable voice can deliver words supplied afterward.

Gradium grew out of a nonprofit research lab backed by philanthropists including Eric Schmidt, Rodolphe Saadé and Xavier Niel. Its open research included Moshi, speech-to-speech translation and the CPU-oriented Pocket TTS. The commercial structure was created to turn that research lineage into products usable in production. Convincing synthesis is therefore the starting point for the talk, not its definition of a finished voice assistant.

In the introductory scene from Her, the human asks his new assistant what to call her. Samantha supplies a name, explains that she chose it herself, and says she likes its sound. The exchange feels spontaneous because the voice, timing and responses work together. Despite the overuse of the film as an industry analogy, it remains a useful reference for the interaction being promised.

Two contemporary demonstrations show both progress and the remaining distance. An ElevenLabs government helper answers a request to start a business with steps covering structure, naming, registration and documents. Zeghidour then shows his own experiment putting streaming voice models into a Reachy Mini robot. Asked to become a gym enthusiast, the robot adopts the name Logan and an energetic gym-bro voice, offering encouragement to lift and train. One demonstration supplies useful procedural information; the other changes voice and personality on request.

Both sound more natural than earlier voice systems, but response delays and simultaneous speech remain problems. Better language-model intelligence now gives developers useful agents to put behind a voice. Yet a system whose reasoning depends on a text transcript cannot use vocal information that the transcript leaves out. Natural-sounding output does not by itself produce natural conversation.

The conventional cascade runs speech-to-text → LLM → text-to-speech. Streaming recognition and synthesis let components begin processing before an entire utterance or response is complete; voice cloning controls the output voice, while semantic voice activity detection helps determine when the user has finished. These are useful building blocks, but fast components do not automatically make the combined interaction fast enough.

Zeghidour reports that TTS alone still takes more than 200 milliseconds in the comparison he presents, against a conversational target of around 200 milliseconds for the complete response. The human comparison needs a precise interpretation: turn-taking research describes short gaps between speakers, with response planning overlapping incoming speech, rather than all comprehension and production happening serially inside that gap. Even the simpler machine latency comparison here excludes tool calls and actual task execution.

Another Her scene raises the bar: Samantha inspects thousands of emails and immediately notices that correspondence concerns a job the user left years ago. Real agents must wait for external work. Zeghidour cites tool-call or OpenRouter delays of 500 milliseconds to four seconds, compared with efforts to save 10–20 milliseconds in TTS. These are illustrative ranges without a specified workload or measurement protocol. The engineering problem becomes resilience to unpredictable tool completion, not merely shaving time from speech generation.

One approach separates the tool operation from the speech that keeps the interaction moving:

1. Send the tool request.
2. Continue speaking naturally while the result is pending.
3. Incorporate the returned information when it becomes available.

The challenge is that the model does not know in advance how much conversational space it needs to fill. The speech must lead naturally into the result rather than sound like an unrelated prerecorded waiting message.

The live travel-agent demonstration follows this pattern. Colin from Wanderlust Travel asks where two people would like to go for their April 10–13, 2026 trip. Zeghidour chooses Tokyo. The assistant first describes the city’s skyscrapers and peaceful shrines, then announces accommodation options and begins mentioning the Fairmont. The destination commentary occupies the interval before the options arrive. Zeghidour calls the prototype unfinished: this manages perceived waiting and conversational continuity; it does not make the underlying retrieval finish sooner.

Replacing the cascade with a single speech-to-speech model removes boundaries between separate recognition, reasoning and synthesis components. Speech goes in and speech comes out, reducing latency. But that architectural choice does not settle whether the system can listen while it speaks.

Zeghidour characterizes prominent alternatives, including OpenAI’s advanced voice offering and Sesame’s voice model, as half duplex at the time of the talk. That is his assessment of contemporary systems, not a permanent classification of their products. The distinction matters because a cough or brief acknowledgment need not mean the user wants the assistant to stop.

Backchanneling is speech that signals attention without taking the floor. Its use varies across languages and cultures. Zeghidour gives Japanese conversation as an example, where frequent acknowledgments can express politeness and active listening, and cites overlap reaching 20% of conversational time in that context. A voice system that treats every acknowledgment as an interruption makes an ordinary listening behavior disruptive.

The hotel-room demonstration exposes that failure directly. Zeghidour asks an assistant to brainstorm a talk about voice AI and Her. As it starts answering, he supplies brief acknowledgments: mm-hmm, yeah and sure. Its responses become interrupted or restarted instead of developing the requested ideas. He explains that he is only signaling attention and asks it to keep going, but the pattern continues. Instructions about conversation cannot compensate for an interaction mechanism that repeatedly yields the floor at the wrong time.

The example is deliberately demanding, as Zeghidour acknowledges, but the behavior it tests is commonplace. It also helps explain why a polished demonstration recorded beside a phone in a quiet room is insufficient evidence of conversational robustness. Overlap, noise and less controlled surroundings expose additional ways an interaction can break.

Moshi supplies the contrasting demonstration. In a recording Zeghidour describes as nearly two years old, his co-founder Alex role-plays a spaceship conversation. He asks for a course to Sirius 22, asks how long the journey will take, and checks whether the ship has everything needed to begin. Moshi responds while the human continues speaking and acknowledging its answers. The fictional navigation exchange demonstrates conversational timing, not an executed navigation tool.

Two behaviors distinguish this from simply ignoring interruptions. The model can anticipate where an utterance is going and begin answering before the user finishes. At the same time, speech arriving during its answer remains available for subsequent responses. Continuing to speak and continuing to listen are compatible operations. Zeghidour also describes Moshi as robust to noise and multiple speakers, although this clip is an illustration of overlap rather than a comparative robustness benchmark.

The next Her exchange introduces a different requirement. Samantha interprets the human’s tone as a challenge and connects it to curiosity about how she works. Zeghidour reads the scene as recognition of discomfort: the assistant responds to how something is said, not only to the literal words. This is paralinguistic understanding.

Speech-to-speech models retain acoustic information that a text transcript can discard. Availability, however, does not guarantee useful behavior. If training consists of spoken versions of factual instruction-following examples, the correct answer may be identical regardless of tone. Such data gives the model little reason to exploit vocal cues. Learning to respond to those cues requires examples in which the manner of speaking changes what a relevant response should be.

Zeghidour then points to NVIDIA’s PersonaPlex, which builds on Moshi. Its full-duplex design also makes clear that Moshi is not literally the only full-duplex model; the useful distinction is the interaction architecture and its descendants. Moshi’s fluid conversation was a strong foundation, but Zeghidour describes the original model as lacking enough intelligence to remain useful after a short exchange. It had neither tool calls nor the ability to perform tasks.

Production introduces further requirements:

• Observability: Developers need to inspect behavior and detect speech that should not be accepted.
• Paralinguistic behavior: Preserved audio cues must actually influence responses; the original Moshi did not provide meaningful understanding of this kind.
• Reliability, intelligence and personalization: Natural interaction must meet the practical expectations already served by cascaded systems.

These requirements explain why conversational flow alone is insufficient to replace an existing agent stack.

Zeghidour consequently revises his earlier opposition to cascades: their practicality and convenience matter. He believes the full-duplex approach, trained on better data, can deliver humanlike conversational flow. But that remains his projection; the harder replacement test is whether such models can also provide the capabilities and controls that make an agent dependable.

Suppose the conversational and agent problems are solved. An assistant resembling Samantha might then be used for hours each day, or remain available throughout work while the user occasionally asks it to do something. That makes scalability a product requirement rather than an optimization to postpone until after voice quality is solved.

Zeghidour argues that large multimodal voice services can be expensive enough to operate at a loss, though he supplies no financial evidence for that claim. His more concrete concern comes from consumer-app developers: in the situations he describes, text LLMs, speech recognition and diarization are relatively inexpensive, while TTS dominates the bill. He recounts seeing teams spend their fundraising on synthesis before they have a chance to grow their user base. The relevant constraint is sustained usage cost, not whether a short demo is affordable.

Persistent use also invites more personal disclosure. As users entrust an assistant with more of their lives, control over where information resides becomes more consequential. Zeghidour invokes fears surrounding Mythos and future database attacks as a motivation for keeping private data local; those fears are not an established forecast of universal compromise. The practical motivation is to reduce the amount of sensitive information that must leave the device.

Gradium Phonon is presented as a first step toward local deployment. Here, on-device means running TTS on a smartphone CPU, not requiring a gaming GPU. Zeghidour describes it as having fewer than one million parameters; Gradium’s April evaluation instead lists approximately 100 million for Phonon 20260318. Those specifications conflict, so the smaller figure should not be treated as settled.

Zeghidour claims strong quality relative to existing on-device models and contrasts Phonon’s voice cloning with Kokoro, which he describes as lacking that capability. The universal superiority claim is broader than the vendor’s published comparison supports: the April evaluation covers particular synthesis tests and alternatives, not every on-device model or smartphone performance condition. The important product combination is local CPU execution with personalized speech.

The accompanying audio demonstration cycles through synthetic character voices discussing local processing, including a voice addressing Morty and another joking about donuts. The clips illustrate expressive synthesis, while their script promotes privacy and independence from servers. Local TTS can keep synthesis on the device; it does not establish privacy for an entire application if other components still send data elsewhere.

Running synthesis on the smartphone avoids a per-request TTS API charge, which is the scaling advantage Zeghidour emphasizes as he announces the private beta. It is not cost-free deployment: the launch announcement describes fixed licensing costs by device and model type. The intended change is to let consumer-app usage grow without a corresponding stream of synthesis API charges.

That leaves a substantive last mile. Voice is not a solved commodity merely because generated speech sounds good. Useful everyday interaction still requires science and engineering across conversation, agent behavior and deployment. Zeghidour closes by returning to the Her analogy and directing builders to Gradium: reaching that experience means making natural interaction dependable and affordable enough to use beyond the demonstration.