Speech-to-Speech Model Research at Google DeepMind — Valeria Wu Fon & Tom Ouyang, Google DeepMind

Adding voice to an agent can look straightforward: put automatic speech recognition (ASR) before the agent and text-to-speech (TTS) after it. But recognizing words and speaking an answer leave much of a conversation to the surrounding system. Valeria Wu Fon, Gemini’s speech-to-speech product lead, and Tom Ouyang, a speech-to-speech engineer, introduce their work through that gap. The intended applications span everyday questions in Search Live and Gemini Live, as well as enterprise voice agents exposed through cloud and API products. 1:33

In Ouyang’s account, speech recognition before roughly 2018 usually required a chain of specialized components: feature extraction, acoustic modeling, pronunciation modeling, language modeling, and a second pass that rescored candidate transcriptions. Moving toward end-to-end neural systems let a model learn the mapping from acoustic input to text, reducing the amount of domain knowledge needed to assemble that chain.

The scope of the learned task remained narrow. Audio went in; a transcription came out. Translation, replies, descriptions of a speaker’s emotion or pace, and the use of images still required additional system work. Collapsing the recognizer’s internal pipeline therefore did not create a general conversational model. Every new capability could bring another integration problem.

Gemini changes where those relationships are learned. Rather than merely attaching audio embeddings to a text model, Ouyang describes pre-training on interleaved multimodal examples. A single training example can begin with a text instruction, continue through a sequence of video and audio, and require an output that depends on both. The modalities meet during the stage that supplies the bulk of the training data. 3:03

Consider the bedtime-story example. The instruction asks for a summary; the following video and audio supply the story. The expected output includes a written summary and timestamps for interesting events. To satisfy that target, the model must turn its understanding of what it hears and sees into text, while retaining where relevant events occurred.

Different training tasks exercise different directions through the same foundation:

• Video captioning: Audio and video both inform the caption, so the text can reflect information carried by either signal.
• Speech recognition and synthesis: Examples connect audio to text and text to audio.
• Agentic tasks: Task-oriented examples join those modality conversions in a unified token embedding space, giving the model a foundation for understanding how audio, video, and text relate.

What does the bedtime-story task require the model to connect? The diagram follows the example from instruction through sensory input to its output target. The summary and event timestamps depend on the audio and video together: training asks the model to express both the story’s meaning and where events happen. This is a picture of the training task, without specifying an encoder design or tokenization scheme.

Live translation puts those learned relationships under a timing constraint. A listener can specify English as the desired output while friends speak Spanish, Italian, or Chinese. The model must translate as they speak. An offline translator has the full utterance available before producing a result; the streaming system begins with incomplete input. 4:33

Ouyang reports that the team finds streaming translation quality about as good as offline systems, though the talk does not specify the evaluation conditions or numerical results for that comparison. The task combines several demands: switching among languages without knowing them beforehand, preserving the source speaker’s voice, understanding multiple speakers, tolerating noise, and producing output in real time. He attributes much of that capability to pre-training, making the application almost a prompting task.

Translation is one mode of the shared foundation. With an instruction to translate and incoming audio, the model produces streaming translation. With image and audio input and a question about the image, it answers. An embodied agent can add a face and tools that show information. The proposed advantage is reuse: different prompts call on relationships already learned by one model. 5:33

Wu Fon organizes the research around three objectives that must coexist in the same product:

• Conversational: Low latency, quick responses, and a natural conversational rhythm.
• Intelligent: Task completion, instruction following, and reasoning that help the model accomplish something useful.
• Multimodal: Inputs can include video, screen sharing, and PDFs alongside audio, while outputs can go beyond speech.

6:10

Internationalization runs through all three. Wu Fon says she believes the majority of Gemini users are non-English speakers, so the goal includes making these capabilities work across the languages customers use. A system that speaks fluently in one locale still has work to do before it can serve the intended audience.

The objectives interfere with one another. Increasing the thinking budget gives the model more time to reason before answering or calling a tool, and Wu Fon reports that this improves intelligence evaluations. But that work delays the first audio response. The user experiences the delay as a pause in the conversation, which can undermine its natural rhythm. The research challenge is to improve useful reasoning while keeping the exchange responsive; the talk presents this as ongoing work. 7:39

The first demonstration returns to translation. The live model is described as supporting 70-plus languages, and a Google Meet example shows the interaction: enable speech translation, select the desired language, and let several participants converse. The exchange moves through weather in Shanghai, a family visit to a park, and a birthday dinner at a restaurant in Sweden. 8:39

The timing matters as much as the translated content. Wu Fon highlights that translation begins shortly after a participant starts speaking, reducing the feeling of a rigid, turn-by-turn relay. The model starts catching the new speech directly, so participants do not have to treat every interruption as a disruption to a separate translation cycle.

The next example moves into Search Live, which Wu Fon says uses the same speech-to-speech model as the developer-facing Live API. In the sofa demonstration, real-time video and audio let the user ask a question without first describing the furniture. A tool call then brings up relevant search cards, adding a visual route to more information alongside the spoken exchange. 11:12

Localization also changes the answer itself. Wu Fon describes the response as Spanish localized for Spain, with the furniture term mid-century left in English because that term is commonly used in Spanish. The useful behavior is selective: preserve a familiar borrowed expression while speaking the surrounding answer in the user’s language. Translating every word would miss that usage.

The roadside-assistance demonstration gives the same model a different job. A driver has blown a tire and pulled over. The agent begins by asking for a name and policy number. When the driver cannot supply the policy number, the agent changes the information it requests: a registration plate and postcode can support the lookup instead. That change keeps the task moving despite a missing identifier. 12:14

A truck horn interrupts the situation, and the driver reacts before providing the requested details. The agent subsequently says it has found the record, warns the driver to stay clear of the road, and asks whether the vehicle is a blue MINI Cooper F series. The driver confirms. The observable progression is from an unavailable policy number to alternate identifiers, then a retrieved vehicle description and confirmation. The demonstration stops there; it does not show assistance being dispatched.

This task makes two requirements especially visible:

• Alphanumeric accuracy: Plates, postcodes, and addresses carry task-critical characters. A plausible-sounding conversation is insufficient if the identifiers used to locate a record are wrong.
• Proactive audio: The model decides when external input warrants a response or interruption. Background noise or another person talking should not automatically make it stop speaking or cut itself short.

13:21

Here, proactive includes knowing when to leave the conversation alone. The model must distinguish relevant input from sound that happens around the user. Wu Fon connects that requirement to the places voice agents will actually be used: on trains, during walks, and on the go. Quiet-office assumptions would make the model’s conversational timing poorly suited to those environments.

The final demonstration adds a visual presence. Wu Fon introduces a pilot with Citi at Cloud Next that supports customized real-time avatars, ranging from hyperrealistic humans to cartoons. The experience uses the same speech-to-speech model and combines low-latency conversation with multilingual lip syncing. An avatar gives the spoken interaction another output whose timing must remain coordinated with the exchange. 14:21

In the example, a user asks about progress toward a daughter’s college fund. The agent replies about the goal, mentions a possible opportunity, and then acknowledges that another person has joined, offering congratulations on college acceptance. Wu Fon uses the demo to bring together multimodal input and output, tool calling for relevant user information, conversational fluidity, and internationalization. It is presented as an early convergence of those capabilities in a pilot.

The closing ambition is broader than giving an assistant a good voice. Wu Fon’s forecast is that AGI will be spoken. For the team, that means one promptable model must let a user move among translation, taking action, brainstorming, and rambling. Each mode changes what a useful response looks like: translation follows someone else’s speech, an agent advances a task, and an open-ended exchange follows the user’s thought. The intended product must switch among those modes while keeping its understanding, timing, language use, and outputs working together. 15:56