Voice Agents Can Just Do Things — Charlie Guo, OpenAI

Charlie Guo, who works on developer experience at OpenAI, opens with a deceptively simple correction: a voice agent does not have to talk back. Speech can be the input while the response appears as an action or a visual change. Once input and output are separated, “voice agent” stops describing one conversational interface and starts describing a wider design space.

That space has three modes: speech-to-speech, where the user talks and the model replies aloud; speech-to-action, where spoken intent leads to tool use; and event-to-speech, where an event causes the system to speak. None is fundamentally new. The Moviefone hotline loosely fits speech-to-speech, while spoken GPS directions fit event-to-speech. What has changed is how much more these patterns can do and how freely products can combine them.

What can happen after audio enters a product? The comparison makes the central design choice visible: the input channel does not determine the output channel, and one product can move among all three paths.

Speech-to-speech remains the familiar loop, and its strongest uses depend on information carried by delivery as well as words.

• Practice and coaching: A language-learning system can hear emphasis or emotion and respond with spoken feedback.
• Concierge support: A richer conversational agent can replace the rigid experience of navigating a phone tree. Guo hopes such support can become more pleasant than an ordinary support interaction, though the talk provides no measured comparison.
• Live translation: Fast speech processing can translate content as it is delivered, suggesting events that stream simultaneous dubbing in several languages.

These cases explain why spoken output sometimes matters: pronunciation feedback must be heard, customer support benefits from conversational pacing, and translation needs to preserve the flow of a live exchange. The mistake is making speech the response by default even when the user primarily needs work completed or state changed.

The underexplored mode is speech-to-action: the user speaks and the model uses tools. Guo calls voice a possible “capability overhang” because many applications already contain useful operations but do not expose them through speech. Three examples widen the idea from a mundane workflow to control of an entire computer.

• Form filling: Instead of manually entering repeated names and addresses, a user could describe the information aloud. Guo imagines replacing an hour of government paperwork with five minutes of speech, having the system complete 90% of the form, and then reviewing the result. Those quantities describe a desired workflow, not a demonstrated measurement. The review step remains essential because spoken input does not guarantee that every resulting field is correct.
• Creative tools: Someone may know what feels wrong in a piece of music or an image without knowing Ableton or Photoshop well enough to make the edit. Voice can help steer the software when “taste exceeds capability,” although it cannot eliminate the difficulty of articulating an aesthetic.
• General computer use: If a model can operate applications effectively, voice control need not stop at one app or terminal. The longer-term interface could address the computer as a whole; this remains a conditional direction rather than an established claim of complete human-level computer control.

The practical on-ramp is the software developers already maintain. A modern web application exposes verbs through API endpoints and React hooks. Turning selected verbs into model-callable tools gives voice input a path into existing behavior: the model interprets the request, chooses a tool, and the application performs the operation. Exposing a tool does not make it safe by itself; Guo explicitly retains the need for guardrails and safety checks while leaving their implementation outside this talk.

The response can stay visual. A successful tool call might populate fields, highlight affected text, change a button color, add a drop shadow, show a notification, or move a ghost cursor through the interface. These established UI signals often communicate progress more precisely—and less intrusively—than a spoken narration of every click.

How would the government-form example visibly change? The flow below follows one request from speech to populated fields and makes the human review point explicit.

Event-to-speech reverses the direction: an event arrives, and the model talks to the user. Guo considers this the least settled mode; a clearly AI-native pattern has not yet emerged. Two situations already justify it. In hands-free or screen-free contexts, the user cannot operate the interface or is attending to something else, as when cooking with a recipe app. In proactive outreach, the system needs to deliver information even though the user is not watching the screen.

Proactive speech must be selective. Applications produce endless events, but speaking every log would be intolerable. Guo instead places voice at the top of an escalation ladder: first animate an element, then show a popup, and speak only if the situation still needs the user’s attention. Audio is powerful precisely because it can interrupt; that makes restraint part of the mechanism, not a cosmetic preference.

Accessibility gives both action and speech a purpose beyond convenience. Guo describes developers who lost hand mobility or finger dexterity and later used language models, coding agents, and voice agents to produce far more code. The reported “orders of magnitude” improvement is anecdotal and lacks a defined measurement period, so it should not be generalized as a performance estimate. The concrete benefit is still clear: moving control away from repetitive manual input can let someone continue programming.

The modes become more useful when combined. In a car, a spoken request can start music while a navigation event later triggers a warning about traffic and rerouting. A game character could converse about the world, execute actions for the player, and react aloud to world events. The product is not choosing one voice architecture forever; it is choosing the appropriate transition at each moment.

Why revisit these old interaction patterns now? Traditional voice agents use a chain: speech is transcribed, text goes to a language model, the model may call tools and writes a text response, and text-to-speech converts that response back into audio. Each stage adds work, and transcription reduces the incoming signal to words.

OpenAI’s Realtime model family instead operates on native audio tokens: audio goes in and audio comes out, with continuous streaming replacing a rigid sequence of conversational turns. Native audio and continuous streaming solve different problems. The former preserves acoustic information for the model; the latter lets the exchange unfold without waiting for a complete turn at every boundary.

A transcript can omit tone, cadence, emotional force, attempted interruption, and background sound. Those cues can affect whether the system should answer, wait, clarify, or recognize urgency. Guo also says OpenAI’s earlier chained voice modes had significantly higher latency than its native approach, but the talk gives no latency values or task conditions, so it does not establish an expected response time for a particular application.

What architectural work disappears in the native path? The comparison shows why fewer representation changes can preserve more of the signal and reduce the number of serial stages before a reply.

Guo introduces GPT Realtime-2 as the latest model in OpenAI’s Realtime family and describes its central change as bringing reasoning to audio: it can spend additional reasoning effort before speaking. In his account, that offers a way to improve answers that would suffer from an immediate response, much as a text reasoning model can think before producing its final output.

Guo also attributes tool calling and parallel tool delegation to the model. These capabilities let a voice interaction check external information rather than improvise immediately, but they create a direct tradeoff: more reasoning and more tool work may improve the result while making the user wait.

A preamble fills the otherwise confusing silence. Before reasoning or calling tools, the model can say what it is about to do. The travel-agent example is ordinary and effective: it can explain that it is checking flight prices and ask for a couple of seconds, then perform the calls in the background. The preamble does not reduce the work or guarantee success. It tells the user why the agent has paused and what progress to expect.

Guo further describes GPT Realtime-2 as having longer context, better domain understanding, more natural voices, and greater steerability, including prompting it to wait until it hears an assigned name. He also says it performs well on recent audio benchmarks. These are capability claims in the presentation; no benchmark scores or wake-word reliability rates are supplied.

The closing question is not “What kind of voice agent should I build?” It is “What role do voice and audio play in this interaction?” That wording forces the design back onto the user’s situation rather than the novelty of the medium.

The answer requires four decisions:

• Perception: What can the model hear or otherwise sense, and which context does it receive?
• Action: Which tools are available, and which operations should it execute safely and correctly?
• Timing: Should it respond now, wait, or continue working in the background?
• Feedback: Does the user need speech, a visual notification, an interface state change, or some combination?

Guo closes with the belief that advanced intelligence will be spoken rather than typed. The more immediately useful conclusion is narrower: voice is one component of an interaction system. Treat it as an input, an action trigger, an accessibility mechanism, or a high-priority alert according to the job at hand—and let the software answer in the medium that communicates the result best.