Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI

The presented harness’s starting constraint is awkward but common: the voice agent needs the judgment of a capable language model, yet a phone caller notices every pause. Bohan Li approaches the problem through an analogy to his previous self-driving work. Instead of treating voice as one opaque model call, the system separates perception, planning, and control so each stage can be accelerated on its own terms.

The mapping is direct:

• Perception — transcription: Audio signals become text the reasoning system can process, much as cameras and lidar turn the road into usable observations.
• Planning — the language model: The model consumes that transcription and decides what the agent should say or do.
• Control — speech synthesis: Generated text becomes audio, analogous to turning a planned trajectory into physical controls.

This decomposition exposes three different waits: recognizing the caller, preparing a useful response, and producing audible speech. The harness can overlap work across those waits instead of demanding that one faster model solve the entire latency problem.

The perception layer uses what Li calls a streaming speculative transcriber. A fast streaming engine emits text immediately, while a slower batch engine processes more audio and conversational context to produce a more accurate interpretation. Downstream work can begin from the fast result instead of waiting for the slower pass every time.

What happens when an accurate result arrives after the conversation has already moved on? The correction path is freshness-aware. If both engines produce the same text, there is nothing to revise. If new streaming audio arrives while an older corrective pass is running, that correction is canceled because it no longer represents all available audio. A carefully computed stale answer loses to a newer observation.

The concrete example begins after the agent asks for a name and date of birth. Early fragments such as “Sure” arrive quickly but do not contain the requested fields. Once the slower engine has enough audio and the context of the question, it can distinguish the name-like portion from the date-of-birth portion. Punctuation-only changes are ignored, and the resulting text is released to the agent.

The diagram answers a timing question: how can the slower path improve accuracy without making every turn wait for it? Both paths begin from the same audio, but only a still-relevant correction replaces the streaming hypothesis. Newer audio can invalidate work already in flight.

The planning layer attacks sequential model round trips. Tool calls are expensive here because a slow but intelligent language model may need to generate a request, wait for execution, ingest the result, and generate again. The presented harness moves suitable tool work into background agents, then inserts the result into the main model’s context as though that model had made the call itself. The talk describes the context insertion but does not specify its message format.

Every new transcription detection can also start an eager generation. Crucially, generation and emission are separate decisions: the model may prepare a candidate response while the caller is still speaking, but the harness does not play it until the end of the utterance is confirmed. This spends some computation speculatively in exchange for a chance to have the right response ready sooner.

The name-and-date example shows the revision process. “Sure” triggers an early generation, but the background extractor has no requested fields yet. More partial text arrives, still without a recognizable name. From that incomplete evidence, the main model starts preparing to ask the caller to spell the name because it suspects a transcription problem. Nothing has been spoken, so this wrong turn remains cheap to discard.

The observable change occurs when the corrected transcription arrives. The background agent can now identify the name and date of birth, including phonetic matching for a possible name mistranscription. The harness cancels the draft created without those results and starts a new generation with the extracted fields in context. Only after turn completion does the selected response leave the system.

Speculation does not remove inference; it deliberately creates work that may be thrown away. Its value depends on how often useful generation overlaps the caller’s remaining speech and how expensive canceled generations are. The talk provides no latency, cost, or cancellation-rate measurements.

The control layer receives the response as a token stream. Waiting for the complete sentence before requesting speech would expose the remaining generation time to the caller, so the goal is to begin playback while the language model is still writing the continuation.

A prefix cache watches the stream for word sequences whose audio has already been generated, either during an earlier response or elsewhere in the same call. It does not immediately accept a one-word match; in the walkthrough it waits until three words before registering the first hit. At the same time, the growing text is sent to Cartesia over a WebSocket for fresh synthesis.

The developing sentence begins with the reusable phrase “You said your name is.” Such scripted openings recur, so their audio is a plausible cache target. The next token introduces the caller’s particular name and causes a miss. That miss becomes the release point: the harness immediately plays the cached generic opening while the personalized continuation is still being generated.

From the caller’s perspective, speech has started. Internally, the language model and speech provider are still finishing the sentence. The talk demonstrates why common openings are reusable, but it does not report a cache-hit rate, so the frequency of this latency win remains unspecified.

Playing cached words creates a second problem: text-to-speech sounds more natural when the provider sees the whole sentence, but the opening has already been heard. The presented harness still sends Cartesia the complete text, including that opening. The provider therefore synthesizes the sentence with its normal context and prosody, unaware that a cache exists.

When the fresh audio returns, the harness suppresses the portion corresponding to the prefix and plays only the remaining audio after the cached clip. This preserves sentence-level context for synthesis while avoiding duplicate playback. Li allows that there may be a small hiccup; the talk does not explain how the two audio versions are aligned at the cut or measure whether listeners can detect the join.

The diagram answers the key control-layer question: how can cached speech start immediately without forcing the provider to synthesize an isolated, context-poor suffix? The same full sentence travels through both branches. One branch supplies early playback; the other supplies a natural continuation, with its duplicate prefix removed before output.

The final demonstration is a prerecorded clinic-booking call. The supplied recording does not establish the caller’s identity or whether the patient details are real or synthetic. The caller asks to schedule an ultrasound. Elise requests a name and date of birth, confirms that the caller is a new patient, and asks permission to text an insurance-upload link. The captioned digits accompanying the birth-date request are not interpreted or expanded here.

The more revealing moment is a change of plan. The agent begins offering the earliest appointment, but the caller interrupts to check a calendar. Elise stops and says, “Sure. Take your time.” When he asks for the following week, it offers two times on Tuesday, July 7. He selects 2:00 p.m., and the agent confirms the booking.

That sequence matters because it is not merely a fixed questionnaire. The conversation pauses, abandons an in-progress offer, accepts a new scheduling constraint, presents alternatives, and reaches a selection. The caller hears short, conventional turns while transcription revisions, tentative generations, field extraction, tool work, and streaming speech can occur behind them.

The demonstration illustrates the intended conversational experience, not a full systems evaluation. It does not expose the underlying tool traces, establish that the demonstration used a live clinical scheduling system, or provide comparative latency and recognition measurements. What it does show is the product goal: substantial background coordination should collapse into an uneventful phone call.

Li closes by placing the engineering work inside EliseAI’s focus on housing and healthcare—areas he describes as critical in people’s lives. The company is headquartered in New York and was working to expand its Bay Area presence. He ends with an invitation to follow the company and join the team.

That framing explains why the harness matters. Natural conversation is not merely a cosmetic layer when a caller is trying to schedule care or handle an important housing workflow. A frontier model alone does not create a realtime voice agent: the surrounding system decides when provisional work may begin, when stale work must be canceled, when tool results should revise a draft, and when cached and fresh audio can safely reach the caller.