Voice and Real-Time AI

Spoken interaction and shared understanding

Turn-taking coordinates who speaks and when. Grounding establishes shared understanding; repair corrects misunderstanding. These processes connect recognized words to an interpreted request and a response the person can use. The dialogue-systems foundation distinguishes accumulated meaning from individual utterances. [2]

A directions assistant receives a request, encounters a hesitation, and starts answering. The person then corrects the destination. Assume route computation is available; the interaction performs no external action. The table separates what was said from what the system must establish. Its conversational consequences need no invented timing measurements. [3]

Speech, text, and response boundaries

Automatic speech recognition, or ASR, maps speech to text. Text to speech, or TTS, produces spoken audio from text. A cascade places a text response generator between them. Direct speech-to-speech processing instead accepts and produces audio through learned representations, changing the interfaces available for inspection and control. [6]

If ASR substitutes Oakland for Auckland, response selection can produce coherent directions for the wrong destination, and TTS can pronounce them perfectly. Inspecting the text boundary helps locate the substitution. More generally, representation loss determines what remains available downstream; output decoding determines how a model representation becomes usable media. [9]

Half-duplex interaction alternates listening and speaking; full-duplex interaction allows both simultaneously. Speech-to-speech describes processing structure, not duplex behavior. A model can accept audio directly yet still wait for an endpoint before answering. Conversely, simultaneous audio streams do not establish reliable interruption, useful answers, or correct recovery. [10][11]

Audio samples, frames, and media clocks

A waveform represents signal amplitude over time. Sampling measures it at discrete times; quantization assigns representable amplitude values. Sample rate counts measurements per second per channel, and channels carry separate signals. Pulse-code modulation, or PCM, stores quantized samples. A codec encodes and decodes audio, often using compression. The speech-feature foundation explains these units. [12]

Frame has several meanings. A sample frame contains one simultaneous sample from every channel. A processing frame groups samples for an algorithm; a codec frame groups encoded audio; a packet carries transport data. Their boundaries need not coincide. Preserving source intervals through these transformations is an application of temporal correspondence. [13][12][14]

For a 16 kHz, two-channel, 16-bit PCM stream, one sample frame occupies $2\times16/8=4$ bytes; one second represents 64,000 bytes. Under the WAVEFORMATEX contract, transfers must begin on block boundaries and contain complete blocks. These fields alone do not specify every detail of byte interpretation. [13]

The selected Windows PCM packing uses signed 16-bit little-endian values, interleaved left then right. The bytes 00 80 FF 7F therefore encode one sample frame: left = −32768, right = 32767. Little-endian means the low byte comes first. PCM alone is insufficient: the same documentation defines 8-bit samples as unsigned. [15]

Interpreting 48,000 unchanged mono samples at 24 kHz instead of 48 kHz changes duration from one second to two; at unit playback rate, frequencies halve. Resampling instead constructs samples on a new time grid: one second at 24 kHz contains 24,000 samples. Changing the declared rate is not conversion. The Web Audio specification distinguishes these operations. [16]

In RTP, sequence numbers track packet order; timestamps identify source sampling instants using the payload clock. Arrival and playback occur later. Different streams can have different clock rates and offsets, so cross-stream timing needs an explicit mapping. A packet timestamp alone does not establish wall-clock time or device playback. [17]

Recognition as a revisable interpretation

Recognition combines acoustic evidence with learned language regularities. Acoustic features summarize short signal intervals; decoding selects candidate output sequences. Spoken words do not arrive as neatly separated packets. Whisper, for example, maps log-Mel features—compressed measurements of energy across frequency bands—to text and task tokens; its transcription, translation, and timestamp outputs serve different purposes. [18]

Connectionist Temporal Classification, or CTC, illustrates the alignment problem. It predicts labels and a blank symbol across acoustic steps. The path A, A, blank, B, B collapses to AB by merging consecutive repetitions and removing blanks. Multiple paths can represent the same output. CTC does not itself guarantee streaming, accurate word timestamps, or conversational completion. [19]

A partial hypothesis is a provisional transcription. A stable prefix is text retained through later hypotheses; estimated stability predicts that persistence, not correctness. Waiting for stronger stability adds lag. Recognition can therefore support early response preparation, but a draft based on an unsettled destination must remain replaceable. [20]

A final segment closes a recognizer's result. It need not close the person's thought. Amazon Transcribe separately exposes segment finalization and fixed items under stabilization. A service-guaranteed fixed item and an estimated stable prefix have different contracts; neither means the words are correct. [21]

Contextual biasing favors vocabulary relevant to the conversation. It can narrow candidate destinations or specialized names, but expected words can also bias interpretation incorrectly. Short answers deserve particular attention because they contain little redundancy. One clinical implementation rescores single-word replies using conversation context; its examples include confusions between a number and a similar-sounding word. [22][23]

Test names, numbers, accents, code-switching—changing languages within an exchange—and missing or delayed transcripts. Timestamp and confidence availability are model-specific. When a confidence score exists, validate what it predicts: word correctness, whole-utterance correctness, or correct task information. Probability calibration explains why fluent wording or high likelihood is insufficient. [24][25]

Speech synthesis and audible commitment

Text normalization chooses spoken forms for written expressions such as numbers and dates. A conventional TTS system predicts an acoustic representation, then a vocoder converts that representation into waveform samples. Choosing the response's meaning is a separate operation. [26]

Pronunciation maps selected words to speech sounds. A room number might be verbalized digit by digit; that choice precedes pronunciation. A lexicon stores pronunciations, while letter-to-sound rules can handle unfamiliar words. Festival's lexicon interface demonstrates this distinction without requiring every modern synthesizer to expose explicit phonemes. [27]

Prosody comprises pitch, rhythm, stress, pauses, and speaking rate. Context can change pronunciation, as with present- and past-tense read, and delivery can change conveyed meaning. SSML defines pronunciation and prosody controls, but their interaction with a processor's automatic choices is implementation-dependent. Supported controls still require listening tests. [28]

Codec-based generation predicts compact audio codes that a decoder turns into a waveform. The codes are representations, not playable sound. VALL-E illustrates this alternative and reports omitted or duplicated words, reinforcing the need to assess content fidelity separately from naturalness. A boundary error can also send delivery instructions to TTS as dialogue, causing them to be spoken aloud. [29][30]

Incremental synthesis starts with a bounded text prefix. Smaller chunks permit earlier output but withhold context that may change pronunciation or phrasing; waiting for a phrase provides more context at the cost of delay. Lookahead experiments with Tacotron 2 found differences from full-context synthesis even with several following words. They do not establish a universal minimum lookahead. [31]

Acoustic conditions and echo control

The microphone captures a mixture. Reverberation consists of delayed room reflections; greater microphone distance can reduce direct speech relative to reflected sound. Background noise and competing speakers introduce other interference. A headset evaluation therefore does not establish room-microphone performance: a reported meeting-transcription comparison found substantial degradation when the recording channel changed. [35][36]

The REVERB report constructs room effects with measured impulse responses: recordings of how a room transforms sound. Its simulated distance comparisons hold signal-to-noise ratio, or SNR, at 20 dB. Across-room comparisons change more than reverberation time. These conditions support separating experimental factors, not a universal recognition penalty per metre. [35]

Acoustic echo cancellation, or AEC, estimates and subtracts loudspeaker audio recaptured by the microphone. It needs the playback reference and its timing relative to capture. Speex's explanation describes an adaptive filter that learns this acoustic path; generated text cannot substitute for the actual reference signal. [37]

Double-talk occurs when local speech and playback echo coexist in capture. Local speech can corrupt echo-filter adaptation. Freezing adaptation means retaining the existing filter, not muting the person or stopping subtraction. Otherwise, an echo-control strategy could suppress exactly the correction the assistant needs to receive. [38]

Clipping is amplitude saturation: the capture path cannot represent larger peaks faithfully. A linear echo filter cannot remove arbitrary nonlinear distortion. Test the complete device path, including speakers; universal headphone use is not a dependable deployment assumption. [37][42]

Concurrent conversation state

Listening, generation, and playback need independent state. Old response R1 may still be playing while new input U2 is captured. Generation may finish before playback finishes. A full-duplex model can process overlapping streams, but the application still needs identities and delivery records to interpret events correctly. [43]

A displayed history may be a copy rather than the model's authoritative context. In the OpenAI workshop implementation, edits required events sent to the session. Changing a screen alone did not change subsequent model input. Streaming delivery and cancellation explains the underlying generation-versus-delivery distinction. [46]

Context precision also depends on placement and capabilities. Pipecat documents its assistant context aggregator after transport output, where it receives text associated with speech progress. Word-level interruption updates require suitable service timing. Neither placement nor timestamps establish that a person attended to or understood the sound. [45]

Turn completion and interruption intent

Voice activity detection, or VAD, estimates whether an interval contains speech. Endpointing closes an input segment. Turn completion decides whether the person is yielding. VAD controls can separate segmentation, response creation, and interruption behavior. [47]

A pause after a completed destination and an equally long pause while recalling one have different conversational meanings. Syntax, meaning, and prosody provide imperfect clues. Voice Activity Projection predicts future activity for both participants, distinguishing patterns associated with holding a turn, shifting it, or acknowledging the speaker. It predicts interaction structure rather than treating silence as proof. [50]

Barge-in is speech during the assistant's answer, usually intended to interrupt or redirect. A backchannel is a brief acknowledgment that may let the answer continue. A short yeah can serve either role. Duration thresholds cannot reliably distinguish them; echo, unrelated speech, and another person's conversation add separate reasons for caution. [51][38]

Explicit stop controls should bypass automatic intent classification. LiveKit's turn handling separates direct interruption, adaptive detection, and false-interruption recovery. Resuming after detected speech produces no transcript is a heuristic, not proof that nothing meaningful occurred. [52]

Cancellation and playback reconciliation

Interruption must stop local playback, discard pending speech, request synthesis and generation cancellation, and prevent obsolete output from returning. Pipecat documents cancellation and queue cleanup across these stages. Those local operations do not acknowledge remote-provider termination. Preserve the new input while cleaning up the old response. [34]

Response identity makes late-result rejection explicit. The following application sketch assumes callbacks and nonblocking local operations run serially on one event loop. Queued chunks retain their response IDs. A threaded implementation needs equivalent synchronization, and remote cancellation needs separate completion tracking. [44][53]

def on_audio(chunk):
    if chunk.response_id != state.active_response:
        return
    output.enqueue_nowait(chunk)


def interrupt():
    old = state.active_response
    state.active_response = None
    player.stop_now()
    output.discard_response(old)
    request_remote_clear(old)
    request_generation_cancel(old)
    record_playback_boundary(old)
    # Input capture and its queue remain intact.

Playback ownership depends on transport. OpenAI Realtime conversations assigns WebSocket clients playback stopping and duration accounting, followed by item truncation. WebRTC and SIP use server-managed buffering and automatic truncation. The documented truncation boundary does not provide a precisely shortened transcript. [54]

Twilio playback marks return after normal playback and after a clear discards buffered audio. Preserve mark order and clear history. A mark outstanding during clear cannot certify complete playback; the associated chunk may have played partially. Keep normally completed earlier marks distinct from this uncertainty. [55]

Repair authoritative context using supported playback progress, not the complete generated answer. If precise alignment is unavailable, retain uncertainty about the suffix. Playback evidence concerns output, not attention or comprehension. This extends inference cancellation to a medium whose already-played content cannot be withdrawn. [54]

Conversational repair and user control

Clarification requests missing or ambiguous information; confirmation checks an interpretation. Explicit confirmation asks the person to verify a destination. Implicit confirmation incorporates it into the next response while leaving room for correction. Once Auckland is unambiguous, replace the destination and retain the origin rather than restarting the conversation. [2]

An apology alone does not update the destination or establish what instructions were delivered. Repeated global confirmation can also increase effort. Repair succeeds when the misunderstanding is resolved and the conversation continues with an appropriate amount of repetition, not merely when the assistant produces a reassuring acknowledgment. [5][32]

Microphone capture, processing, and playback indicators should reflect separate operations. Mute governs input; stop governs output; interrupted work needs its own status. Actual progress and control explains these interface obligations. Text and push-to-talk alternatives should remain usable through changing accessibility states, without assuming either mode suits everyone. [54][49]

Conversational latency boundaries

The response gap runs from actual end of user speech to the first response audio played. It includes capture, transport, processing, and playback effects. Model time to first token measures an intermediate boundary. Serving measurement conventions likewise require explicit start and end events. [57]

First useful content is a separate application boundary: an acknowledgment may precede directions. A brief task announcement can communicate waiting while backend work proceeds, but it does not shorten that work. Likewise, streaming recognition moves work before turn completion, and early playback overlaps synthesis; neither makes total computation disappear. [60][33]

Mouth-to-ear delay follows captured speech through transmission to rendered sound; a conversational response gap additionally concerns producing an answer. Interruption-stop delay needs a declared initiating event—such as explicit stop activation—and an observed cessation boundary. Report within-answer stalls and latency distributions under representative load, alongside nonresponse rates rather than silently excluding unanswered turns. [61][62][59]

Continuous playback and bounded queues

Jitter is variation in arrival timing. A jitter buffer holds audio before playout to absorb that variation. Larger buffers can reduce underruns—periods with no playable audio—but increase delay. The buffer plot isolates this tradeoff with identical arrivals and different playback starts; requested buffering and achieved delay can differ. [1]

A playout deadline is the time audio is needed for output. NetEq discards packets that arrive too late and can conceal missing samples by generating substitute output. Concealment does not recover the original sound. Its adaptive delay and acceleration or slowing of playback also account for arrival variation and clock drift. [63]

Pacing releases audio according to playback needs. Backpressure makes a producer wait when downstream capacity is exhausted. With ten queued chunks of 20 ms each, pending duration is bounded at 200 ms. Variable-duration chunks need explicit duration or byte accounting. A count limit alone does not bound their time or memory. [64]

An awaited enqueue blocks for capacity only if the queue implements that bound. asyncio.Queue defaults to unbounded capacity; the inspected Pipecat FrameQueue inherits that default. Its interruption reset provides cleanup, not continuous backpressure. Bounds elsewhere may still exist, and upstream provider buffers require separate accounting. [64][65]

Media transport evaluates data against playback deadlines. An ordered byte stream provides ordered delivery, leaving the application to decide whether delayed audio remains useful. Buffer growth can preserve every chunk while making speech increasingly stale. Chunk size and synthesis lookahead therefore interact with both continuity and the amount of unwanted output awaiting cancellation. [63][64][31]

Correlated evidence across audio and model events

A trace connects execution records; spans describe timed operations and events mark instants. Observability separates temporal overlap from execution parentage. For voice, correlate input intervals, transcript versions, response identities, output chunks, and playback observations. Audio and tool-context inspection can then distinguish misrecognition from an incorrect downstream decision. [66][67]

A delayed-stop investigation needs evidence from the boundary where sound continued. The following ledger describes a constructed incident with incomplete client coverage. It uses event relationships rather than invented cross-clock durations; the missing playback observation prevents a complete diagnosis. [66][53]

Preserve each timestamp's clock domain, mapping, and uncertainty. Packet source time, server processing time, and device output time are not interchangeable. An absent event may mean missing telemetry. Record evidence coverage explicitly before attributing a delay to the last visible component. [17][66]

Collect recordings and transcripts for defined diagnostic purposes, with limited access and retention. They are separate data products with different downstream copies. Privacy and Data Governance explains minimization; deletion and withdrawal also need to reach retained derivatives rather than only the original recording. [69]

Speech and turn-behavior diagnostics

Here WER is $(1+0+1)/3$, about 66.7%. Report normalization, tokenization, reference segmentation, and scorer settings. They change the evaluated sequences. The destination substitution has a specific task consequence that WER alone does not express. General proxy-metric limits apply even when the arithmetic is correct. [70]

Alignment search and error counting can differ. NIST SCTK's documented default search assigns different costs to substitutions versus insertions and deletions, while ordinary WER counts each resulting error once. Reference alternatives and time-mediated alignment can also change the result. Preserve the scorer version and options alongside the score. [71]

Diarization identifies who spoke when. Joining its intervals to ASR words is not automatically reliable: timestamps can disagree, and overlapping speakers may produce only one recognized word. Assigning the nearest speaker cannot restore omitted speech. [73]

Intelligibility concerns whether listeners understand the spoken content; naturalness concerns how the speech sounds. Mean opinion score, or MOS, averages subjective ratings under a specified question and scale. State listener population, equipment, room conditions, speech material, and uncertainty. P.800 treats quality, listening effort, and loudness as distinct judgments; pleasant sound can still contain wrong words. [74]

Temporal failure and recovery tests

A temporal test supplies timed input and disturbances, then inspects retained input, output admission, rendered audio, and context. Final text alone misses playback races. The WebRTC media test framework demonstrates scheduled network changes, echo emulation, audio dumps, and playback diagnostics; response identity and conversation repair require additional application assertions. [68]

Include bounded non-speech inputs and inspect whether recognized words have acoustic support. A Whisper large-v3 study produced text from constructed non-speech recordings and found incomplete mitigation. Its prevalence depends on that dataset and configuration; it is not an expected failure rate for ordinary calls. [76]

Recorded-event replay fixes selected callbacks; media replay exercises capture or transport processing; fresh model execution introduces new recognition and generation outcomes. Keep these evidence claims separate, as described in Observability. Agent-to-agent persona conversations expand dialogue coverage but do not establish realistic acoustics or human interruption behavior. [68][77]

Useful interaction under realistic conditions

Use the same directions-and-correction task to compare appropriate automatic-conversation, push-to-talk, and text configurations. Keep required information and success criteria fixed. Measure correct destination, completed exchange, successful repair, repetition burden, interruption effectiveness, intelligibility, and subjective experience separately. Shorter dialogue is not automatically better when extra clarification prevents a wrong result. [5]

Recruit intended users and vary the conditions they actually encounter: devices, rooms, connectivity, accents, languages, speaking rates, and relevant speech or hearing differences. Report disaggregated outcomes and untested coverage. Usability and accessibility evaluation concerns specified people completing work; one fluent demonstration cannot establish that scope. [74][78]

Interactive evaluation matters because people adapt their pauses, repetitions, and interruptions to the system. A controlled human–human VoIP study found delay sensitivity depended on task interactivity; it does not supply a universal AI response threshold. Live evaluation complements controlled tests by exposing the behavior of the coupled person-and-system interaction. [61]

Fixed-input Full-Duplex-Bench streams audio and records synchronized responses to assess pauses, backchannels, transitions, and interruptions. It offers reproducible temporal scenarios, but fixed inputs cannot adapt to unexpected replies. Treat it as complementary to interactive sessions, not a substitute for them. [62]

Begin investigation with observable interactions and labels from people who understand the task. Convert consequential failures into regression cases and validate changes offline before reassessing live use. Natural conversation remains insufficient when the system cannot reliably convey correct information or expose what happened during a failure. [79][80][81]

Open questions

1. 01

   Turn policies still need evidence across speaking styles. Short acknowledgments and genuine redirects can share words and duration, making adaptive yielding difficult. Progress would reduce false interruptions without increasing missed stops or excessive waits across explicitly identified language and speaking-rate groups. [50][52]

2. 02

   Partial-playout reconciliation remains limited by acknowledgment and alignment granularity. A cleared chunk may have played in part, yet its exact words remain unknown. Progress would provide trustworthy output offsets and bounded alignment uncertainty while keeping emitted sound distinct from human comprehension. [55][54]

3. 03

   Adaptive synthesis lookahead must balance early speech with faithful pronunciation and phrasing. Future context can change delivery after an earlier chunk has already played. Progress would preserve critical names and numbers while reducing onset delay under a shared listening and content-fidelity protocol. [31][74]

4. 04

   Full-duplex evaluations need stronger links to sustained human use. Fixed-input tests and generated continuations cannot reveal all adaptations, accessibility barriers, or accumulated repair effort. Progress would compare the same tasks interactively across defined populations and report completion, corrections, and effort alongside temporal diagnostics. [62][78]