"My name is... my name is...": A Linguistic Map for Voice Agents — Midam Kim, ServiceNow

Midam Kim, an ML engineer at ServiceNow and a researcher of speech communication in the wild, starts with a support call that deteriorates one failure at a time. The voice agent asks her to spell her first name. She supplies M-I-D-A-M slowly; it confirms M-I-D-A-N. She corrects the final letter, and the agent thanks her—then still pronounces her name incorrectly. The confirmation language sounds successful, but the spoken result shows that the correction did not survive the trip through the system.

The next request compounds the problem. The agent asks for an account number without making clear which identifier it means. Kim searches for the unfamiliar string and begins reading it carefully. Because she needs extra time, the agent cuts her off, reports that it cannot find the record, and asks her to repeat the same information. Nothing about the second attempt has changed: no narrower question, no partial confirmation, no alternative input method. She asks for a person.

This is one observable failure—a caller abandons the automated path—but it has several causes. A letter was misrecognized, a name was mispronounced, the requested identifier was unclear, turn detection fired too early, and the repair strategy ignored what had just happened. Treating the episode as one generic “voice AI bug” would hide the engineering decisions needed to fix it. Kim’s proposed diagnostic lens is linguistics: separate the mechanisms, then inspect how they interact across the whole call.

Human conversation is a joint activity. One participant produces sounds and words; the other listens and responds; the exchange moves back and forth through interaction. Meanwhile, both participants continuously update their mental models: what the other person means, what has already been established, and what should happen next. People bring those expectations to a voice agent even if its implementation is organized as a sequence of models and services.

Replaying the call through that lens separates listening from speaking. Confusing M with N belongs to the agent’s listening side: the recognized signal does not preserve the supplied spelling. Incorrectly saying the resulting name belongs to its speaking side: the text-to-speech system applies English-centric pronunciation rules that do not fit the name. Fixing recognition alone would therefore leave the pronunciation problem intact.

The account-number exchange crosses even more layers. Kim adapts to an unclear request by searching for the identifier and reading it slowly. The agent fails to recognize the relevant unit, decides that her turn is over, and speaks over her. When it requests an unchanged repetition, it also fails at conversational repair. A human might confirm a prefix, ask for one character at a time, explain which identifier is needed, or offer another channel. The agent instead resets the burden onto the caller, making escalation the rational choice.

What question does the linguistic map answer? It shows where a failure occurs by crossing two channels—listening and speaking—with four levels: sounds, words, interaction, and mental model. The resulting eight cells distinguish responsibilities that are often collapsed into “the voice stack.” The relationship made visible is symmetry: receiving speech and producing speech each require acoustic competence, understandable language, appropriate timing, and an adequate model of the user’s goal.

On the listening side:

• Sounds — recognition: Does the agent recognize the user’s speech correctly?
• Words — understanding: Does it understand the words the user supplied?
• Interaction — turn detection: Does it wait until the user has finished?
• Mental model — intention: Does it understand what the user is trying to accomplish?

On the speaking side:

• Sounds — pronunciation: Does the synthesized speech say names and other content appropriately?
• Words — vocabulary: Does the agent choose language the user can understand?
• Interaction — turn taking: Does it speak at the right time?
• Mental model — useful information: Does it provide what the user actually needs next?

The grid is diagnostic, not a claim that its cells can be optimized independently. Sound-level work has to account for words: recognizing individual phonetic material is not enough if the assembled identifier is wrong. Sounds and words must then align with interaction: even accurate transcription cannot help if endpointing cuts the user off before the unit is complete. Task completion at the mental-model layer depends on all of them working together.

The name example makes the dependency concrete. First, recognition changes the final letter from M to N. After correction, the speaking path still applies unsuitable pronunciation rules. The user then enters the account-number task already carrying evidence that the agent may not handle her input reliably. Premature turn detection adds another failure, and the unchanged repetition request confirms that the agent has not learned from the exchange. Each event updates the interpretation of the next one.

What survives this sequence after the audio is gone? The timeline below separates transient signals from persistent understanding. Unlike chat, which leaves a visible text history, spoken sounds vanish as air vibrations. What accumulates for the caller is a mental model: whether the agent listens, whether corrections matter, whether waiting is safe, and whether another attempt is worth making. That accumulating interpretation is the relationship the visual makes visible—and the experience the system ultimately has to improve.

The map turns into several parallel areas of work rather than one preferred model choice:

• Recognition: Select and configure automatic speech recognition, then add appropriate post-processing.
• Speech generation: Select and configure text-to-speech, with preprocessing for what the system must say.
• Shared vocabulary: Curate words and expressions that both the agent and its users can understand.
• Interaction: Manage turn detection, latency, and turn taking together.
• Conversation state: Retain context across all these layers, and detect and handle changes in the user’s emotional state.

The orchestration must also change during the call. A child and an adult may have different timing, vocabulary, and clarification needs. The same person may slow down while reading an unfamiliar identifier, become frustrated after an interruption, or stop trusting confirmations after a failed correction. Static settings chosen before the call cannot fully account for those changes; the agent has to manage them along the timeline, including when the user is unhappy.

This is why “voice is natural” can be misleading as an engineering description. Speech feels natural to people because humans perform a great deal of linguistic coordination automatically. A voice agent has to recreate enough of that coordination across recognition, generation, timing, context, and repair. When those mechanisms disagree, the failure is experienced as one conversation rather than a set of isolated component errors.

Kim connects this work to user frustration, task failure, live-agent escalation, abandoned calls, and silent failures. The talk does not report measured reductions for those outcomes. She also points to ServiceNow’s EVA Bench as an end-to-end diagnostic benchmark, but does not describe its evaluation procedure or results here; its role in the talk is as a possible way to assess a voice agent’s current state.

The closing distinction is practical: voice AI is a joint activity between the user and the agent, not merely a pipeline that transforms audio into text and back again. Needs must be served across multiple layers in real time. There is no universal configuration that repairs every agent; the map helps a team locate its own failure, understand adjacent dependencies, and decide what to change. Kim’s next steps are direct: evaluate the system, learn linguistics, and involve linguists in its design.

The longer-term challenge is adaptation. During any conversation, people learn one another’s accent, vocabulary, rhythm, and preferred ways of explaining things. Users will likewise adapt to a voice agent over the course of a call. A system should ask whether that learning can make the next exchange easier rather than forcing the user to rediscover the same limitations each time.

Language itself also moves. Pronunciations, vocabulary, and expectations can change over a year—or even six months. A voice agent that performs acceptably today therefore needs mechanisms for reevaluation and revision. The useful map is not a finished checklist. It is a way to keep finding where a changing agent, a changing user, and a changing language have fallen out of alignment.