Voice Agents: the good, the bad, and the ugly

How do you know whether a voice agent is having a good conversation? A plausible response is not enough. It can hallucinate, miss the purpose of a question, or leave the user waiting long enough to break the rhythm. Eddie Siegel, CTO at Fractional AI, starts with this engineering problem: conversational quality is difficult to evaluate, and latency matters when the interface is meant to feel like talking to a person.

Audio adds two more difficulties. Transcription can introduce errors before anyone evaluates the conversation, and the application must handle a continuous stream rather than discrete text exchanges. The system has to manage both what the conversation means and how it unfolds.

The concrete application is qualitative research inside a large company: interviewing employees to understand how they do their jobs, not interviewing candidates for employment. Imagine sending consultants into Coca-Cola to ask employees about their work. Each conversation consumes consultant time, and arranging consecutive interviews creates calendar coordination and scheduling overhead. Coca-Cola is Siegel’s illustrative example, not an identified deployment.

A form removes much of that overhead, but it also changes what people tell you. An interviewer builds trust, decides when to improvise, and asks follow-up questions in response to an unexpected answer. Someone speaking freely may reveal details they would never type into a fixed questionnaire. The agent therefore needs to preserve the freedom to ramble while still gathering the information the interview is meant to collect.

The intended capacity was hundreds of simultaneous interviews, without the scheduling burden of assigning a consultant to each conversation. Automatic transcripts would then support extracting information and aggregating findings across interviews. These are the product goals that make conversational flexibility worth the engineering effort.

The demonstration uses a short test interview; the backend can configure other interview use cases. The agent first asks for the participant’s name and organizational role. Siegel identifies himself, and the agent moves to his goals for the interview. He explains that he wants to demonstrate its capabilities and discuss what it took to build. The agent paraphrases that goal, then offers either another question or a chance to add more.

When Siegel says he has nothing more to add, the agent thanks him and closes the conversation. The exchange looks straightforward: ask, listen, acknowledge, move on. Making those transitions reliable is where the implementation becomes more interesting.

The first implementation used the OpenAI Realtime API with a large prompt containing the interview instructions, all the questions, and guidance on navigating between them. That approach became awkward when the team wanted a sidebar roadmap: a list of questions, an indication of the current question, and controls for jumping between them. The application could not reliably tell which question the model was asking, or coax it into the right place after a user clicked elsewhere.

The team changed the contract. Supply only the current question, and give the model a tool for requesting the next one. When the model calls that tool, application code deterministically supplies the next question. A transition becomes observable because it must pass through the application. The following TypeScript captures that initial navigation rule using the two questions from the demo:

const questions = [
  {
    id: "role",
    text: "What is your name and role in your organization?",
  },
  {
    id: "goals",
    text: "What do you hope to accomplish with this interview?",
  },
] as const;

let currentIndex = 0;

function currentQuestion() {
  return questions[currentIndex] ?? null;
}

function handleNextQuestionRequest() {
  currentIndex = Math.min(currentIndex + 1, questions.length);
  return currentQuestion();
}

Here, the tool handler advances stored state and returns either the next question or null when none remains. The model requests progression; it does not choose an arbitrary index.

User navigation needs an additional signal. When someone skips ahead or revisits a question, the application injects a prompt explaining why the current question changed. Separate prompts distinguish those cases, allowing the agent to acknowledge that it can return to a skipped topic later or resume something already discussed. This preserves conversational continuity around a transition the user initiated.

A next-question tool solves navigation, but it does not ensure the model will use it at the right time. In repeated interviews, the agent kept encouraging the participant, asking follow-ups, and digging into rabbit holes. Stronger instructions to move on reduced that behavior, but also suppressed the improvisation the product needed.

The team introduced a Drift Detector: a separate text LLM call, with its own prompt, that examines the conversation history in the background. It checks whether the conversation remains on topic, whether the current question has been answered, and whether it is time to advance. The settings shown in the recording distinguish CONTINUE, OFF_TOPIC, and NEXT_QUESTION.

When the detector returns a sufficiently strong indication that the interview should move on, the application requires tool use through an OpenAI API setting. That turns an advisory judgment into an enforced transition. The voice agent can improvise during the answer, while a separate evaluator decides when continued exploration has stopped serving the interview.

Controlling drift exposed another limitation. An interviewer can follow up too little or too much, and even a reasonable rephrasing can change what a question is meant to uncover. Supplying one question at a time also limits the model’s awareness of the wider interview. With only a linear next-question action, it tends either to dig deeper into the current topic or ask for whatever comes next.

The interview plan therefore gained goals and priorities as explicit fields. The model receives not just the wording of a question, but why it is asking. That purpose guides both rephrasing and follow-up. A separate next question agent adds another decision: what should the interviewer ask next?

The admin example asks, “What are your main responsibilities and daily tasks?” Without further guidance, the model might repeat the question, lightly rephrase it, or decide inconsistently whether an answer is sufficient. For an imaginary interview intended to find opportunities for AI assistance, the plan adds two priorities:

The higher-priority goal establishes what the interview must learn. The second gives follow-ups a useful direction without replacing the primary purpose.

The next question agent runs over the transcript while the conversation proceeds. Its prompt teaches it how to assess the interview’s direction and choose another path when appropriate. This separates two judgments that initially looked like one: whether to stop exploring the current question, and what to explore after it.

The transcript now has two audiences: the background agents that use it to make decisions, and the participant watching the live conversation. In the implementation Siegel describes, the Realtime API’s core model understands audio directly, while a separate Whisper model transcribes the user’s speech. Those are different operations. The audio model can identify a clap, cough, or bang on a table; the transcription model tries to turn its input into text.

That distinction produced some conspicuous failures. Siegel reports sending silence and receiving a transcript that appeared to switch languages despite an English conversation history. In another example, background noise with no speech produced “Creeping dippity, dippity, dippity, dippity.” Displaying such text tells the participant that the system heard words they never said.

Siegel reports that the Realtime integration available to this implementation did not expose the transcription controls needed to address these errors, even though working with Whisper at a lower level might offer tuning options. That is a historical constraint: today’s Realtime transcription guide describes model-dependent controls for context and expected language. The team’s response was to add another agent that examines the full transcript and decides whether a suspected error should be hidden from the user. Its displayed prompt considers grammatical coherence, relevance, speech patterns, and context.

Hiding a transcript block does not correct the transcription. The system still captures the block; it suppresses its display. Meanwhile, the core audio model can recognize that it did not receive an intelligible answer and ask the participant to repeat it. That makes the visible interaction more coherent without pretending that the stored transcript has become accurate.

Each additional agent addressed a concrete failure, but the growing collection of prompts made iteration harder. Initially, the team would conduct a test interview, notice something wrong, and patch the behavior. Once several agents were involved, it became difficult to identify which prompt needed changing or whether the change helped. Fixing one Drift Detector failure could reintroduce a problem from an earlier interview.

The team added an automated evaluation suite that runs over conversations, using an LLM as judge with a separately tuned prompt for each attribute:

• Clarity: How clear was the conversation?
• Completeness: How complete was the conversation?
• Professionalism: How professionally did the agent behave?

These checks give prompt changes a more systematic basis than whether the latest manual interview felt good.

The judges still need tuning, and there is no perfect ground truth for a good interview. An ideal evaluation setup would combine objective metrics with historical data against which to compare the current system; this application does not have that combination. The value of the suite is structured measurement that can guide iteration despite that uncertainty.

The next problem is obtaining enough conversations to evaluate before users encounter an annoying behavior or a serious edge case. In Siegel’s hypothetical Coca-Cola rollout, discovering those problems from the client would be too late. Repeatedly talking to the computer and clicking through interviews is also tiring. The team instead uses LLMs to play interviewees in synthetic conversations.

The process is straightforward:

1. Create a persona prompt describing the interviewee. Siegel’s deliberately unusual example is a snarky teenager in charge of a Fortune 500 company.
2. Build a roster that varies the personalities and job functions expected among actual interviewees.
3. Have the interview agent question each persona, with the simulated participant answering in character.
4. Run the same evaluation suite over the resulting conversations and average the metrics across the simulated population.

Those averages estimate behavior across the intended user types; they do not establish real-user quality. The immediate benefit is an automated way to exercise the interviewer against more than the developers’ own conversational habits.

An API integration and a well-written prompt can take a voice application a long way. In this case, robustness required additional decisions around the live conversation. Separate text agents checked whether the interview remained useful and redirected it when necessary, leaving the voice model to handle the spoken exchange.

Tool use also supplied something beyond behavioral constraints: instrumentation. If the model must call a tool to change questions, the application knows when that action happens. The same boundary that limits what the agent can do makes its behavior easier to observe.

Evaluation then gives that visibility a purpose. It measures success, but also helps decide what to change next. Even without an objective source of truth for conversational quality, a consistent evaluation process makes development more deliberate than repeatedly editing prompts until one interview sounds right.