# Voice Agents: the good, the bad, and the ugly

Eddie Siegel, Fractional AI

AI Engineer Summit 2025

[Talk page](https://ai.engineer/talks/pzmbleiOfCM-voice-agents-good-bad-ugly) · [Video](https://www.youtube.com/watch?v=pzmbleiOfCM)

## Voice Agents: the good, the bad, and the ugly

A consulting interview agent shows why natural voice conversations need explicit navigation, background checks, transcript filtering, and evaluations that go beyond a good prompt.

Familiarity with LLM prompts, tool calling, and application state will help with the implementation discussion.

### What makes a voice conversation work?

[0:02](https://www.youtube.com/watch?v=pzmbleiOfCM&t=2s)

How do you know whether a voice agent is having a good conversation? A plausible response is not enough. It can hallucinate, miss the purpose of a question, or leave the user waiting long enough to break the rhythm. Eddie Siegel, CTO at Fractional AI, starts with this engineering problem: conversational quality is difficult to evaluate, and latency matters when the interface is meant to feel like talking to a person.

Audio adds two more difficulties. Transcription can introduce errors before anyone evaluates the conversation, and the application must handle a continuous stream rather than discrete text exchanges. **The system has to manage both what the conversation means and how it unfolds.**

![Slide titled “Voice models are tough to wrangle,” listing hallucinations, evals, and latency alongside transcription and streaming.](https://ai.engineer/api/talk-essay-media/9406b4a5b454cc668dfa991de7780befec9d5717340fd9a1dfdf7616ac4c6e89)

Voice models add transcription and streaming challenges to familiar LLM problems.

### An interview should reveal more than a form

[1:14](https://www.youtube.com/watch?v=pzmbleiOfCM&t=74s)

The concrete application is qualitative research inside a large company: interviewing employees to understand how they do their jobs, not interviewing candidates for employment. Imagine sending consultants into Coca-Cola to ask employees about their work. Each conversation consumes consultant time, and arranging consecutive interviews creates calendar coordination and scheduling overhead. Coca-Cola is Siegel’s illustrative example, not an identified deployment.

A form removes much of that overhead, but it also changes what people tell you. An interviewer builds trust, decides when to improvise, and asks follow-up questions in response to an unexpected answer. Someone speaking freely may reveal details they would never type into a fixed questionnaire. The agent therefore needs to preserve the freedom to ramble while still gathering the information the interview is meant to collect.

The intended capacity was hundreds of simultaneous interviews, without the scheduling burden of assigning a consultant to each conversation. Automatic transcripts would then support extracting information and aggregating findings across interviews. These are the product goals that make conversational flexibility worth the engineering effort.

### The finished interview experience

[3:23](https://www.youtube.com/watch?v=pzmbleiOfCM&t=203s)

The demonstration uses a short test interview; the backend can configure other interview use cases. The agent first asks for the participant’s name and organizational role. Siegel identifies himself, and the agent moves to his goals for the interview. He explains that he wants to demonstrate its capabilities and discuss what it took to build. The agent paraphrases that goal, then offers either another question or a chance to add more.

![Interview interface with a progress sidebar showing one of two questions completed and a conversation about the participant’s role and demo goals.](https://ai.engineer/api/talk-essay-media/1767ef2cd21018c7dd5b70e82e0dcb433ea1b2a77d655c7240145ae08ee26212)

The interview interface shows two questions, participant replies, and an agent follow-up.

When Siegel says he has nothing more to add, the agent thanks him and closes the conversation. The exchange looks straightforward: ask, listen, acknowledge, move on. Making those transitions reliable is where the implementation becomes more interesting.

### Make question navigation an application event

[4:46](https://www.youtube.com/watch?v=pzmbleiOfCM&t=286s)

The first implementation used the [OpenAI Realtime API](https://openai.com/index/introducing-the-realtime-api/) with a large prompt containing the interview instructions, all the questions, and guidance on navigating between them. That approach became awkward when the team wanted a sidebar roadmap: a list of questions, an indication of the current question, and controls for jumping between them. The application could not reliably tell which question the model was asking, or coax it into the right place after a user clicked elsewhere.

The team changed the contract. Supply only the current question, and give the model a tool for requesting the next one. When the model calls that tool, application code deterministically supplies the next question. **A transition becomes observable because it must pass through the application.** The following TypeScript captures that initial navigation rule using the two questions from the demo:

```typescript
const questions = [
  {
    id: "role",
    text: "What is your name and role in your organization?",
  },
  {
    id: "goals",
    text: "What do you hope to accomplish with this interview?",
  },
] as const;

let currentIndex = 0;

function currentQuestion() {
  return questions[currentIndex] ?? null;
}

function handleNextQuestionRequest() {
  currentIndex = Math.min(currentIndex + 1, questions.length);
  return currentQuestion();
}
```

Here, the tool handler advances stored state and returns either the next question or `null` when none remains. The model requests progression; it does not choose an arbitrary index.

User navigation needs an additional signal. When someone skips ahead or revisits a question, the application injects a prompt explaining why the current question changed. Separate prompts distinguish those cases, allowing the agent to acknowledge that it can return to a skipped topic later or resume something already discussed. This preserves conversational continuity around a transition the user initiated.

### Let the interviewer improvise, but detect rabbit holes

[6:56](https://www.youtube.com/watch?v=pzmbleiOfCM&t=416s)

A next-question tool solves navigation, but it does not ensure the model will use it at the right time. In repeated interviews, the agent kept encouraging the participant, asking follow-ups, and digging into rabbit holes. Stronger instructions to move on reduced that behavior, but also suppressed the improvisation the product needed.

The team introduced a **Drift Detector**: a separate text LLM call, with its own prompt, that examines the conversation history in the background. It checks whether the conversation remains on topic, whether the current question has been answered, and whether it is time to advance. The settings shown in the recording distinguish `CONTINUE`, `OFF_TOPIC`, and `NEXT_QUESTION`.

![“Rabbit holes” slide showing the Drift Detector settings, model field, temperature slider, and system prompt with CONTINUE, OFF_TOPIC, and NEXT_QUESTION outcomes.](https://ai.engineer/api/talk-essay-media/f2e5d9937c64f185dcfaa441dcb289beaa398fee4100f9c5ae45184890ec8318)

The Drift Detector prompt distinguishes continuing, going off topic, and moving to the next question.

When the detector returns a sufficiently strong indication that the interview should move on, the application requires tool use through an OpenAI API setting. That turns an advisory judgment into an enforced transition. The voice agent can improvise during the answer, while a separate evaluator decides when continued exploration has stopped serving the interview.

### Give questions a purpose, then choose the next path

[8:21](https://www.youtube.com/watch?v=pzmbleiOfCM&t=501s)

Controlling drift exposed another limitation. An interviewer can follow up too little or too much, and even a reasonable rephrasing can change what a question is meant to uncover. Supplying one question at a time also limits the model’s awareness of the wider interview. With only a linear next-question action, it tends either to dig deeper into the current topic or ask for whatever comes next.

The interview plan therefore gained **goals and priorities** as explicit fields. The model receives not just the wording of a question, but why it is asking. That purpose guides both rephrasing and follow-up. A separate next question agent adds another decision: what should the interviewer ask next?

The admin example asks, “What are your main responsibilities and daily tasks?” Without further guidance, the model might repeat the question, lightly rephrase it, or decide inconsistently whether an answer is sufficient. For an imaginary interview intended to find opportunities for AI assistance, the plan adds two priorities:

| Priority | Goal |
| --- | --- |
| High | Get a clear picture of regular activities |
| Medium | Identify where AI might be useful |

The higher-priority goal establishes what the interview must learn. The second gives follow-ups a useful direction without replacing the primary purpose.

The next question agent runs over the transcript while the conversation proceeds. Its prompt teaches it how to assess the interview’s direction and choose another path when appropriate. This separates two judgments that initially looked like one: whether to stop exploring the current question, and what to explore after it.

### The transcript is not the audio model’s understanding

[10:51](https://www.youtube.com/watch?v=pzmbleiOfCM&t=651s)

The transcript now has two audiences: the background agents that use it to make decisions, and the participant watching the live conversation. In the implementation Siegel describes, the Realtime API’s core model understands audio directly, while a separate [**Whisper** model](https://github.com/openai/whisper/blob/main/model-card.md) transcribes the user’s speech. Those are different operations. The audio model can identify a clap, cough, or bang on a table; the transcription model tries to turn its input into text.

That distinction produced some conspicuous failures. Siegel reports sending silence and receiving a transcript that appeared to switch languages despite an English conversation history. In another example, background noise with no speech produced “Creeping dippity, dippity, dippity, dippity.” Displaying such text tells the participant that the system heard words they never said.

Siegel reports that the Realtime integration available to this implementation did not expose the transcription controls needed to address these errors, even though working with Whisper at a lower level might offer tuning options. That is a historical constraint: today’s [Realtime transcription guide](https://developers.openai.com/api/docs/guides/realtime-transcription) describes model-dependent controls for context and expected language. The team’s response was to add another agent that examines the full transcript and decides whether a suspected error should be hidden from the user. Its displayed prompt considers grammatical coherence, relevance, speech patterns, and context.

![“Transcript woes” slide with a Transcript filter agent label, robot illustration, and prompt listing four transcript quality criteria.](https://ai.engineer/api/talk-essay-media/f06645a4615419b517a52ce5a48077823e5d4d2c61f0e8898ceca7a4a635295b)

A transcript filter evaluates grammatical coherence, relevance, speech patterns, and context.

**Hiding a transcript block does not correct the transcription.** The system still captures the block; it suppresses its display. Meanwhile, the core audio model can recognize that it did not receive an intelligible answer and ask the participant to repeat it. That makes the visible interaction more coherent without pretending that the stored transcript has become accurate.

### Evaluate the conversation and the fixes

[13:26](https://www.youtube.com/watch?v=pzmbleiOfCM&t=806s)

Each additional agent addressed a concrete failure, but the growing collection of prompts made iteration harder. Initially, the team would conduct a test interview, notice something wrong, and patch the behavior. Once several agents were involved, it became difficult to identify which prompt needed changing or whether the change helped. Fixing one Drift Detector failure could reintroduce a problem from an earlier interview.

The team added an automated evaluation suite that runs over conversations, using an **LLM as judge** with a separately tuned prompt for each attribute:

- **Clarity:** How clear was the conversation?
- **Completeness:** How complete was the conversation?
- **Professionalism:** How professionally did the agent behave?

These checks give prompt changes a more systematic basis than whether the latest manual interview felt good.

The judges still need tuning, and there is no perfect ground truth for a good interview. An ideal evaluation setup would combine objective metrics with historical data against which to compare the current system; this application does not have that combination. The value of the suite is structured measurement that can guide iteration despite that uncertainty.

### Test with a roster of synthetic interviewees

[15:51](https://www.youtube.com/watch?v=pzmbleiOfCM&t=951s)

The next problem is obtaining enough conversations to evaluate before users encounter an annoying behavior or a serious edge case. In Siegel’s hypothetical Coca-Cola rollout, discovering those problems from the client would be too late. Repeatedly talking to the computer and clicking through interviews is also tiring. The team instead uses LLMs to play interviewees in synthetic conversations.

The process is straightforward:

1. Create a persona prompt describing the interviewee. Siegel’s deliberately unusual example is a snarky teenager in charge of a Fortune 500 company.
2. Build a roster that varies the personalities and job functions expected among actual interviewees.
3. Have the interview agent question each persona, with the simulated participant answering in character.
4. Run the same evaluation suite over the resulting conversations and average the metrics across the simulated population.

Those averages estimate behavior across the intended user types; they do not establish real-user quality. The immediate benefit is an automated way to exercise the interviewer against more than the developers’ own conversational habits.

### Make behavior observable enough to improve

[17:18](https://www.youtube.com/watch?v=pzmbleiOfCM&t=1038s)

An API integration and a well-written prompt can take a voice application a long way. In this case, robustness required additional decisions around the live conversation. Separate text agents checked whether the interview remained useful and redirected it when necessary, leaving the voice model to handle the spoken exchange.

Tool use also supplied something beyond behavioral constraints: **instrumentation**. If the model must call a tool to change questions, the application knows when that action happens. The same boundary that limits what the agent can do makes its behavior easier to observe.

Evaluation then gives that visibility a purpose. It measures success, but also helps decide what to change next. Even without an objective source of truth for conversational quality, a consistent evaluation process makes development more deliberate than repeatedly editing prompts until one interview sounds right.

## Resources

- [Introducing the Realtime API](https://openai.com/index/introducing-the-realtime-api/) — OpenAI's original announcement explains the speech-to-speech API and its function-calling capabilities.
- [Whisper model card](https://github.com/openai/whisper/blob/main/model-card.md) — Documents Whisper's intended uses and limitations, including transcription of words that were not spoken.
- [Superintelligent’s AI Interview Agent](https://www.fractional.ai/case-study/automating-enterprise-discovery-superintelligents-ai-interview-agent) — Fractional AI's companion case study describes interview navigation, supporting agents, synthetic evaluations and reported production outcomes.
- [Realtime transcription guide](https://developers.openai.com/api/docs/guides/realtime-transcription) — Current guidance for configuring live transcription, contextual hints, expected languages and latency tradeoffs.

## Slides

### Voice models are tough to wrangle

Voice agents inherit LLM challenges and add transcription and streaming. Evaluating conversations is difficult, and latency can disrupt their fluidity.

[0:14](https://www.youtube.com/watch?v=pzmbleiOfCM&t=14s) [Slide lists hallucinations, evaluations and latency as LLM challenges, and transcription and streaming as additional audio challenges.](https://assets.ai.engineer/sites/aiecode2025/sha256/ee8be5dd0de0aa6e9cdd196cb5c6ad3c3a902470530f79a8aa2565e3ae334695.webp)

Voice models are tough to wrangle

FRACTIONAL

Typical LLM challenges:
• Hallucinations
• Evals
• Latency

Audio is on hard mode:
• Transcription
• Streaming

### Case study: automating consulting-style interviews

Consultants interview employees to understand how a company works. The process takes time and scheduling, but interviewers also need judgment to build trust, improvise, and ask follow-up questions.

[1:31](https://www.youtube.com/watch?v=pzmbleiOfCM&t=91s) [Slide describes consultant-led corporate interviews, scheduling overhead and the human judgment needed for trust and follow-up questions; an interview photo appears at right.](https://assets.ai.engineer/sites/aiecode2025/sha256/e1a4b25bc826c692532f3cf995bc47e7ecfbb2d6211ca68fa96fcd73dd9dd8ba.webp)

Case study: automating “consulting style” interviews

FRACTIONAL

Before:
• Corporate interviews
• In-depth qualitative research done by consultants
• Costly and inefficient - consultant time, scheduling overhead
• Needs the human touch
  ◦ Judgment to improvise, build trust, ask follow-ups
  ◦ Interviewee provides more info when rambling

### Goal: AI interview agent

The proposed agent would make interviews feel like conversations, interview hundreds of people at once without scheduling, and transcribe them for later analysis.

[2:48](https://www.youtube.com/watch?v=pzmbleiOfCM&t=168s) [Slide proposes human-like interviews at scale without scheduling, with automatic transcription for later summarization.](https://assets.ai.engineer/sites/aiecode2025/sha256/ef4de3b7a1f18fb73e6e3cf4b2aa4513e4374faf28560dc8d9a9a3448ec646be.webp)

Goal: AI interview agent

• Conduct interviews like a human would
• Interview hundreds of people at once
• No scheduling
• Automatic transcription for later summarization

FRACTIONAL

### Demonstration Interview

In a two-question demo, Eddie gives his name and role. The progress panel marks the first question complete and selects the second.

[3:29](https://www.youtube.com/watch?v=pzmbleiOfCM&t=209s) [Demo start screen describes a two-question interview and provides microphone-test and start controls.](https://assets.ai.engineer/sites/aiecode2025/sha256/3f9663b5e995bf828554d264b90f51eec392e33f462245d15523f26cc0f0a413.webp)

Demonstration Interview

An automated interview designed to learn a little more about you

What to expect

This interview will consist of 2 questions.

• Talk for as long as you like before submitting your response
• Use the Done Talking button when you're finished talking
• Click on questions to skip to or revisit them
• Your responses will be saved automatically
• You can pause and resume the interview at any time, even if you close your browser

Microphone Test

Interface text:
Test your microphone
Start Interview

[3:42](https://www.youtube.com/watch?v=pzmbleiOfCM&t=222s) [Interview interface showing the first question completed and the second question active.](https://assets.ai.engineer/sites/aiecode2025/sha256/7bdba69903121eff3b0792336b5aca5041cdf868c5095f0d85b0fa32de6ad5cf.webp)

FRACTIONAL

Interview Progress
1 of 2 questions completed

What is your name and role in your organization?
What do you hope to accomplish with this interview?

Interview Agent
Hello! Let's begin. What is your name and role in your organization?

Interface text:
Or type your response instead...
Send
shift + Enter
00:16
Responses saved
You're muted
Done Talking
Skip Question
End Interview

### Interview questions completed

After Eddie identifies himself and explains his demo goal, the interview agent restates that goal and asks whether he has anything to add. The progress panel shows both questions completed.

[4:15](https://www.youtube.com/watch?v=pzmbleiOfCM&t=255s) [Interview interface showing two completed questions and the conversation about the demonstration.](https://assets.ai.engineer/sites/aiecode2025/sha256/774976f443accd87febe9945f5461059c3bfc2a8697d3e431e6a679b3ce6c442.webp)

FRACTIONAL

Interview Progress
2 of 2 questions completed

What is your name and role in your organization?
What do you hope to accomplish with this interview?

Interview Agent
Hello! Let's begin. What is your name and role in your organization?

Hi, I'm Eddie Siegel. I'm the CTO at Fractional.ai.

Interview Agent
Pleasure to meet you, Eddie. As we continue, I'm curious to know what you hope to accomplish with this interview. What are your main goals?

Mainly, I just want to show a quick demo of what your capabilities are so that I can talk about what it took to build you.

Interview Agent
Got it, you'd like to showcase a demo of what I can do and discuss what went into creating me. That's a clear goal. Shall we move on to another question, or is there anything else you'd like to add?

Interface text:
00:49
Saving your response...
Or type your response instead...
Send
shift + Enter
You're muted
Done Talking
Skip Question
End Interview

### Interview conclusion

After both questions are completed, Eddie says he has nothing more to add. The agent closes by acknowledging his role and goal, and the interface shows that the responses are saved.

[4:33](https://www.youtube.com/watch?v=pzmbleiOfCM&t=273s) [Completed interview interface showing the participant's final reply and the agent's closing message.](https://assets.ai.engineer/sites/aiecode2025/sha256/a1ac307c0aaff7d95aa9616f4ac595285430a200c6d1a57fee38a7037ceda655.webp)

FRACTIONAL

Interview Progress
2 of 2 questions completed

What is your name and role in your organization?
What do you hope to accomplish with this interview?

Hi, I'm Eddie Siegel. I'm the CTO at Fractional.ai.

Interview Agent
Pleasure to meet you, Eddie. As we continue, I'm curious to know what you hope to accomplish with this interview. What are your main goals?

Mainly, I just want to show a quick demo of what your capabilities are so that I can talk about what it took to build you.

Interview Agent
Got it, you'd like to showcase a demo of what I can do and discuss what went into creating me. That's a clear goal. Shall we move on to another question, or is there anything else you'd like to add?

No, I think that's all I got for my end, thanks.

Interview Agent
Thank you for taking the time to talk to me, Eddie. It was great hearing about your role and your intentions to showcase a demo of my capabilities. I'm looking forward to being a part of your demonstration.

Interface text:
00:49
Responses saved
Or type your response instead...
Send
shift + Enter
You're muted
Done Talking
Skip Question
End Interview

### Our starting point

The first interview agent used the OpenAI Realtime API with one large prompt containing interview context, questions, and instructions for navigating them.

[4:51](https://www.youtube.com/watch?v=pzmbleiOfCM&t=291s) [Slide describing an initial approach using the OpenAI Realtime API and a prompt containing interview context and questions.](https://assets.ai.engineer/sites/aiecode2025/sha256/e9fb5db2312f3d18937cf8e6c1e643bc34faf495529807602e9b27b3156b88e1.webp)

FRACTIONAL

Our starting point

• OpenAI Realtime API
• Big prompt w/context and interview questions

### Interview roadmap

The roadmap shows the current question and lets users jump around. The agent receives one question at a time, uses a tool to advance, and gets extra prompts when the user changes questions.

[5:16](https://www.youtube.com/watch?v=pzmbleiOfCM&t=316s) [Slide describing interview progress, skipping between questions, tool use, and additional prompting, alongside a progress-panel example.](https://assets.ai.engineer/sites/aiecode2025/sha256/b914e235c122d0efa4eecbe3fe3b3709b2288e62fbf7ee8bea002b52615d1b44.webp)

FRACTIONAL

“Interview roadmap”

• Want to show users what question they’re on
• Allow users to skip around
• One question at a time + tool use
• Additional prompting when user is jumping around

Interview Progress
0 of 11 questions completed

What is your role in the organization?
Which teams or departments do you usually work with?
What are your main responsibilities and daily tasks?
Which parts of your work are most repetitive or time-consuming?
What tasks require the most complex thinking or problem-solving?
How is your success measured, and how do you personally judge your ow[illegible]
Do you use AI at work [illegible]
Have you ever tried using [illegible]

### Case instructions

The next-question tool returns different prompts when a user revisits or skips to a question, or when the agent moves on. The prompts tell the agent to acknowledge prior answers and minimize repetition when revisiting.

[6:31](https://www.youtube.com/watch?v=pzmbleiOfCM&t=391s) [Configuration screen with separate instructions for revisiting a question, skipping to one, and advancing via the next-question tool.](https://assets.ai.engineer/sites/aiecode2025/sha256/b54eb4ec7d926b238bfcddf4296bd90698ad3872b3f81e9c3fcc4be456a927b8.webp)

Case Instructions
These are instructions returned to the AI when calling the next question tool in various scenarios.

Revisit Instructions
User has asked to revisit a question that has already been answered.

The user has asked to revisit this question. Minimize repetition, and acknowledge that we're revisiting it in your first response.

Skip To Instructions
User has asked to skip to a question before the agent decided to move on from a question.

The user asked to skip to this question. Remember to acknowledge parts of it that have already been answered in your first response.

Tool Instructions
The agent decided to use the next question tool on its own.

Remember to acknowledge any parts of this question that have already been answered in your first response.

### Rabbit holes

The interview agent lingered on follow-ups instead of calling the next-question tool. Pushing it too hard to move on limited its ability to improvise.

[6:54](https://www.youtube.com/watch?v=pzmbleiOfCM&t=414s) [Slide describing the interview agent's reluctance to move on and tendency to ask follow-up questions.](https://assets.ai.engineer/sites/aiecode2025/sha256/ee9a8352c459d69dc7f3277c0103ec021eea042bc09e01e9287ec6d0dbf581e9.webp)

Rabbit holes

• Tool use → reluctance to move on
• LLMs love chatting and asking follow-ups!

FRACTIONAL

### Rabbit holes: drift detector

A separate drift detector reviews the transcript to judge whether the interview is on track or ready to move on, helping prevent endless follow-ups.

[7:30](https://www.youtube.com/watch?v=pzmbleiOfCM&t=450s) [Rabbit-holes slide adds a drift detector agent as a response to excessive follow-up questions.](https://assets.ai.engineer/sites/aiecode2025/sha256/b9cd7f570513bb3272f640d80da907ac74311d5b3fb3aa974d983f494217d1ee.webp)

Rabbit holes

• Tool use → reluctance to move on
• LLMs love chatting and asking follow-ups!
• Drift detector agent

FRACTIONAL

### Drift detector configuration

A separate text-based agent checks the interview history for relevance and whether the question has been answered. A strong move-on result can trigger forced tool use to advance to the next question.

[7:48](https://www.youtube.com/watch?v=pzmbleiOfCM&t=468s) [Configuration screen showing the drift detector model, temperature, and prompt for classifying interview progress.](https://assets.ai.engineer/sites/aiecode2025/sha256/308325973c7d691597499cd3656594e8f181b7122c6a90fee771403fda30a26a.webp)

Model
gpt-4o-mini

Temperature (0.3)

System Prompt
You are an interview conversation analyzer. Your job is to determine if a conversation is staying on track with the original interview question.

Respond with one of the following:
- CONTINUE: The conversation is relevant to the interview question. The interviewer should continue asking follow-ups.
- OFF_TOPIC: The question has not yet been answered, but the conversation has drifted off-topic.
- NEXT_QUESTION: The question has been answered. The interviewer should move on to the next question.

Always prefer answering with 'QUESTION_ANSWERED' if the question has been answered, regardless of whether the conversation is [illegible]

Interface text:
Voice
Instructions
Next Question
Transcript
Drift Detector
Save Change[illegible]

### Human-like interviews: hard to tune

Goals and priorities give the interviewer context for rephrasing and follow-ups; a separate next-question agent decides what to ask next.

[9:22](https://www.youtube.com/watch?v=pzmbleiOfCM&t=562s) [Slide describes follow-up and linear-flow challenges, goals and priorities, and an extra next-question agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/6e7b7c9c78d67f2546d553bc38fa41492c2f7af3368b32da67d7651df677d0fd.webp)

Human-like interviews: hard to tune

• Guide model toward effective follow-ups & rephrasing
• Linear flow is restrictive
• First-class goals & priorities
• Extra “next question” agent

FRACTIONAL

### Interview question goals and priorities

In a hypothetical AI-assistance interview, the editor prioritizes understanding regular activities over finding AI opportunities. These goals help guide follow-up questions.

[9:32](https://www.youtube.com/watch?v=pzmbleiOfCM&t=572s) [Question editor with a free-form question and high- and medium-priority goals.](https://assets.ai.engineer/sites/aiecode2025/sha256/df5f2613a5d366dd7cc67fd9922e0e56b1bd3053ef2385926bcd891e20cb185e.webp)

Question 3

What are your main responsibilities and daily tasks?

Question Type
Free Form

Goals
Add Goal

Get a clear picture of regular activities — Low Medium High
Identify where AI might be useful — Low Medium High

### Transcript woes

Silence produced a Japanese transcript; background noise produced a nonsensical English one. The team added a separate agent to flag and hide suspected transcription errors while retaining the transcript.

[11:53](https://www.youtube.com/watch?v=pzmbleiOfCM&t=713s) [Slide contrasts the audio model with transcription and shows an unexpected transcript produced during silence.](https://assets.ai.engineer/sites/aiecode2025/sha256/d8e2d72b0d1e51f669416ad135f2efbcd03bf8800006105e18dc7467b8851130.webp)

FRACTIONAL

Transcript woes

• OpenAI Realtime API uses whisper for transcription
• Core model understands sound, whisper does not

Could you please clarify what you hope to achieve with this interview?
ボタンを押すと弾きやすくなります。

Interface text:
1 minute ago
less than a minute ago

[12:07](https://www.youtube.com/watch?v=pzmbleiOfCM&t=727s) [Slide showing examples of erroneous transcripts produced from silence or background noise.](https://assets.ai.engineer/sites/aiecode2025/sha256/e4e40992dc4e572d6dddfe88f2b4def7af1ef8bef32a8a8d1cea36aee21dbcfc.webp)

FRACTIONAL

Transcript woes

• OpenAI Realti[illegible]
• Core model u[illegible]

What do you hope to accomplish with this interview?
Hey, it's my pleasure, and I'm gonna be a little bit of a monkey. Creeping, dippity, dippity, dippity, dippity, dippity, dippity, dippity, dippity, dippity, dippity, dippity. I swaddle a gala.

Could you please clarify what you ho[illegible]
ボタンを押すと弾きやすくなります。

Interface text:
1 minute ago
1 minute ago
1 minute ago
less than a minute ago

### Transcript filter

A separate agent uses conversation context to spot likely transcription errors and hide them from the user, while still retaining the transcript.

[12:34](https://www.youtube.com/watch?v=pzmbleiOfCM&t=754s) [Transcript filter prompt evaluating coherence, relevance, speech patterns, and context.](https://assets.ai.engineer/sites/aiecode2025/sha256/37380c9e56def2839ecac6d181f010a14e2f24e84e1052343e8cd28a1257c4c7.webp)

Transcript Filter

You are an expert in analyzing audio transcripts for accuracy and coherence.
Your task is to evaluate the quality and likely accuracy of a given transcript based on:
1. Grammatical coherence
2. Semantic relevance to conversation
3. Natural speech patterns
4. Contextual appropriateness

Rate transcripts on a scale of 1-4 where:
1 = Clearly erroneous/nonsensical
2 = Likely to be accurate, but may contain minor errors
[illegible]

Interface text:
Voice
Instructions
Next Question
Transcript
Drift Detector

## Full transcript

[0:02](https://www.youtube.com/watch?v=pzmbleiOfCM&t=2s) Hey, everyone. My name is Eddie Siegel. I'm the CTO at Fractional AI. Uh, and today I'm here to talk to you about AI voice agents and, uh, all the challenges we've run into building with them and ways we've been able to work around those challenges.

[0:15](https://www.youtube.com/watch?v=pzmbleiOfCM&t=15s) So to start, uh, voice models are just generally tough to wrangle. I mean, building with LLMs is tough to begin with, right? They hallucinate. They're generally difficult to evaluate.

[0:24](https://www.youtube.com/watch?v=pzmbleiOfCM&t=24s) Coming up with metrics to know how well your system works is really challenging, especially if you're dealing with something that is conversational in nature, where there's no objective metric to know how well you did or how poorly you did in a particular interaction.

[0:36](https://www.youtube.com/watch?v=pzmbleiOfCM&t=36s) And latency is challenging for many different kinds of applications that leverage LLMs. They're especially, uh, it's an especially important challenge when you're dealing with conversational UIs that are meant to, to be, uh, sort of fluid the way a human conversation would be fluid.

[0:51](https://www.youtube.com/watch?v=pzmbleiOfCM&t=51s) And when we're dealing with audio, uh, and these voice agents, everything's on hard mode. You deal with all these existing, uh, AI problems, plus you're trying to deal with things like transcription, which is not as straightforward as you might think.

[1:04](https://www.youtube.com/watch?v=pzmbleiOfCM&t=64s) You're dealing with a streaming environment instead of the sort of batched back and forth, uh, uh, kind of interaction that you get with text, and this all makes it really, really hard to do development.

[1:14](https://www.youtube.com/watch?v=pzmbleiOfCM&t=74s) So let's go into a case study with an actual application that we built recently so that we can see in practice why the, the sort of theory, uh, was not exactly what we encountered in practice.

[1:24](https://www.youtube.com/watch?v=pzmbleiOfCM&t=84s) Um, so for this case study, we're, we're talking about, uh, automating this sort of consulting style of, of interview. And what I mean by that is like going into big companies and interviewing employees there to learn information.

[1:38](https://www.youtube.com/watch?v=pzmbleiOfCM&t=98s) It's not, not interview as in a job interview, but interview as in, let me go, uh, into some Fortune 500 company and understand how people are doing their jobs.

[1:45](https://www.youtube.com/watch?v=pzmbleiOfCM&t=105s) This is a thing that consultants do all the time. When they're trying to research how something works within a company, they have to go in and do this sort of in-depth qualitative research where they're going and meeting with employees at a company and asking them questions.

[1:57](https://www.youtube.com/watch?v=pzmbleiOfCM&t=117s) It's a pretty expensive process. Uh, you know, if I wanna send a bunch of consultants into Coca-Cola to ask employees about things they're doing, the consultants have to spend a whole bunch of time.

[2:07](https://www.youtube.com/watch?v=pzmbleiOfCM&t=127s) It's very inefficient. You've got people's schedules to line up and do the sort of, you know, how do, how do you cram a calendar full of these interviews back to back?

[2:13](https://www.youtube.com/watch?v=pzmbleiOfCM&t=133s) Lots of overhead, but it's necessary. This is a kind of a difficult thing to automate, right? You might think, why not just send them, uh, you know, a form to fill out or something?

[2:22](https://www.youtube.com/watch?v=pzmbleiOfCM&t=142s) And there's certain cases where you probably can, but in a lot of cases, you need the human touch, right? You need the interviewer to, to be navigating, you know, when do I sort of improvise?

[2:31](https://www.youtube.com/watch?v=pzmbleiOfCM&t=151s) How do I build trust? What follow-up questions do I wanna ask? Um, and also people being interviewed answer differently when they're asked to type out, uh, questions to specific or answers to specific questions.

[2:43](https://www.youtube.com/watch?v=pzmbleiOfCM&t=163s) Whereas like if they're given the freedom to just ramble, they often say more stuff. And so the goal here is let's build an AI interview agent that can replace this function here, where our AI system should conduct interviews just like a human would conduct these interviews.

[2:58](https://www.youtube.com/watch?v=pzmbleiOfCM&t=178s) It should feel less like filling out a form and more like having a conversation. But our system will be able to interview hundreds of people at once and not deal with this sort of scheduling issue and the high cost of sending consultants into a company.

[3:09](https://www.youtube.com/watch?v=pzmbleiOfCM&t=189s) Um, and one nice outcome of the system should be that we get automatic transcription of what happened during the conversation so that later on we can extract whatever data we need to extract, uh, and aggregate it across all these different interviews.

[3:23](https://www.youtube.com/watch?v=pzmbleiOfCM&t=203s) Uh, so let me just dive into a quick demo and show you what we ended up building so you can see what it looks like in practice.

[3:29](https://www.youtube.com/watch?v=pzmbleiOfCM&t=209s) So here I just set up a test interview that's gonna ask me just a, a few basic questions like, you know, who I am and, and what I'm here to do.

[3:35](https://www.youtube.com/watch?v=pzmbleiOfCM&t=215s) But on the back end, this can be configured with all sorts of different sort of use cases that we'd have in mind.

[3:43](https://www.youtube.com/watch?v=pzmbleiOfCM&t=223s) Hello. Let's begin. What is your name and role in your organization?

[3:50](https://www.youtube.com/watch?v=pzmbleiOfCM&t=230s) Hi, I'm Eddie Siegel. I'm the CTO at Fractional AI.

[3:58](https://www.youtube.com/watch?v=pzmbleiOfCM&t=238s) Pleasure to meet you, Eddie. As we continue, I'm curious to know what you hope to accomplish with this interview. What are your main goals?

[4:06](https://www.youtube.com/watch?v=pzmbleiOfCM&t=246s) Mainly, I just want to show a quick demo of what your capabilities are so that I can talk about what it took to build you.

[4:12](https://www.youtube.com/watch?v=pzmbleiOfCM&t=252s) Got it. You'd like to showcase a demo of what I can do and discuss what went into creating me. That's a clear goal. Shall we move on to another question, or is there anything else you'd like to add?

[4:26](https://www.youtube.com/watch?v=pzmbleiOfCM&t=266s) No, I think that's all I got from my end. Thanks.

[4:31](https://www.youtube.com/watch?v=pzmbleiOfCM&t=271s) Thank you for taking the time to talk to me, Eddie. It was great hearing about your role and your intentions to showcase a demo of my capabilities. I'm looking forward to being a part of your demonstration.

[4:46](https://www.youtube.com/watch?v=pzmbleiOfCM&t=286s) So now you see what the final product looks like, we can talk a little bit about how we started building something like this. And when we first started, we figured there's this great OpenAI Realtime API that lets you build, uh, uh, you know, voice agents like this.

[4:59](https://www.youtube.com/watch?v=pzmbleiOfCM&t=299s) Um, and we just did the naive starting thing. We integrated with the OpenAI API. We built a big prompt that explains, you know, you're trying to conduct this kind of interview, and it tells them all these things about the interview, and then puts all the interview questions we might wanna ask into the prompt and tries to explain

[5:15](https://www.youtube.com/watch?v=pzmbleiOfCM&t=315s) how to navigate asking those questions. Um, the problem is, like, we quickly realized that we wanted some additional functionality here, and the f-- one of the early things we realized we wanted was we wanted to show a little roadmap on the side of the screen that, that shows the list of questions you're gonna be asked, which one's

[5:30](https://www.youtube.com/watch?v=pzmbleiOfCM&t=330s) being asked right now, and ideally would let you click on this and jump around from question to question. Um, and this really just was not well suited to the monolithic prompt approach that we had at the beginning because there's really no way to know what question the LLM is currently asking, and it's difficult to sort of coax

[5:47](https://www.youtube.com/watch?v=pzmbleiOfCM&t=347s) it into moving around and, and asking a different question if they do click around. So something we introduced here was, let's just ask one question at a time, put that one question into the prompt that we're sending to the LLM, uh And then introduce some tool use here where we give the LLM access to a tool where

[6:03](https://www.youtube.com/watch?v=pzmbleiOfCM&t=363s) it can say, "I want to move on to the next question now." And when it says it wants to move on, on to the next question, then we'll feed it the next tool, sort of determine-- Oh, sorry, the next question deterministically.

[6:14](https://www.youtube.com/watch?v=pzmbleiOfCM&t=374s) And then we also added some additional prompts that we inject into the stream with the LLM in the case where the user just clicked around in this roadmap so that the LLM can be made aware, uh, the user is, you know, we're moving on to this question because the user clicked next or because the user skipped around.

[6:32](https://www.youtube.com/watch?v=pzmbleiOfCM&t=392s) And so this is what it ends up looking like on our back end. We have different prompts for what happens when a user's revisiting a question, what's happening when a user is skipping to something new, uh, and this enables some really cool behavior.

[6:42](https://www.youtube.com/watch?v=pzmbleiOfCM&t=402s) When we skip around the, uh, the agent can now do things like saying, "Sure, let's move on to, you know, XYZ. We can come back to that later," or, "Let's go revisit that question we were talking about, about A-ABC."

[6:56](https://www.youtube.com/watch?v=pzmbleiOfCM&t=416s) Very quickly, once we started playing with this and actually using this in practice over and over again, we found that the LLM had a real tendency to, to go down these rabbit holes where, you know, now that we gave it this tool to use to say, "I want to move on to the next question," it was hard

[7:10](https://www.youtube.com/watch?v=pzmbleiOfCM&t=430s) to get it to call that tool the appropriate amount of time. LLMs just love chit-chatting and asking follow-up questions and digging down rabbit holes and being very encouraging about things that they hear.

[7:21](https://www.youtube.com/watch?v=pzmbleiOfCM&t=441s) Uh, and so this resulted in sort of a reluctance to move on to the next question. Uh, and if we forced it too hard, then we would eliminate its ability to improvise at all, which is obviously not what we want.

[7:32](https://www.youtube.com/watch?v=pzmbleiOfCM&t=452s) So we introduced at this point an additional background agent that's listening to the conversation and running a separate side thread to decide with a totally separate call to a te-- non-voice text-based LLM based on the transcript right now, do we think we're kind of like off track or on track?

[7:47](https://www.youtube.com/watch?v=pzmbleiOfCM&t=467s) And we call this the Drift detector agent. And so this is like a totally separate prompt, and it's just looking through the history, and it's trying to decide, how are we doing?

[7:56](https://www.youtube.com/watch?v=pzmbleiOfCM&t=476s) Are we on topic? Are we off topic? Is, has the question we're trying to a-ask been answered, and is it time to move on? And when we get back a result that says pretty strongly it's time to move on, we can prevent this sort of rabbit holing from continuing any further by like forcing the tool use and

[8:11](https://www.youtube.com/watch?v=pzmbleiOfCM&t=491s) s-- you know, flipping the flag on the API call to OpenAI to say, "Okay, you have to use this tool now and have to move on to the next question."

[8:21](https://www.youtube.com/watch?v=pzmbleiOfCM&t=501s) Broadly, it's pretty hard to get these human-like interviews tuned into the right place. They're always either like following up too little or following up too much. You want them to be able to rephrase questions, but sometimes they rephrase them in ways that are not conducive to the overall interview.

[8:35](https://www.youtube.com/watch?v=pzmbleiOfCM&t=515s) And because of our approach of, uh, you know, giving it one question at a time, it doesn't always know what the whole interview is gonna be like. Um, and also this linear flow, the only thing I can do is decide to go on to the next question, is pretty restrictive.

[8:49](https://www.youtube.com/watch?v=pzmbleiOfCM&t=529s) It's either digging in really hard into the question it's on, or it's saying, "Okay, give me what's next." Um, and this sort of limited the, the natural flow of the conversation.

[8:59](https://www.youtube.com/watch?v=pzmbleiOfCM&t=539s) And so we introduced some things to try and correct for this even further.

[9:04](https://www.youtube.com/watch?v=pzmbleiOfCM&t=544s) Um, the first is we added goals and priorities as a first-class concept, uh, in the sort of plan for the interview. So rather than just telling the LLM, "Here's this-- the question you're asking right now," we're telling it the why behind the question so that it can be informed when it's trying to rephrase or informed when it's

[9:21](https://www.youtube.com/watch?v=pzmbleiOfCM&t=561s) like trying to decide how to ask follow-up questions. Um, and then we introduced yet another side agent, uh, which we call our next question agent, and this guy is just in charge of figuring out what should we ask next.

[9:33](https://www.youtube.com/watch?v=pzmbleiOfCM&t=573s) And so the first point is the-- this is what the goals look like. This is a screenshot from our admin interface, but, you know, we have this question that it's, that it's gonna ask.

[9:41](https://www.youtube.com/watch?v=pzmbleiOfCM&t=581s) You know, in this case, the question is, what are your main responsibilities and daily tasks? And without the goals, you know, it, it may repeat this question verbatim. It'll probably repha-phrase it ever so slightly, but it's gonna ask something pretty similar.

[9:54](https://www.youtube.com/watch?v=pzmbleiOfCM&t=594s) And then when it comes to digging into follow-up questions, who knows? Like, if it's gonna be satisfied after your answer, like, "Okay, I understand this person's responsibilities and tasks" or not, sometimes it's gonna be satisfied, sometimes it won't.

[10:06](https://www.youtube.com/watch?v=pzmbleiOfCM&t=606s) So instead, we add in these goals, and we say there's a high priority goal o-of we want to get a clear picture of this person's regular activities and a medium priority goal of we'd like to be able to start to suss out where AI might be useful.

[10:18](https://www.youtube.com/watch?v=pzmbleiOfCM&t=618s) This is an imaginary use case where you, um, the reason you're doing these interviews is you're trying to find ways to help with AI. And so this really can like help guide what the follow-up questions will look like.

[10:28](https://www.youtube.com/watch?v=pzmbleiOfCM&t=628s) And then we have this, the tuning of this side, uh, you know, next question bot that's just running on the transcript in the background, uh, as the conversation's going, and it's sort of taught how to be a good interviewer, and is this interviewer on track?

[10:41](https://www.youtube.com/watch?v=pzmbleiOfCM&t=641s) And it has the ability to say, "No," like, "it's time to go, go down this path now," um, a-and, and guide the rest of the conversation.

[10:51](https://www.youtube.com/watch?v=pzmbleiOfCM&t=651s) Um, speaking of transcripts, you know, these, these background bots are running off of the transcript of the conversation, and you also see the transcript live. When you're being interviewed, you don't just see what the model is asking you, you see what you said to the model.

[11:07](https://www.youtube.com/watch?v=pzmbleiOfCM&t=667s) And luckily, OpenAI makes this really easy. When you make API calls, you get back, you know, what, what did the user actually say? You get a transcript kind of, you know, baked into the API response.

[11:17](https://www.youtube.com/watch?v=pzmbleiOfCM&t=677s) But, uh, a challenge in the implementation here is that, uh, the OpenAI Realtime API, it has its core model that's listening and understanding you, but it uses a side model called Whisper that's doing the transcription of what the user said.

[11:32](https://www.youtube.com/watch?v=pzmbleiOfCM&t=692s) That core model is this fully integrated model that's able to think about what it's hearing. It's operating in the sound domain. It-- if you clap or if you cough or if you bang on the table and you ask it what you just did, it'll be able to tell you.

[11:45](https://www.youtube.com/watch?v=pzmbleiOfCM&t=705s) Whisper does not work that way. Whisper is converting everything into text. And so, uh, you know, if you clap, you might get some surprising results. And so, like, this is the result of me just not speaking.

[11:57](https://www.youtube.com/watch?v=pzmbleiOfCM&t=717s) I just sent some silence for a little while. And despite the long conversation history and all the evidence that I was speaking English, it decided, "I guess he's not speaking English anymore."

[12:06](https://www.youtube.com/watch?v=pzmbleiOfCM&t=726s) Um, or another one where there's like a little bit of background noise, and I didn't say anything, uh, [chuckles] this is the transcript that came out. So, like, clearly, I was not saying, "Creeping dippity, dippity, dippity, dippity," but you know, it said that I was.

[12:19](https://www.youtube.com/watch?v=pzmbleiOfCM&t=739s) And this is not a great user experience, and there's no way to tune this. There's no, uh, you know, if we were working with, um, Whisper at a low level, there might be things we can do here.

[12:31](https://www.youtube.com/watch?v=pzmbleiOfCM&t=751s) The OpenAI Realtime API does not give you knobs to really control this. And so to handle this at the UX level, we added an entire separate agent that is taking in the whole context of the conversation as it's happened so far.

[12:45](https://www.youtube.com/watch?v=pzmbleiOfCM&t=765s) Here's the entirety of the transcript, and, uh, its, its entire task is should we hide this piece of the transcript from the user? Do we think we have a transcription error?

[12:54](https://www.youtube.com/watch?v=pzmbleiOfCM&t=774s) Um, and we still capture the transcript in that case, but now we can at least hide, you know, these embarrassing examples so that it just doesn't show up as anything to the user.

[13:02](https://www.youtube.com/watch?v=pzmbleiOfCM&t=782s) And the nice thing about that is if these, these, uh, blocks are hidden, the core model still knows what's going on. So I don't include in these screenshots what the response was, but typically the response was like, you know, "I didn't really get that.

[13:14](https://www.youtube.com/watch?v=pzmbleiOfCM&t=794s) What..." You know, and sort of re-re-ask the question, "What, what do you hope to accomplish with this interview? I didn't hear what you said before." And so hiding the transcript actually helps a lot in these cases for the user experience.

[13:26](https://www.youtube.com/watch?v=pzmbleiOfCM&t=806s) So as you can tell, this is getting kind of complex. Like, we're just adding lots and lots of agents to sort of Band-Aid over different things. We're doing a test interview and seeing how it went and realizing that we've got these issues and then trying to, uh, you know, piecemeal, uh, uh, fix them in this kind of

[13:43](https://www.youtube.com/watch?v=pzmbleiOfCM&t=823s) vibes-driven way, which is not the worst thing for the very beginning of the development cycle, but this thing is starting to get complex, and now we have a lot of different prompts to update.

[13:51](https://www.youtube.com/watch?v=pzmbleiOfCM&t=831s) We have all these side agents. When we have an issue, it's not clear which one to update or if it even worked, and sometimes you're gonna introduce regressions, right?

[13:59](https://www.youtube.com/watch?v=pzmbleiOfCM&t=839s) You perform poorly in one interview setting, and so you update your, you know, uh, um, your Drift detector to sort of handle that case, and maybe you just made yourself worse at the last thing you've tried to fix with that Drift detector.

[14:12](https://www.youtube.com/watch?v=pzmbleiOfCM&t=852s) Um, this is a very challenging situation. It's common to all LLM-based development, but it's extra hard in this domain. We wanna know how well it's working, and the answer is evals.

[14:23](https://www.youtube.com/watch?v=pzmbleiOfCM&t=863s) We need some systematic way to sort of measure, uh, the performance of this system. And so what we ended up coming up with in this case was a set of metrics that, that measure all the different attributes we wanted, uh, to, to see.

[14:36](https://www.youtube.com/watch?v=pzmbleiOfCM&t=876s) Um, and each one of these metrics you're looking at is, you know, something, an automated test suite that we're able to run over a conversation, and then we ask an LLM as judge to measure how clear was this conversation, how complete was this conversation, how professional was the agent, um, all of these different things we see here.

[14:53](https://www.youtube.com/watch?v=pzmbleiOfCM&t=893s) And behind each one of these is a, uh, a prompt that we've tuned to try and do this evaluation. Um, this is really, really useful for getting, uh, more objective.

[15:03](https://www.youtube.com/watch?v=pzmbleiOfCM&t=903s) It's still not completely objective. You know, we don't have a perfect ground truth to measure against here. Um, there's always improvement that we can make in the prompts that are measuring these things, but this is really useful for getting further outside of this sort of purely vibes-driven style of, of iteration and more sort of metrics-driven style of

[15:21](https://www.youtube.com/watch?v=pzmbleiOfCM&t=921s) iteration. Um, as I mentioned, there's no good ground truth here. When you're designing evals for, uh, you know, any system that is gonna do sort of, uh, you know, the sorts of things you'd want to use AI to do, your ideal situation is you've got some objective metrics, you've got some historical data to run, uh, your metrics

[15:43](https://www.youtube.com/watch?v=pzmbleiOfCM&t=943s) and your current system against, and you can know objectively, like, how well are you doing. That clearly doesn't exist here. There's no way to say

[15:51](https://www.youtube.com/watch?v=pzmbleiOfCM&t=951s) with perfect, uh, understanding how well does my system work. But still, we wanna make sure that, like, we don't ship a system that we think is okay and then find out from users in practice, you know, we have a bunch of, uh, these interview agents go out into Coca-Cola, and we find out from Coca-Cola we're annoying all

[16:07](https://www.youtube.com/watch?v=pzmbleiOfCM&t=967s) their users for some reason, or we've got this horrible edge case. Uh, so as we thought about this problem, um, what we ended up introducing was synthetic conversations where we would come up with-- we would use LLMs to fake users and to do fake interviews as the interviewee, and we'd run a whole bunch of these and at

[16:26](https://www.youtube.com/watch?v=pzmbleiOfCM&t=986s) least be able to measure, uh, how well we think we would do against the types of users we want to be able to interview. This also just helps with automation, so that it's not us just clicking around a bunch and taking these interviews.

[16:35](https://www.youtube.com/watch?v=pzmbleiOfCM&t=995s) It gets very tiring to talk to your computer to, to test this thing out. Um, and so the way these synthetic conversations work is we can just create a persona.

[16:44](https://www.youtube.com/watch?v=pzmbleiOfCM&t=1004s) That persona is used as a prompt to the LLM, where we tell it who it is. You know, in this case, I said, "You're a snarky teenager that's in charge of a Fortune 500 company."

[16:52](https://www.youtube.com/watch?v=pzmbleiOfCM&t=1012s) In practice, we've come up with a whole list of the types of people we expect to be interviewing and their different personalities, their different job functions, uh, and we have a whole roster of them.

[17:02](https://www.youtube.com/watch?v=pzmbleiOfCM&t=1022s) And then for each one, we can run it, uh, where the, the agent interviews it, and then it uses that persona to try and answer the questions. Um, and then afterward, we can run that very same eval suite over them, and now we can get average metrics over a broad population of the types of people we'd be,

[17:18](https://www.youtube.com/watch?v=pzmbleiOfCM&t=1038s) uh, we'd be interviewing. So to wrap it up, you know, that was a case study with us building with voice, but I think it highlights in general that this approach of, uh, you know, maybe we just call OpenAI, it's got these nice voice capabilities, now we'll have a nice robust voice app, is kind of wishful thinking.

[17:38](https://www.youtube.com/watch?v=pzmbleiOfCM&t=1058s) Um, you know, the basic kinda call the API, do some prompt engineering, and get it into a good place is very helpful. It gets you a very, uh, uh, uh, gets you very far in the development process, but it's not enough to build you a robust app.

[17:52](https://www.youtube.com/watch?v=pzmbleiOfCM&t=1072s) As you're trying to build that more robust app, some really helpful additions are adding some out-of-band checks, where you've got separate agents operating in the text domain instead of the audio domain, trying to make decisions about whether the thing you're trying to do is, is working properly, um, trying to get the, the overall thing back on track.

[18:09](https://www.youtube.com/watch?v=pzmbleiOfCM&t=1089s) And tool use can be very, um, powerful for constraining behavior and for getting, uh, sort of, uh, as like sort of a hack to instrument the LLM and understand what's going on.

[18:20](https://www.youtube.com/watch?v=pzmbleiOfCM&t=1100s) Instead of letting it just sort of do whatever it wants, it's gotta call your tools in order to do what it wants, and so now your tools know what's happening.

[18:27](https://www.youtube.com/watch?v=pzmbleiOfCM&t=1107s) Um, and then lastly, as with all LLM-based projects, evals are critical to measure success. Uh, they're also critical to guide development. It's, it's hard to know what to do without some metric.

[18:36](https://www.youtube.com/watch?v=pzmbleiOfCM&t=1116s) And even in this domain where you've got no objective source of truth, there are still ways to harness evals, uh, to get a really robust development process.
