# Your Voice Agent Doesn't Need a Frontier Model

Joel Allou · Ornella Bahidika

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/fnLBmfsI_Fg-your-voice-agent-doesnt-need-frontier-model) · [Video](https://www.youtube.com/watch?v=fnLBmfsI_Fg)

## Your Voice Agent Doesn’t Need a Frontier Model

Ace keeps its voice tutor responsive by moving lesson state, mastery tracking, and next-step decisions into code, leaving a small model to turn those decisions into speech.

### The pause is the problem

[0:00](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=0s)

A student asks a voice tutor a question. The tutor goes silent. Is it thinking, or has the app stopped working? For Ornella Bahidika and Joel Allou, who built the live AI voice tutor Ace, that ambiguity is the opening design problem. Bahidika describes even a one-second pause as enough to make the interaction feel broken.

![Slide reading “A few seconds pause feels broken,” with two waveform sections separated by a gap labeled “dead air · 3.0s.”](https://ai.engineer/api/talk-essay-media/0b753cc4c58621a596d9e00318d3c6d7cf5c8247a64c7afa670c13eed4978b22)

A pause in a voice conversation feels broken.

When an answer feels wrong, reaching for a smarter, larger model is an understandable response. But a voice application also has to preserve the rhythm of conversation. **Ace’s target is to start talking in about 950 milliseconds.** In that budget, a model that spends a full second thinking has already waited too long, however good its eventual answer. The target is Ace’s design constraint, rather than a universal threshold for conversation.

### Give the model a turn summary, not the whole lesson

[1:04](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=64s)

Ace makes a small model viable by taking responsibilities away from it. The surrounding system tracks what happened in the lesson, what the student knows, and what should happen next. Every turn, it hands the model a summary. The model’s remaining job is to express the prepared result conversationally.

Allou uses Anthropic’s **[Claude Opus 4.7](https://www.anthropic.com/news/claude-opus-4-7)** as the frontier-model example, initially calling it Claude 4.7. Such a model can reason through a lesson, interpret a student’s question, and work out an answer. But doing that reasoning during the conversational turn can consume the seconds the voice interface cannot spare. Ace moves that work outside the speaking model.

A **state machine** encodes anticipated lesson scenarios and coordinates transitions between steps. An additional intelligent layer derives the student’s mastery and helps determine whether the lesson is complete. Together, these components decide the next action, the content to display, and how a question should be answered. Their output goes to the model to verbalize. This separation makes the speaking model a consumer of lesson decisions instead of the owner of lesson progression.

### The same question, two different systems

[3:07](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=187s)

The recorded comparison starts with Opus 4.7 without Ace’s implementation. A simple question leaves the model visibly thinking before returning an answer. The second example asks the same question using the smaller **[Haiku 4.5](https://www.anthropic.com/news/claude-haiku-4-5)** with the surrounding architecture in place.

| Configuration | Reported response |
| --- | --- |
| Opus 4.7 without Ace’s scaffolding | Allou describes a couple of seconds to return an answer. |
| Haiku 4.5 with Ace’s scaffolding | Allou reports an answer in about 900 milliseconds. |

Both the model and the scaffolding change, so the comparison illustrates the combined design rather than isolating either contribution. The recording does not specify the timing boundaries or repeated-trial conditions; the reported response time should not automatically be read as time to first text token or first audible speech.

Allou attributes the improvement to moving logic and reasoning into code before the speaking model is called. He also describes the smaller model as cost effective, without supplying cost figures. The visible tutor result shows a lesson titled “The Eight Parts of Speech,” alongside chat messages and a developer panel. The interaction feels almost instant to him because the decisions needed for the response have already been made.

![Slide headed “It feels instant” beside a tutor interface showing “The Eight Parts of Speech,” chat messages, and a developer panel.](https://ai.engineer/api/talk-essay-media/60651e6e78d152488007a067bcd35636bb19c423bb90b2722d1acc7daa20bcb3)

The tutor demo displays “The Eight Parts of Speech.”

### Pay for structure in code

[4:15](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=255s)

The smaller model creates its own engineering obligation. Allou reports that Haiku 4.5 without scaffolding tends to drift over long structures and needs strict rules to stay organized. **Scaffolding is the price of narrowing the model’s job.** He frames that price as development work paid in code, rather than reasoning performed afresh on every conversational turn. His selection rule is to choose the fastest model the latency budget allows, then spend the remaining effort on scaffolding.

![Dark slide with the model-selection rule in white and “Spend the rest on scaffolding” highlighted in orange.](https://ai.engineer/api/talk-essay-media/898dc52f530a2531b90dce59880b3778f032455f3c91786fb5059ec26d9f4b76)

Pick the fastest model your latency allows. Spend the rest on scaffolding.

That work means enumerating scenarios, defining transitions, and deciding how each situation should be handled. A small TypeScript teaching example can make the boundary concrete. Suppose a parts-of-speech lesson asks the student to identify a noun. An upstream evaluator supplies `correct`; the transition function updates lesson state and prepares the speaking instruction:

```typescript
type LessonState = {
  step: "noun-practice" | "verb-practice";
  nounMastered: boolean;
};

type Evaluation = {
  answer: string;
  correct: boolean;
};

function handleNounAnswer(
  state: LessonState,
  evaluation: Evaluation,
) {
  if (state.step !== "noun-practice") {
    throw new Error("Expected noun practice");
  }

  const next: LessonState = evaluation.correct
    ? { step: "verb-practice", nounMastered: true }
    : { ...state };

  return {
    state: next,
    display: evaluation.correct ? "Verbs" : "Nouns",
    speakingSummary: {
      studentAnswer: evaluation.answer,
      correct: evaluation.correct,
      instruction: evaluation.correct
        ? "Confirm the noun answer, then introduce verbs."
        : "Explain that a noun names a person, place, or thing; ask again.",
    },
  };
}

const preparedTurn = handleNounAnswer(
  { step: "noun-practice", nounMastered: false },
  { answer: "cat", correct: true },
);
```

Here, the example’s deliberately simple mastery rule advances after one correct answer. Ace’s actual mastery algorithm is not specified. The architectural point is the handoff: `preparedTurn` contains the next state, display choice, and speaking summary before any language generation occurs. The model receives the prepared instruction; it does not choose the transition or assign mastery.

### A small model inside a larger system

[5:16](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=316s)

Allou extends the pattern beyond voice tutoring to other real-time applications where latency matters, and to high-volume workloads where repeated reasoning carries a recurring cost. The surrounding system owns the durable structure: state, scenarios, decisions, and progression. The model supplies the language at the point where language is needed. In that architecture, the model can become the smallest part of the system while the application remains capable of guiding a complete interaction.

## Resources

- [Introducing Claude Opus 4.7](https://www.anthropic.com/news/claude-opus-4-7) — Anthropic's launch announcement for Opus 4.7, covering capabilities, availability, pricing, and reasoning-effort controls.
- [Introducing Claude Haiku 4.5](https://www.anthropic.com/news/claude-haiku-4-5) — Anthropic's introduction to its small Haiku 4.5 model, including low-latency use cases and launch pricing.
- [Measuring voice-agent latency with LiveKit](https://docs.livekit.io/deploy/observability/data/) — Metric definitions and instrumentation examples for separating model, speech synthesis, and end-to-end conversational latency.

## Slides

### Your voice agent doesn't need a frontier model

Ornella Bahidika and Joel Allou built Ace, a live AI voice tutor that deliberately runs on a small model.

[0:00](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=0s) [Title slide introducing a talk about using a smaller model for a voice agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/13fe063baab59362708aca02166ffe137ea906c7d830a4c573acf799d58daef3.webp)

Ace • AI ENGINEER · LIGHTNING TALK

Your voice agent
doesn't need a
frontier model.

Joel Allou
ENGINEERING

Ornella Bahidika
PRODUCT

Interface text:
Ornella Bahidika
Joel Allou

### A few seconds' pause feels broken

A waveform breaks for 3.0 seconds of dead air. Ornella says even a one-second pause can make a voice agent feel like a broken app.

[0:12](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=12s) [A waveform has a gap labeled three seconds of dead air.](https://assets.ai.engineer/sites/aiecode2025/sha256/531a35fb40132505564b2409f79f7945733c49392ead564a9b6e107379e0c718.webp)

A few seconds pause feels broken.

Ace

// dead air · 3.0s

Voice has no spinner.

Interface text:
Ornella Bahidika
Joel Allou

### The bigger-model instinct

For voice, the instinct to use a bigger model is backwards: a model that pauses for a full second can lose the room, however good its answer.

[0:29](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=29s) [The phrase 'bigger model → better' is crossed out; the slide says that instinct is backwards for voice.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d0022eac9efa513eda1ee9740e50142702950d7303f2c1c1a8df9125e1d3255.webp)

Ace

THE INSTINCT
bigger model → better

In voice, that's exactly backwards.

Interface text:
Ornella Bahidika
Joel Allou

### Approximately 950 milliseconds

The slide sets a roughly 950 ms time-to-first-token budget. The speaker says a second of silence can make a voice tutor feel broken.

[0:42](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=42s) [The slide shows approximately 950 milliseconds to first token and emphasizes a budget measured in milliseconds.](https://assets.ai.engineer/sites/aiecode2025/sha256/50b603e0a0837e2155260a92356db9a46558fcc5265eb1d99471d2cdc8051465.webp)

Ace

≈ 950 ms
TIME TO FIRST TOKEN

Our budget was never IQ. It's milliseconds.

Interface text:
Ornella Bahidika
Joel Allou

### Take the hard jobs out of the model

Ace uses deterministic code for lesson flow, mastery tracking and planning, then sends a structured brief each turn to a small, fast model that speaks.

[1:02](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=62s) [Control flow, mastery tracking and planning feed a structured brief to a small, fast model that produces voice output.](https://assets.ai.engineer/sites/aiecode2025/sha256/be7b06dcc6f28e187043b8514b508c6065246ec0ae46620f93ff1c506edb339e.webp)

Ace

Take the hard jobs out of the model.

Control flow
what happens next
state machine

What they know
mastery tracking
BKT + FSRS

What's next
planning
prereq scheduler

DETERMINISTIC CODE
↓ ↓ ↓
structured brief, every turn

The model
just talks · small + fast LLM
↓
voice out · ≈150ms to first token

Interface text:
Ornella Bahidika
Joel Allou

### Grammar Foundations demo

In the baseline Opus 4.7 lesson, the student asks what they’re learning. The console shows TTFT readings of 1,992 and 2,444 ms.

[3:13](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=193s) [A grammar lesson canvas beside a chat asking what the lesson will cover.](https://assets.ai.engineer/sites/aiecode2025/sha256/2c10eaf2b6932c8b0aad0dab0cbf781f5b04ebd41d2f089c0933edae61a4fd34.webp)

Grammar Foundations

Hey Jordan, glad you're here. Let's build the grammar toolkit you'll use on every

Hey. What are we learning today?

Interface text:
DAY 2 OF 150 · ENGLISH
English Foundations: Parts of Speech, Sentence Structure, and Verb Tense
CH. GRAMMAR FOUND... 0/6
LIVE
END
TTFT
4 Issues
use-livekit-lesson.ts:373
[TTFT] 1992 ms (model=claude-opus-4-7)
use-livekit-lesson.ts:373
[TTFT] 2444 ms (model=claude-opus-4-7)
Type a question
Listening
100%

### The Eight Parts of Speech demo

The presenter reports an answer in about 900 milliseconds with Haiku 4.5 after moving reasoning and logic into code.

[3:37](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=217s) [A lesson canvas titled The Eight Parts of Speech beside a chat conversation.](https://assets.ai.engineer/sites/aiecode2025/sha256/c8c1d6109aa63c1555830ead33042360e9d3c88fd07e64e2bd2f73b942c4f151.webp)

The Eight Parts of Speech

Hey Jordan, great to see you again.

Today we're building your grammar toolkit for the A C

Hey. What are we learning today?

Perfect timing. We're diving into the

Interface text:
DAY 2 OF 150 · ENGLISH
English Foundations: Parts of Speech, Sentence Structure, and Verb Tense
CH. ENGLISH FOUNDA... 0/6
TEACHING
END
Type a question
Pax is speaking...
100%
ttft
Default levels
4 Issues
[TTFT] 1105 ms (model=[illegible])
[TTFT] 939 ms (model=[illegible])
[TTFT] 903 ms (model=[illegible])

### Small model without scaffolding

Small models can drift over long structures and need strict rules to stay organized. Scaffolding puts those rules in code, a cost paid once rather than on every turn.

[4:14](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=254s) [A slide says a small model without scaffolding drifts on long structure and needs strict rules to stay organized.](https://assets.ai.engineer/sites/aiecode2025/sha256/19abbc42cced666c48f8e2e65e8ae021044fab008c98cb5c749f281961fafc99.webp)

Ace

Small model + no scaffolding =

FAILURE 01
Drifts on long structure.

FAILURE 02
Needs strict rules to stay organized.

// you pay the scaffolding once, in code. Not on every turn.

Interface text:
Ornella Bahidika
Joel Allou

### Model speed and scaffolding

Choose the fastest model your latency budget allows, then build scaffolding such as a state machine and scenario logic outside the model so it can focus on its core task.

[4:43](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=283s) [A slide recommends picking the fastest model latency allows and spending the rest on scaffolding.](https://assets.ai.engineer/sites/aiecode2025/sha256/d261896d13deebd3923c01d27814dc1ba05e9fd78ea18d809ffcf9525ac2b91b.webp)

Ace •

Pick the fastest model your latency allows.

Spend the rest on scaffolding.

voice · realtime · high-volume · the model is the smallest part

Interface text:
Ornella Bahidika
Joel Allou

## Full transcript

[0:00](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=0s) Hi. I'm Ornella, and that's Joel. And we built Ace, a live AI voice tutor. It run on a small model on purpose, and I want to tell you more why that's not a compromise.

[0:14](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=14s) Quick gut check. That silence on a voice call, that's the difference between a tutor and a broken app. When a voice agent pause for even a second, your brain says it's dead.

[0:31](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=31s) So when the answer feel a little off, every instinct says reach for the smartest, biggest model. In voice, that instinct is actually backwards.

[0:44](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=44s) Because our budget was never IQ, it's millisecond.

[0:49](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=49s) The AI model need to start talking in about 950 milliseconds. A frontier model that think for a full second has already lost the room, no matter how good the answer is.

[1:04](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=64s) So we made the model small and took the hardest po- jobs away from it. It doesn't decide when happen... what happened in the lesson. It ha- it doesn't track what the student knows.

[1:17](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=77s) It doesn't plan what's next. We have a system in place to do that, and it hands the model a summary every turn. What's left for the model is one thing it's really good at, talking, and that's it, right.

[1:32](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=92s) Joel, go on and show them what it actually feel like.

[1:36](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=96s) Yeah. If maybe I can add some color to what Ornella was mentioning. So if you think about the models of today, especially the frontier model, let's take Claude 4.7, which is, uh, from Anthropic.

[1:49](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=109s) The model is really good at reasoning. You can give it a problem, in this case a lesson, and it can reason through it, it can reason through what the student is asking, and it can come up with the answer.

[2:00](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=120s) But that is actually precisely the problem because the reasoning can take couple of seconds, and those seconds are really valuable when you are building voice applications. So what we are doing is saying, "Hey, let's extract all of the thinking away from the model so that the model focuses on only what matters," which is speaking, in our

[2:24](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=144s) case. So all of the thinking is extracted into a state machine. So for Ace, we have thought about all the scenarios that are needed for a lesson. We have built a state machine that is able to coordinate each step to the next, and we've also added intelligent layer on top to derive some of the mastery that a

[2:45](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=165s) student might need for the lesson to be complete. So everything, when it comes to what happens next, when it comes to what needs to be displayed, when it comes to how to actually answer a question, it's all done outside of the model, and we simply feed that output to the model to speak out.

[3:07](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=187s) And so let's go ahead and look at an example and see how that works in real time. So the first video here is without the implementation we've done. So it's a simple Opus 4.7.

[3:19](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=199s) We ask a very simple question, and as you can see, the model is thinking, it's reasoning, and it takes couple of seconds to return the answer back to the user.

[3:28](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=208s) In this video, we've added everything we just talked about on Haiku 4.5, which is a much smaller model. Same question, but now you see that the answer comes in about 900 milliseconds.

[3:42](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=222s) And so that's the beauty of building around the model. So by removing all of the thinking, all of the logic, all of the reasoning from the model and actually putting it within the code, we actually saved a lot of time and allows us to use smaller models, which are cost effective and actually better at real-time voice applications.

[4:04](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=244s) And as you can see, this feels almost instant. And again, that's because all of the smart parts have already happened prior to the model actually speaking.

[4:15](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=255s) But I have to be honest because this isn't necessarily free. It has a cost, right? A small model like the Haiku 4.5, if it doesn't have any scaffolding, tend to drift on long structure and really needs strict rules in order to be able to stay organized.

[4:35](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=275s) So the scaffolding piece is the price. But the good thing is you pay it once and in code, right? Not on every single turn. So here's the rule: pick the fastest model that your latency budget allows, and then spend the rest of your time actually building the scaffolding.

[4:54](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=294s) So in our case, right, maybe you build a state machine. You build the reasoning process. You think about scenarios. What happens if this happens? How should your model handle it?

[5:04](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=304s) Everything that comes with the logic, everything that comes with the reasoning, you do that outside of the model and then allowing the model to focus on that one thing that it's really good at.

[5:16](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=316s) And so that's true for voice applications like Ace. That's true for real-time applications where latency is of priority, and that's really true for anything that is high volume, right?

[5:28](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=328s) In those cases, the model is the smallest part of the system. So this is Joel and Ornella, and we are building Ace again. And if you have any questions, let us know.

[5:41](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=341s) Thank you.

[5:42](https://www.youtube.com/watch?v=fnLBmfsI_Fg&t=342s) Thank you.
