# ChatGPT is poorly designed. So I fixed it

Ben Holmes

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/y6L5RkEqQ8g-chatgpt-is-poorly-designed-so-i-fixed) · [Video](https://www.youtube.com/watch?v=y6L5RkEqQ8g)

## ChatGPT is poorly designed. So I fixed it

Ben Holmes’s FixGPT prototype keeps written answers inside a live voice conversation and uses tool calls to hand deeper questions to a reasoning model.

### An email draft trapped in a voice conversation

[0:00](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=0s)

Why should asking for an email by voice make it harder to work with the written draft? Ben Holmes opens with a broader frustration: an app he describes as one of history’s fastest-growing has become surprisingly confusing to use.

The first sign is two separate voice controls: one for **voice-to-text**, another for **voice-to-voice**. Holmes opens the conversational mode and exchanges greetings. The response sounds natural enough that he comments on its vocal fry. Then he asks for something more useful: a group email expressing excitement about the AI Engineer World’s Fair and suggesting a lunch meetup.

The assistant starts reading a suitably worded email aloud. But listening to a draft is different from collaborating on it. In the interface shown, Holmes has to end the call and find the transcript to get the formatted written response. The draft exists; the interaction makes him leave one mode to use it in another. This is the historical interface in the recording: OpenAI later announced integrated voice, text and visuals in its [November 2025 release notes](https://help.openai.com/en/articles/6825453-chatgpt-release-notes).

![ChatGPT text conversation showing an email subject and lunch invitation beside the presenter.](https://ai.engineer/api/talk-essay-media/7bab72a75cc34082d131bb4995e9b785425b331ba6f02cd38e6740f818c7a084)

ChatGPT displays an email draft about meeting for lunch at the AI Engineer World's Fair.

### When the organization becomes the interface

[1:13](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=73s)

Voice and text ought to be complementary parts of one conversation. Instead, Holmes says, they feel like products built by different companies. He connects that feeling to Scott Hanselman’s phrase “shipping the org chart.” In the clip Holmes plays, Hanselman describes sitting in an electric vehicle and noticing different fonts on the map, climate controls and speedometer. Realizing that he is looking at three Android tablets chained together makes the manufacturer’s organizational divisions suddenly visible.

Model selection creates a similar problem. Holmes sketches a sequence in which a researcher ships a technical improvement, consumers unexpectedly want it, and marketing has never been consulted. This is his critique of how products reach users, rather than an established account of OpenAI’s internal process. The resulting menu becomes a science fair of names: **o3** and **GPT-4o**, followed in his joke by Xbox 360, One Series S and Pro. The user inherits the work of interpreting the labels before accomplishing a task.

### Keep the call open and put the draft beside it

[2:07](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=127s)

The redesign has two goals: **allow voice and text at the same time**, and choose a suitable model from the user’s request. Holmes proposes using **GPT-4o Realtime** for the live audio conversation, with **tool calls** providing the other capabilities. A text tool can deliver links or long drafts; a research tool can hand a question to a more capable model and return its answer. The [Realtime API](https://openai.com/index/introducing-the-realtime-api/) supplies the conversational foundation without requiring every task to be completed through speech.

The interface starts in a dormant state. Pressing the voice button opens a familiar floating orb, with mute and end-call controls. A new chat button opens an iMessage-like panel while the call continues. Holmes compares it to texting a friend during FaceTime: the call controls stay at the top, and the conversation remains available underneath.

The panel includes a reminder of the spoken request, followed by the detailed written response. An email draft can therefore be read while the voice conversation remains active. The selected frame shows a gray request labeled “From voice” above a blue email response beginning “Hey Cora team.” With the output problem addressed, Holmes turns to a second question: what should happen when the request needs more work? He introduces Warp Terminal, a developer tool for writing code across environments.

![Enlarged chat panel with a gray “From voice” request and a blue email response beginning “Hey Cora team.”](https://ai.engineer/api/talk-essay-media/5dc38481277a9a29fd47e79770ad41ecbde5e1be30a280543c0ca1990530a9c5)

A voice request appears above a written email draft.

### Let the request determine the amount of reasoning

[3:19](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=199s)

In Warp, Holmes asks to undo his last commit. The request goes to a coding agent that prepares a terminal command. The visible interface includes a command approval panel and a Run button: it shows the proposed action awaiting approval, rather than establishing that the undo completed.

![Terminal showing the undo request, an explanation, and a command approval panel with a Run button.](https://ai.engineer/api/talk-essay-media/bfef35a9706a7869ed26eec6326d814f0c05eaf36bc0eabc27e3cbfe58f6fdfb)

Warp asks for approval to run a command after “Undo my last commit.”

He then raises the scope to refactoring the entire codebase to use Flutter. For this request, he describes Warp detecting the complexity and using a reasoning model to write a plan before proceeding. The distinction is between choosing an immediate action and first working out a sequence of changes; producing a plan does not establish that the full refactor succeeds.

| Request | Response path |
|---|---|
| Undo the last commit | Prepare a terminal command for approval |
| Refactor the codebase to Flutter | Use a reasoning model to plan the work |
| Ask for depth or pros and cons | Proposed heuristic for a reasoning handoff |

Holmes suggests that explicit requests for detail or tradeoffs can serve as simple routing signals. While the reasoning model works, the interface can communicate how long it is thinking, then return the fuller response. The user expresses the depth of help they need without first having to select a model name.

### From a park link to a deeper history

[3:56](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=236s)

The prototype demonstration brings the two changes together using off-the-shelf APIs. After a brief greeting, Holmes says he has just landed in California and asks for a link to a park he should visit. The assistant replies aloud that it will send a link to a popular park. A link is a useful test of the combined interface: it belongs in text, even when the request arrives through speech.

Holmes follows up by naming Yosemite and asking for its history: “Go deep.” The assistant gives a short spoken introduction, mentioning Yosemite’s establishment as a national park, then directs him to the chat for more detail. Speech maintains the conversation while the written channel accommodates the longer answer.

### Two tools connect the experience

[4:37](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=277s)

The written output is handled through a send-chat-message tool. Its description tells the conversational model to send details that are easier to explain in text. Holmes reports that this description was sufficient to elicit the behavior in his demonstration; he did not need an additional system prompt for it. The tool makes text delivery an action available during the voice conversation.

A separate tool handles requests that need deeper reasoning. The visible definition names it `ask_reasoning_model` and requires a string field called `details`. A compact Dart definition expresses that contract:

```dart
final reasoningTool = <String, Object>{
  'type': 'function',
  'name': 'ask_reasoning_model',
  'description':
      'Ask a reasoning model for a deeper answer when the user '
      'wants to explore a topic in detail.',
  'parameters': <String, Object>{
    'type': 'object',
    'properties': <String, Object>{
      'details': <String, Object>{
        'type': 'string',
        'description':
            'The question and context needed for the deeper answer.',
      },
    },
    'required': <String>['details'],
  },
};
```

For the Yosemite follow-up, `details` needs to carry the subject and the request for an in-depth history. The tool description supplies the routing cue; the argument supplies the work to hand off.

![Code editor close-up showing ask_reasoning_model, its description, and a required string parameter named details beside the presenter.](https://ai.engineer/api/talk-essay-media/76fce83b8ffe7ede30296c2202a542498ba8af0bb65333019c657f43586e136a)

The reasoning tool definition includes a required details field.

Once the reasoning result arrives, the application has two delivery options:

- **Return it to the conversational model:** let the voice interaction continue with the information it needs to answer.
- **Send it directly to the client:** place the detailed result in the app’s written conversation.

This completes the design: live audio remains the conversational entry point, while tool calls deliver readable material and invoke deeper reasoning when requested. Holmes closes by pointing viewers to [FixGPT](https://github.com/bholmesdev/fixgpt), the source code on his GitHub.

## Resources

- [FixGPT source code](https://github.com/bholmesdev/fixgpt) — Flutter and Go prototype combining voice conversations, written messages and reasoning-model handoffs, with local setup instructions.
- [Introducing the Realtime API](https://openai.com/index/introducing-the-realtime-api/) — The original announcement explains GPT-4o speech-to-speech streaming and function calling available before the talk.
- [Why is AI marketing so, so bad?](https://www.gettheleverage.com/p/why-is-ai-marketing-so-so-bad) — Evan Armstrong examines confusing AI model names and organizational coordination in a companion essay linked from the video description. Full access is restricted.
- [Warp 2.0 and reasoning-based planning](https://www.warp.dev/blog/reimagining-coding-agentic-development-environment) — Warp's subsequent launch announcement describes coding agents and a dedicated planning mode using reasoning models.
- [ChatGPT's later integrated Voice interface](https://help.openai.com/en/articles/6825453-chatgpt-release-notes) — The November 25, 2025 entry announces simultaneous spoken answers, visible text and typed interaction within ChatGPT conversations.

## Slides

### ChatGPT voice controls

The presenter points out two adjacent voice controls: one converts speech to text, while the other starts a voice conversation. The cursor points to the voice conversation button.

[0:22](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=22s) [ChatGPT input area with microphone and voice conversation buttons.](https://assets.ai.engineer/sites/aiecode2025/sha256/f7000d3553d5dd9b457b4cdce919139303397b2e0a6563cb2b9ba3881b8f6ca4.webp)

### AI product development and model choices

The speaker argues that technical improvements shipped without marketing input can leave users facing a confusing menu of models. He proposes choosing the model based on the user’s request.

[1:56](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=116s) [Highlighted article excerpt above a ChatGPT model selection menu.](https://assets.ai.engineer/sites/aiecode2025/sha256/6dbd510a7da90d3afbde3a83fa8755bfa72c14b774f6cac04fc46b3cb14276e3.webp)

4) Some whiz kid on the research team makes a technical improvement, but no one is sure how it will land with users. Management demands they ship it anyway because they are in a desperate race to stay ahead of their competition.

5) Turns out that this new feature was exactly what consumers were craving. Servers burn to the ground, growth hockey sticks, and everyone is excited.

6) Marketing is never consulted because no one knew this would be a big thing.

You do this across a company with thousands of employees and dozens of different models, and you get a product organizational hierarchy that looks like this:

Source: The Leverage - Why is AI marketing so, so bad?

Interface text:
ChatGPT 4o
Model
GPT-4o
Great for most tasks
GPT-4.5
RESEARCH PREVIEW
Good for writing and exploring ideas
o3
Uses advanced reasoning
o4-mini
Fastest at advanced reasoning
o4-mini-high
What's on t[illegible]

### Voice mode controls

The FixGPT voice mockup adds a Chat button alongside Mute and End. The presenter says it opens a text panel while the call remains active.

[2:48](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=168s) [Voice interface mockup with a blue orb and enlarged chat, mute and end buttons.](https://assets.ai.engineer/sites/aiecode2025/sha256/b379b04be2e4ecaeb8b1794acd90d2c27b740ed382702cbcdd14521fc5ec49a3.webp)

FixGPT

Chat
Mute
End

### Undoing a commit

In Warp, a request to undo the last commit hands off to a coding agent. It displays a git reset command intended to keep the changes in the working directory.

[3:28](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=208s) [A terminal agent responds to a request to undo the last commit and displays a reset command.](https://assets.ai.engineer/sites/aiecode2025/sha256/5b675b4fa4e81830b69e36bfe305e2bba00118bc1552a6aaa0ae50b0d70532b7.webp)

Undo my last commit.

I'll help you undo your last commit. Since this is a git repository, I'll use git reset to undo the last commit while keeping the changes in your working directory.

git reset HEAD^

Interface text:
~/Projects/bholmesdev
auto
Undo my last commit.
Ask a follow up, backspace to exit

### Planning a complex coding request

After a simple Git reset request, the demo flags a full Flutter refactor as complex and offers a choice: create a plan with o3 or proceed without one.

[3:36](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=216s) [A terminal shows a request to refactor a codebase and options to create a plan or proceed without one.](https://assets.ai.engineer/sites/aiecode2025/sha256/d39abd813a7cf2946e016d2866883e74f20ea9246e8ac7f9032a7c661a86da34.webp)

Undo my last commit.

I'll help you undo your last commit. Since this is a git repository, I'll use git reset to undo the last commit while keeping the changes in your working directory.

git reset HEAD^

Refactor this entire codebase to use Flutter instead.

This job is quite complex. Would you like to plan or proceed w[illegible]

Create plan with o3
Proceed without plan

Proceeding without a plan is faster and requires fewer tokens. Manage p[illegible]

Interface text:
~/Projects/bholmesdev

### Detailed reasoning response

The proposed design hands a pros-and-cons request to a reasoning model and shows its thinking time before returning a detailed answer. The example displays 30 seconds and a list of Flutter advantages.

[3:51](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=231s) [A chat response shows a thinking duration and part of a list of Flutter advantages.](https://assets.ai.engineer/sites/aiecode2025/sha256/2753266acb77665e84666c92703bd46969048685161e878de1a6634f3e0ab5b1.webp)

Give me some detailed pros and cons for each

Thought for 30 sec

1. Flutter (Dart - Google)
Pros
• Near-native performance & visuals – Ahead-of-time (AOT) compiled Dart plus a custom GPU renderer (Impeller/Skia) removes most “bridge” overhead. Recent roadmap adds Impeller support on Android and drops the older Skia path on iOS for leaner binaries. RedditMedium
• Truly single UI codebase – You write one widget tree and the engine paints the same pixels on iOS, Android, web, desktop, embedded, and even cars.
• Fast iteration – “Hot reload” is realtime and rock-solid; tight [illegible]

Interface text:
Message

## Full transcript

[0:00](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=0s) ChatGPT is one of the fastest-growing, fastest-growing apps in history. But with hundreds of millions of people using this every day, has no one stopped to ask, "Why is this app so confusing now?" [upbeat music]

[0:16](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=16s) Let's demonstrate that with a conversation. I have ChatGPT pulled up, and we actually see two buttons to interact with voice. This is the voice-to-text option, and this is voice-to-voice.

[0:27](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=27s) Hey, OpenAI. How's it going?

[0:30](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=30s) Hey there. I'm doing great. Thanks for asking. How about you? What's new on your end?

[0:34](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=34s) Sounds pretty good, with a little bit of vocal fry. But let's try something a little bit more descriptive. I want you to write a group email just saying I'm super excited for the AI Engineer World's Fair, and I want to meet up for lunch sometime.

[0:50](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=50s) Sure thing. How about something like this? "Excited for the AI Engineer World's Fair. Let's meet for lunch. I'm super excited about the upcoming AI Engineer World's Fair."

[0:59](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=59s) It does respond with a nicely worded email, but it can only respond through voice. If you want to collaborate on the written email, the best you can do is end the call and look for a voice transcript with the nice formatting at the end.

[1:13](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=73s) Ideally, this would be something multimodal with text and voice together, but it really feels like these two apps were built by two different companies. Scott Hanselman called this shipping the org chart, which he explained really well talking about sitting in an EV.

[1:28](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=88s) I'm looking at the map, and then I turn, and then I'm looking at the climate thing, and then I'm looking at the speedometer, and they're all different fonts. And then I realized it was three Android tablets chained together, and I could suddenly see the organona- organizational chart of this large international auto company.

[1:46](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=106s) And OpenAI is guilty of this as well. You have some whiz kid who ships a technical improvement. It turns out it's exactly what consumers are craving. Marketing's never consulted, and you end up with a science fair full of potential options, like o3, 4o, Xbox 360, One Series S, Pro.

[2:07](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=127s) You get the picture. So let's see how to fix it. There's two things I want to change about ChatGPT today: allowing voice and text at the same time and smartly choosing the right model depending on your ask.

[2:20](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=140s) And we can do this with off-the-shelf tools. 4o Realtime gives you a live audio chat, and Tool Calls can handle the rest. You can ask it to send a text for longer details, like links and drafts, and maybe a research tool could hand off to a smarter model and come back with an answer.

[2:39](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=159s) Let's bring this concept to life. We have our voice button that takes us from our dormant state to voice mode, which works exactly the same, floating orb and all.

[2:48](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=168s) You have your mute, your end call, and also a new button for chat. This pulls up a panel that looks like iMessage. It kind of feels like texting a friend while you're on a FaceTime call, with your call controls at the top, a little reminder of what you might have asked if you're looking through history, and then

[3:05](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=185s) a text response in case it needs more detail for an email draft. Now, what if we want to ask something that requires more detail? This is something we explore at Warp Terminal, which is a developer tool to write code in any environment.

[3:19](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=199s) I have a project pulled up here, and I can ask it for simple things, like undo my last commit. This hands off to a coding agent and decides to help me out by running commands in my terminal.

[3:30](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=210s) It can also handle more complex questions, though, like refactor this entire code base to use Flutter instead. It detects that it's complex and decides to write a plan with the reasoning model to make sure the code actually works.

[3:43](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=223s) This is a really effective pattern that you can do with some heuristics. If you asked for details and pros and cons, for example, we could hand off to reasoning, tell you how long it's thinking, and hand back a more detailed response.

[3:56](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=236s) Now, let's go build it using off-the-shelf APIs. Hey there. How's it going?

[4:02](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=242s) Hey there. I'm doing well, thank you. How about you? How's your day going?

[4:07](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=247s) Pretty good. I just landed in California. Can you send me a link to a park that I should go visit?

[4:13](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=253s) Of course. I'll send you a link to a popular park in California.

[4:18](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=258s) Yeah, I've heard about Yosemite. Can you tell me more about its history? Go deep.

[4:25](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=265s) Yosemite National Park's history is rich, beginning with its ancient [REDACTED:origin] heritage and leading to its establishment as a national park. Check the chat for more details.

[4:37](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=277s) And I'm handling text input just using tool calls. So we have this send chat message tool, where you send details that are easier to explain via text. Didn't even need a system prompt, just added this description, and it was smartly able to send the right stuff with text.

[4:52](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=292s) It's honestly amazing how far you can go with very simple prompts these days. And for reasoning models, we have another tool call. This is sending off. Whenever a user wants to go deeper on a topic, you can send some details and then respond back to the model with whatever it needs to say or just dump it straight

[5:09](https://www.youtube.com/watch?v=y6L5RkEqQ8g&t=309s) in the client if you're building an app like this. If you want to see the source code, it's available on my GitHub under FixGPT. With that, subscribe and enjoy the rest of the conference.
