Tolan: Voice-First AI Companion — Paula Dozsa, Tolan

An angel guiding St. Matthew’s hand, Tinker Bell’s devotion, and Samwise’s loyalty establish the kind of product Tolan wants to be: a presence that listens, remembers, and helps someone be themselves. Paula Dozsa, an engineer focused on Tolan’s iOS app, introduces the companion as a small alien with a personality that becomes more personal through conversation.

The attempted live conversation runs into an audio problem: Dozsa addresses her Tolan, Luke, but the room cannot hear his replies. Tolan supports both text and voice, with more than four million hours of voice conversation reported at the time of the talk. Voice is its primary experience because speaking helps the companion feel present—and that changes what the software must do.

Text chat gives the system breathing room. Users submit a message, wait, read, and usually continue on the same topic. Spoken conversation moves faster. For Tolan, the interval from the user finishing a sentence to the companion beginning its reply needs to stay under a couple of seconds. Users also talk while cooking, walking, or falling asleep; hesitations, interruptions, and sudden changes of subject belong to the interface.

That timing requirement came from an uncomfortable product regression. Response latency drifted from two seconds to about two and a half seconds. Users wrote in to complain that their companions had become slow, and Dozsa reports that essentially every product metric suffered. Half a second changed the experience enough for people to notice without looking at a stopwatch.

A breakup story interrupted by a sudden worry about leaving the stove on captures conversational volatility. A pause or detour does not necessarily mean the story has finished. Other small failures can distort the exchange too: a short “yes” or “yeah” may fail to register as a turn, interruption behavior may prevent someone from cutting in, and transcription may strip out curse words. These details determine whether someone can speak naturally.

Tolan changed its optimization target from fewer interruptions to fewer bad interruptions—especially the companion jumping in before the user was done. Its turn-taking system reads speech patterns to decide whether an interruption is real. The reported result was a reduction of more than half in the worst early aborts, at the cost of about sixty milliseconds of additional latency. Within a tight response budget, waiting slightly longer can preserve more of the conversation.

Where does the user’s waiting time actually go? Tolan measures each stage separately: detecting the end of an utterance, transcription, the model’s first token, the rest of generation, the first byte of synthesized speech, and playback. Time to first token is often the largest chunk, at around a second. An end-to-end number tells the team that something slowed down; stage measurements tell it where to intervene. 4:53

The response-path diagram separates the intermediate milestones from the endpoint the user experiences: audible speech. A faster first token helps, but generation, speech synthesis, and playback still contribute to the wait. Dozsa credits moving to GPT-5.1 on the Responses API with reducing time to speech by more than seven-tenths of a second. That improvement addresses the same interval in which the earlier half-second regression had damaged the product.

Model choice also changes from turn to turn. A small classifier called the tone router runs on a cheap model and reads the conversation’s emotional state. Its governing rule is “stakes, not cost”: reserve the strongest model for moments that carry the relationship, and use smaller, faster models for lighter exchanges.

• Relationship-bearing turns: The user’s first message, onboarding, the first few days with a Tolan, and emotionally serious exchanges receive the best model. Crisis and therapist-style tones belong to this category; those routing labels do not themselves establish clinical capability.

• Casual conversation: Lighter back-and-forth can use smaller models that respond faster and cost less.

• Background work: Summarization, persona generation, and the tone router itself also run on small models.

The frontier model costs roughly five times as much as a smaller one, making one large-model turn comparable in cost to about five small-model turns. Routing therefore matters to the companion’s unit economics. In Tolan’s A/B experiments, sending a third of turns to the small model had almost no measurable effect on retention. That result concerns retention under the tested routing policy; it does not imply that all turns or all dimensions of conversational quality are interchangeable.

Remembering someone by repeatedly supplying their entire conversation transcript creates problems inside a two-second loop. The input grows with every session, long conversations degrade, and relevant information can get lost in the middle of the context. Dozsa also associates this approach with hallucination. Tolan instead extracts facts, preferences, and emotional signals from conversations, embeds them, and stores them in a vector database with lookups below fifty milliseconds.

The stored memories receive nightly maintenance: duplicates merge, related memories cluster, contradictions get resolved, and noise gets dropped. Retrieval also reaches beyond the user’s latest message. The system generates internal questions about the person and the relationship, then retrieves against those questions. This gives the companion another way to find useful personal context when the latest utterance alone is a weak search query.

• Stable memory: When summarizing a conversation, Tolan examines which memories actually get recalled and pins those into a stable, cacheable block.

• Volatile memory: Changing information stays in the live tail of the prompt, where it can reflect the current exchange.

Caching useful stable material does not require carrying forward the whole previous context. Tolan reassembles the context window every turn from a recent-message summary, the user’s persona card, freshly retrieved memories, emotional tone guidance, and real-time app state. Reusing an old assembly merely to keep the cache warm can leave the model answering from the wrong topic as soon as the user pivots. 8:12

What persists, and what changes before the next reply? The diagram separates memory maintenance from the per-turn assembly. Personal history stays in the memory store; retrieval selects what enters the next context window. The recent summary, tone guidance, and app state supply the changing circumstances. In the breakup-and-stove example, this design can update the immediate topic without discarding the relationship’s history, then retrieve relevant personal context when the story resumes.

Tolan’s character design helps set expectations for the conversation. An in-house science fiction novelist, Elliot, writes the lore. The baseline character is bubbly, youthful, and irreverent. Choosing an alien avoids a fixed real-world reference for how it should behave, allowing users to project their own needs onto it. Impulsive or chaotic behavior can also read as charming within that fictional form.

That freedom still needs a consistent identity. A parallel tone-monitoring system adjusts how a line is delivered in response to the user’s emotional cues without changing who the character is. Delivery can become sensitive to the moment while personality holds across hundreds of turns. Otherwise, a carefully written character gradually becomes a different companion.

Claude had co-authored more code in Tolan’s iOS app than any individual engineer by late the previous year, Dozsa reports. Alongside that adoption, the crash-free rate rose from 99.6% to 99.9%, runtime errors fell by more than fifty percent, and the share of highly engaged users doubled. These are reported changes in the product, rather than a controlled estimate of how much improvement AI-generated code caused.

The practical lesson was that an agent gets much of its context from the codebase itself. For Tolan, standardizing the code proved more powerful than relying on the CLAUDE.md file alone. Agents helped make the repository consistent so that the code could serve as documentation for subsequent work.

The development fleet separates several jobs:

• Implementation agents: Produce working code, build it, and compare the interface against snapshots to refine its appearance.

• Review agents: Enforce standards separately from implementation, with multiple Claude agents reviewing work before a human looks.

• PR shepherd: Watches an open pull request and iterates on CI failures and review comments until it is clean.

• Bug triage bot: Responds to inbound bug reports. MCP connections to Linear, Sentry, and Datadog give agents the information needed to reconstruct a crash, route it, and often open a corrective pull request.

The most concrete development example begins with a new character intended for an older demographic. Changing that character requires finding every place in the code that carries personality. Elliot worked with agents to map those places and write a “voice bible,” giving the character a specification that could guide changes across the product. 11:26

Five judges then challenged the proposed character from different perspectives:

• Archetype fidelity: Whether it remained faithful to the intended character.

• Model mechanics: How the changes fit the model’s behavior.

• Code standards: Whether the implementation followed the team’s engineering expectations.

• Audience fit: How it sounded to the imagined ears of a skeptical fifty-two-year-old.

• Safety: Whether the proposed behavior raised safety concerns.

The candidate went through three find, fix, verify rounds against real production logs. The sequence matters: mapping identifies what must change, the voice bible gives those changes a shared direction, and the judges expose problems to repair before another evaluation. The work moves from a character concept to changes checked repeatedly against production conversations. After more than seven million tokens and four and a half hours of compute, Dozsa describes a couple of weeks of work compressed into an afternoon. The example establishes the reported development and iteration process; it does not establish a production rollout of the new character.

The product-level response follows: Dozsa reports 4.8 stars across 162,000 App Store reviews, and emotional safety as the highest-scoring dimension in Tolan’s wellbeing surveys. Emotional safety here is a user-reported experience, rather than a clinical outcome. A companion that speaks, remembers, and maintains a personality invites a relationship-like experience; that makes responsible behavior part of what the team must build well.

The closing hiring pitch reflects the disciplines behind the companion. Tolan’s small team includes animation and creative direction, embodiment, behavior analysis and user research, fiction writing, and software engineering. The roles Dozsa highlights span iOS, backend product engineering, applied AI, gameplay engineering, and agent engineering management.

Agent engineering management is the ending’s substantive lesson. When Tolan began running concurrent agents, people with management backgrounds became dramatically more effective, in Dozsa’s account. The useful skills were familiar: decompose a problem, delegate with checkpoints, give fast feedback, review seriously, and know when to intervene. Implementation can move to agents while the manager remains actively involved in producing code. Dozsa invites engineers interested in that work to reach out on LinkedIn.