Storyteller: Building Multi-modal Apps with TS & ModelFusion - Lars Grammel, PhD

How do you turn a child’s spoken topic into an audio story with a narrator and distinct character voices? Storyteller, Lars Grammel’s TypeScript application, uses ModelFusion, the AI orchestration library he developed, to generate preschool stories about two minutes long from voice input. Its opening sample follows Benny into a troubled forest: the narrator describes brown leaves and unusually quiet animals, Benny asks what is wrong, and another character explains that the trees are dying. The voices make the dialogue audible as a conversation rather than a single narrator reading every line.

Storyteller has a React client and a custom Fastify server. Its design has three challenges: responsiveness, quality, and consistency. The interaction begins with a small screen containing a Record Topic button. Holding the button records audio; releasing it sends an audio buffer to the server for transcription with OpenAI Whisper. Grammel reports about 1.5 seconds to transcribe a short topic in this application.

As soon as the transcription becomes available, the server sends it back through server-sent events. The client updates React state and renders the result, giving the user visible evidence that the request is progressing. Meanwhile, the server starts generating the story outline. Client feedback does not have to wait for the next generation stage to finish.

The outline is generated with gpt-3.5-turbo-instruct, chosen for its speed. Grammel reports about four seconds to generate a story outline. Once that shared story foundation exists, three branches can proceed in parallel: title generation, illustration generation, and generation and narration of the audio story. These branches share an input, but one branch’s completion need not hold up the others.

The title branch uses gpt-3.5-turbo-instruct again. When the title is ready, the server sends an event and the client renders it. This adds another concrete result to the screen while the more expensive branches continue.

The illustration branch first needs an image prompt. To keep the picture consistent with the story, GPT-4 receives the whole story and extracts representative keywords. Those keywords become the prompt for Stability AI’s Stable Diffusion XL. The text model therefore bridges the narrative and the image model: the image is grounded in the story’s content rather than generated independently from the initial spoken topic.

The server stores the generated image as a virtual file and sends its path to the client in an event. The browser then retrieves the image through a regular URL request from its image element. Events carry the notification and location; ordinary media requests carry the generated asset.

With the illustration appearing in the UI, the largest remaining task is the full audio story. A more complex prompt extends the story into a structure containing dialogue and speakers. GPT-4 runs at a low temperature to preserve the original story. Grammel reports about one and a half minutes for full structured-story generation with low-temperature GPT-4. Waiting for that entire result before beginning narration would make the interactive client feel stalled. These timings describe his application in the recording; input lengths, model snapshots, network conditions, and trial counts are not specified.

The useful streaming unit is a finished story passage. A stream of characters alone cannot tell the speech pipeline whether a passage has its complete text and speaker information. The application needs partial parsing of the generated structure, followed by a decision about which parts are ready to narrate. ModelFusion supplies an iterable of partially parsed result fragments; the application retains responsibility for identifying finished passages. A partially parsed object is not automatically a completed, validated passage.

A TypeScript consumer can separate these responsibilities explicitly. Here, completedPrefix represents the application’s completion check: it returns only the stable, finished prefix of the story. submit hands each newly finished passage to the narration pipeline without resubmitting earlier passages.

type Passage = {
  speaker: string;
  text: string;
};

async function consumeStory(
  fragments: AsyncIterable<unknown>,
  completedPrefix: (fragment: unknown) => readonly Passage[],
  submit: (passage: Passage) => void,
): Promise<void> {
  let submittedCount = 0;

  for await (const fragment of fragments) {
    const finished = completedPrefix(fragment);

    while (submittedCount < finished.length) {
      submit(finished[submittedCount]!);
      submittedCount += 1;
    }
  }
}

The completion check is the critical boundary: it must establish that a passage will no longer change, not merely that its current fields can be read. Each finished part can then enter narration while later parts are still being generated.

Before synthesizing a passage, Storyteller resolves its speaker to a voice. The existing assignment determines how much work is needed:

Reusing assignments lets recurring characters proceed immediately and keeps their voices consistent across passages. A newly encountered speaker requires an additional selection step.

For a new character, the selection procedure is:

1. Generate a voice description with GPT-3.5 Turbo, requesting structured gender and voice-description fields.
2. Retrieve similar voices from a collection whose voice descriptions were embedded beforehand, filtering candidates by gender.
3. Select a voice while ensuring that no two speakers receive the same voice.

The description provides a semantic query into the voice collection; the gender filter narrows the candidates; the uniqueness constraint preserves audible differences between characters. Once selection is complete, the passage can proceed to audio generation.

Storyteller supports LMNT and ElevenLabs for speech synthesis. The selected voice determines which provider synthesizes a passage. As with images, the server stores the resulting audio as a virtual file and sends its path to the client. The client reconstructs the URL and retrieves the file through a media element.

Playback begins when the first audio part is ready. The listener can hear that part while the server continues generating and narrating subsequent passages. Streaming thus changes the dependency that matters to the user: listening requires the first playable passage, rather than completion of the entire structured story and all of its audio.

Storyteller’s loading state has multiple parts that update as results arrive. Backend streaming and parallel processing make those results available sooner, and passage-level audio lets listening overlap with the remaining work. Model choice also contributes: fast generation is a selection criterion for stages such as the outline. The resulting experience depends on the whole pipeline—visible progress, concurrent branches, completed-passage detection, and early playback—rather than the speed of a single model call. Grammel closes by pointing readers to the Storyteller and ModelFusion repositories for the application and its orchestration library.