# Storyteller: Building Multi-modal Apps with TS & ModelFusion - Lars Grammel, PhD

Lars Grammel

AI Engineer Summit 2023

[Talk page](https://ai.engineer/talks/KnndhPSVF3E-storyteller-building-multi-modal-apps-ts) · [Video](https://www.youtube.com/watch?v=KnndhPSVF3E)

## Storyteller: Building Multi-modal Apps with TS & ModelFusion

A spoken topic becomes an illustrated, multivoice children’s story by combining parallel generation, partial structure parsing, and playback before the full story is ready.

Familiarity with TypeScript, React state, and asynchronous iteration will help with the implementation details.

### A forest in trouble, told in several voices

[0:00](https://www.youtube.com/watch?v=KnndhPSVF3E&t=0s)

How do you turn a child’s spoken topic into an audio story with a narrator and distinct character voices? [Storyteller](https://github.com/lgrammel/storyteller), Lars Grammel’s TypeScript application, uses [ModelFusion](https://github.com/vercel/modelfusion), the AI orchestration library he developed, to generate preschool stories about two minutes long from voice input. Its opening sample follows Benny into a troubled forest: the narrator describes brown leaves and unusually quiet animals, Benny asks what is wrong, and another character explains that the trees are dying. The voices make the dialogue audible as a conversation rather than a single narrator reading every line.

![A story card titled “Benny and the Magical Forest Rescue” shows a bear in a boat and audio controls. OpenAI, LMNT, ElevenLabs, and stability.ai appear beside their respective capabilities.](https://ai.engineer/api/talk-essay-media/5429fd0961e153a9ee2ad3a02fbec74496583c4ff77de5849afb713da28ff369)

Storyteller’s illustrated audio story alongside its model providers.

### Make the first result visible

[1:00](https://www.youtube.com/watch?v=KnndhPSVF3E&t=60s)

Storyteller has a React client and a custom Fastify server. Its design has three challenges: **responsiveness**, quality, and consistency. The interaction begins with a small screen containing a **Record Topic** button. Holding the button records audio; releasing it sends an audio buffer to the server for transcription with **OpenAI Whisper**. Grammel reports about 1.5 seconds to transcribe a short topic in this application.

As soon as the transcription becomes available, the server sends it back through **server-sent events**. The client updates React state and renders the result, giving the user visible evidence that the request is progressing. Meanwhile, the server starts generating the story outline. Client feedback does not have to wait for the next generation stage to finish.

### Use the outline to start parallel work

[2:14](https://www.youtube.com/watch?v=KnndhPSVF3E&t=134s)

The outline is generated with **`gpt-3.5-turbo-instruct`**, chosen for its speed. Grammel reports about four seconds to generate a story outline. Once that shared story foundation exists, three branches can proceed in parallel: title generation, illustration generation, and generation and narration of the audio story. These branches share an input, but one branch’s completion need not hold up the others.

![A client-server diagram shows audio recording flowing into topic transcription and story-outline generation, then branching into three generation tasks.](https://ai.engineer/api/talk-essay-media/1218ef1bdd13967bb9e525f5af1ef014903fa70aad332169cfce0a6c7841c885)

The story outline branches into title, image-prompt, and audio-story generation.

The title branch uses `gpt-3.5-turbo-instruct` again. When the title is ready, the server sends an event and the client renders it. This adds another concrete result to the screen while the more expensive branches continue.

### Derive the illustration from the story

[2:57](https://www.youtube.com/watch?v=KnndhPSVF3E&t=177s)

The illustration branch first needs an image prompt. To keep the picture consistent with the story, **GPT-4** receives the whole story and extracts representative keywords. Those keywords become the prompt for Stability AI’s **Stable Diffusion XL**. The text model therefore bridges the narrative and the image model: the image is grounded in the story’s content rather than generated independently from the initial spoken topic.

The server stores the generated image as a **virtual file** and sends its path to the client in an event. The browser then retrieves the image through a regular URL request from its image element. Events carry the notification and location; ordinary media requests carry the generated asset.

### Stream completed passages, not just tokens

[3:45](https://www.youtube.com/watch?v=KnndhPSVF3E&t=225s)

With the illustration appearing in the UI, the largest remaining task is the full audio story. A more complex prompt extends the story into a structure containing dialogue and speakers. GPT-4 runs at a low temperature to preserve the original story. Grammel reports about one and a half minutes for full structured-story generation with low-temperature GPT-4. Waiting for that entire result before beginning narration would make the interactive client feel stalled. These timings describe his application in the recording; input lengths, model snapshots, network conditions, and trial counts are not specified.

**The useful streaming unit is a finished story passage.** A stream of characters alone cannot tell the speech pipeline whether a passage has its complete text and speaker information. The application needs partial parsing of the generated structure, followed by a decision about which parts are ready to narrate. ModelFusion supplies an iterable of partially parsed result fragments; the application retains responsibility for identifying finished passages. A partially parsed object is not automatically a completed, validated passage.

A TypeScript consumer can separate these responsibilities explicitly. Here, `completedPrefix` represents the application’s completion check: it returns only the stable, finished prefix of the story. `submit` hands each newly finished passage to the narration pipeline without resubmitting earlier passages.

```typescript
type Passage = {
  speaker: string;
  text: string;
};

async function consumeStory(
  fragments: AsyncIterable<unknown>,
  completedPrefix: (fragment: unknown) => readonly Passage[],
  submit: (passage: Passage) => void,
): Promise<void> {
  let submittedCount = 0;

  for await (const fragment of fragments) {
    const finished = completedPrefix(fragment);

    while (submittedCount < finished.length) {
      submit(finished[submittedCount]!);
      submittedCount += 1;
    }
  }
}
```

The completion check is the critical boundary: it must establish that a passage will no longer change, not merely that its current fields can be read. Each finished part can then enter narration while later parts are still being generated.

### Give each speaker a consistent voice

[5:03](https://www.youtube.com/watch?v=KnndhPSVF3E&t=303s)

Before synthesizing a passage, Storyteller resolves its speaker to a voice. The existing assignment determines how much work is needed:

| Speaker state | Next action |
| --- | --- |
| Narrator | Use the predefined narrator voice |
| Previously encountered character | Reuse the assigned voice |
| New character | Find and assign a voice |

Reusing assignments lets recurring characters proceed immediately and keeps their voices consistent across passages. A newly encountered speaker requires an additional selection step.

For a new character, the selection procedure is:

1. Generate a voice description with **GPT-3.5 Turbo**, requesting structured gender and voice-description fields.
2. Retrieve similar voices from a collection whose voice descriptions were embedded beforehand, filtering candidates by gender.
3. Select a voice while ensuring that no two speakers receive the same voice.

The description provides a semantic query into the voice collection; the gender filter narrows the candidates; the uniqueness constraint preserves audible differences between characters. Once selection is complete, the passage can proceed to audio generation.

![A flowchart routes a speaker with a voice ID directly to audio generation. The other path generates a voice description, retrieves similar voice IDs, and selects a voice ID before joining audio generation.](https://ai.engineer/api/talk-essay-media/8dcc6a28ef4e14ed89db19337ffc8be269194f32ea81776a504199dcbaf8f9e8)

Voice selection feeds audio generation for each story part.

### Start listening while generation continues

[5:54](https://www.youtube.com/watch?v=KnndhPSVF3E&t=354s)

Storyteller supports LMNT and ElevenLabs for speech synthesis. The selected voice determines which provider synthesizes a passage. As with images, the server stores the resulting audio as a virtual file and sends its path to the client. The client reconstructs the URL and retrieves the file through a media element.

**Playback begins when the first audio part is ready.** The listener can hear that part while the server continues generating and narrating subsequent passages. Streaming thus changes the dependency that matters to the user: listening requires the first playable passage, rather than completion of the entire structured story and all of its audio.

### Responsiveness across the whole pipeline

[6:36](https://www.youtube.com/watch?v=KnndhPSVF3E&t=396s)

Storyteller’s loading state has multiple parts that update as results arrive. Backend streaming and parallel processing make those results available sooner, and passage-level audio lets listening overlap with the remaining work. Model choice also contributes: fast generation is a selection criterion for stages such as the outline. The resulting experience depends on the whole pipeline—visible progress, concurrent branches, completed-passage detection, and early playback—rather than the speed of a single model call. Grammel closes by pointing readers to the Storyteller and ModelFusion repositories for the application and its orchestration library.

## Resources

- [Storyteller source and setup](https://github.com/lgrammel/storyteller) — Source and setup instructions for the experimental preschool storytelling app, including its service requirements and documented limitations.
- [ModelFusion](https://github.com/vercel/modelfusion) — Archived TypeScript library with examples of text, structured-object, image and speech generation.
- [Storyteller's voice selection code](https://github.com/lgrammel/storyteller/blob/main/src/storyteller/VoiceManager.ts) — Implementation of character voice descriptions, embedding-based retrieval, voice reuse and speech-provider selection.
- [Lars Grammel's introduction to ModelFusion](https://dev.to/lgrammel/introducing-modelfusion-build-ai-apps-with-javascript-and-typescript-4aia) — Grammel explains ModelFusion's motivation, explicit application control and multimodal integrations. Updated in January 2024.
- [ModelFusion joins Vercel AI SDK](https://vercel.com/blog/vercel-ai-sdk-3-1-modelfusion-joins-the-team) — May 2024 announcement explaining ModelFusion's contribution to AI SDK Core and its text, structured-object and tool-call primitives.

## Slides

### Diamond sponsors

[0:09](https://www.youtube.com/watch?v=KnndhPSVF3E&t=9s) [Diamond sponsors slide showing Fixie and Supabase logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/b3edd6c6072875ff1282697e19201c1cfe0bada6599bffeb282f6fe6e5368755.webp)

DIAMOND SPONSORS

FIXIE
supabase

### Storyteller

Storyteller turns voice input into roughly two-minute audio stories for preschool children. The slide shows three illustrated story examples built with TypeScript and ModelFusion.

[0:14](https://www.youtube.com/watch?v=KnndhPSVF3E&t=14s) [Storyteller introduction showing three illustrated audio story examples, with a presenter overlay at lower left.](https://assets.ai.engineer/sites/aiecode2025/sha256/52183aff1a623fd11ca17a463c4b5fdc35ab41093842fca2f1cb5995ae0d1292.webp)

Storyteller

Building a multi-modal app with TypeScript and ModelFusion

"a bear in the forest."
Ben the Brave Bear and the Lost Cub

"Timmy and his dad are building a snowman."
Timmy and the Magic Snowman

"Pirate Jack finding the golden treasure."
Pirate Jack and the Golden Treasure Adventure

Interface text:
0:11 / 0:29
Part 1 / ...
0:06 / 0:14
Part 5 / 9

### Storyteller models and services

Storyteller pairs an illustrated, narrated bear story with OpenAI for text and transcription, LMNT and ElevenLabs for speech synthesis, and stability.ai for image generation.

[0:52](https://www.youtube.com/watch?v=KnndhPSVF3E&t=52s) [An illustrated bear story card alongside providers for text, speech, and image generation.](https://assets.ai.engineer/sites/aiecode2025/sha256/06d6c1f6da6b3defed79d488cc5b1e9a14d9fc332792f266dcf2b0d50c0b336d.webp)

"A tale about a bear living in the forest."
Benny and the Magical Forest Rescue

Text Generation, Embeddings, & Transcription
OpenAI

Speech Synthesis
LMNT
ElevenLabs

Image Generation
stability.ai

Interface text:
0:07 / 0:12
Part 5 / 14

### Transcribing the story topic

The client sends the recording to the server, where Whisper transcribes a short topic in about 1.5 seconds. The server then sends the topic back to the client as an event.

[1:39](https://www.youtube.com/watch?v=KnndhPSVF3E&t=99s) [Client-server diagram showing an audio recording sent for transcription, with a Whisper code example and estimated latency.](https://assets.ai.engineer/sites/aiecode2025/sha256/fcdaed9ad03da4409678f390c8f360154459e48b66bde1b72fcb6acb9be13b64.webp)

Client
Server

audio recording
transcribe topic

Speech-to-Text: OpenAI whisper

const transcription = await transcribe(
  new OpenAITranscriptionModel({ model: "whisper-1" }),
  { type: "mp3", data: audioRecording }
);

~1.5 sec for short topic

[1:47](https://www.youtube.com/watch?v=KnndhPSVF3E&t=107s) [Client-server diagram showing an audio recording sent for transcription and a topic event returned to the client.](https://assets.ai.engineer/sites/aiecode2025/sha256/489a6cee23455e39f44979d0e80b1ea2b86d7cc318c551427da683b37f34bba8.webp)

Client
Server

audio recording
transcribe topic
event with topic

### Generate the story outline

While the client receives the transcribed topic, the server generates a preschool story outline with gpt-3.5-turbo-instruct. The outline takes about 4 seconds and enables subsequent tasks to run in parallel.

[2:10](https://www.youtube.com/watch?v=KnndhPSVF3E&t=130s) [Workflow and code for generating a preschool story outline, with an approximately four-second timing.](https://assets.ai.engineer/sites/aiecode2025/sha256/503557ad515399bd5369d95726ec24144a954bfc9aea3c7cfb31e19d320e9d43.webp)

Story outline: OpenAI gpt-3.5-turbo-instruct

Client
“a bear in the forest.”

Server
audio recording
transcribe topic
event with topic
generate story outline

async function generateStory(topic: string) {
  return generateText(
    new OpenAITextGenerationModel({
      model: "gpt-3.5-turbo-instruct",
      temperature: 1.2,
      maxCompletionTokens: 1000,
    }),
    [
      "Generate a story aimed at preschoolers on the following topic: ",
      `'${topic}'.`,
    ].join("\n")
  );
}

~ 4 sec for generating story outline

### Generate the title

After the story outline is ready, title generation runs in parallel with the image and audio tasks. The title takes about 0.8 seconds and is sent to the client as an event.

[2:39](https://www.youtube.com/watch?v=KnndhPSVF3E&t=159s) [Workflow and code for generating a story title, with an approximately 0.8-second timing.](https://assets.ai.engineer/sites/aiecode2025/sha256/db6e1c932a5e79211aba2c4e8d6a0e9ded17897da2c464d4d57bceffa24fa9e9.webp)

Title: OpenAI gpt-3.5-turbo-instruct

Client
“a bear in the forest.”

Server
audio recording
transcribe topic
event with topic
generate story outline
generate title
generate image prompt
generate full audio story

async function generateTitle(story: string) {
  return generateText(
    new OpenAITextGenerationModel({
      model: "gpt-3.5-turbo-instruct",
      temperature: 0.7,
      maxCompletionTokens: 200,
      stopSequences: ['"'],
    }),
    [
      "Generate short title for the following story for pre-school children: ",
      "",
      `'${story}'.`,
      "",
      'Title: "',
    ].join("\n")
  );
}

~0.8 sec for generating title

### Generate the image prompt

In parallel with title generation, GPT-4 extracts representative keywords from the story for an image prompt. The slide reports about 3.8 seconds for this step.

[3:06](https://www.youtube.com/watch?v=KnndhPSVF3E&t=186s) [Workflow and code for deriving an image prompt from the story, with an approximately 3.8-second timing.](https://assets.ai.engineer/sites/aiecode2025/sha256/9d9ef833db9953b73bad146f640c1222a2f5eb572dbeda0535167b6a6b5309ae.webp)

Image prompt: OpenAI gpt-4

const imagePrompt = await generateText(
  new OpenAIChatModel({
    model: "gpt-4",
    temperature: 0,
    maxCompletionTokens: 500,
  }).withPromptFormat(mapInstructionPromptToOpenAIChatFormat()),
  {
    instruction:
      "Generate an short image generation prompt " +
      "(only abstract keywords, max 8 keywords) for the following story:",
    input: story,
  }
);

~3.8 sec for image prompt

Client
Server

audio recording
transcribe topic
event with topic
generate story outline
generate title
event with title
generate image prompt
generate full audio story

Interface text:
“a bear in the forest.”
Ben the Brave Bear and the Lost Cub

### Generate the image

The image prompt feeds Stability AI’s Stable Diffusion XL to create a colorful preschool story illustration. The slide gives generation time as up to 15 seconds.

[3:18](https://www.youtube.com/watch?v=KnndhPSVF3E&t=198s) [Workflow and code for generating a story illustration, with a timing of up to 15 seconds.](https://assets.ai.engineer/sites/aiecode2025/sha256/baad85663108a23165dfa298eb95fd9536023eddf3b12f28c71500e57d569195.webp)

Image: Stability AI stable-diffusion-xl-v1.0

Client
Server

audio recording
transcribe topic
event with topic
generate story outline
generate title
event with title
generate image prompt
generate image
generate full audio story

const image = await generateImage(
  new StabilityImageGenerationModel({
    model: "stable-diffusion-xl-1024-v1-0",
    cfgScale: 7,
    height: 1024,
    width: 1024,
    samples: 1,
    steps: 30,
  }),
  [
    {
      text:
        `${imagePrompt} style of colorful illustration ` +
        `for a preschooler story`,
    },
  ]
);

Up to 15 sec

Interface text:
“a bear in the forest.”
Ben the Brave Bear and the Lost Cub

### Story generation workflow

The server sends the image path to the client, which retrieves and displays the image. The story outline also feeds full audio story generation.

[3:25](https://www.youtube.com/watch?v=KnndhPSVF3E&t=205s) [Client-server diagram showing an image file, an event carrying its path, and a client retrieval request.](https://assets.ai.engineer/sites/aiecode2025/sha256/0eeb6938ede9f4a08359bd784f1683df09db78374c21dd548ea45c4f74fa3811.webp)

“a bear in the forest.”
Ben the Brave Bear and the Lost Cub

Client
Server

audio recording
transcribe topic
event with topic
generate story outline
generate title
event with title
generate image prompt
generate image
event with path
image file
retrieve image
generate full audio story

[3:43](https://www.youtube.com/watch?v=KnndhPSVF3E&t=223s) [Client-server workflow highlights full audio story generation after the story outline.](https://assets.ai.engineer/sites/aiecode2025/sha256/3f1d18a72044ab0054e1a729c3ae9e8628492638cda2965d8dc0b63ed169339a.webp)

“a bear in the forest.”
Ben the Brave Bear and the Lost Cub

Client
Server

audio recording
transcribe topic
event with topic
generate story outline
generate title
event with title
generate image prompt
generate image
event with path
image file
retrieve image
generate full audio story

### Generating a structured audio story

GPT-4 expands the outline into narration and dialogue, but generation takes about 1.5 minutes. Grammel proposes streaming the structure and parsing new passages for speech.

[3:53](https://www.youtube.com/watch?v=KnndhPSVF3E&t=233s) [Code uses GPT-4 to generate story parts with narration, dialogue and speakers; the slide shows an estimated time of 1.5 minutes.](https://assets.ai.engineer/sites/aiecode2025/sha256/18ab2eb43903622d26520c8d7f472a0cdee0761325da9e611f306591ec8bb1d0.webp)

Full audio story: OpenAI gpt-4

async function generateAudioStory(story: string) {
  return streamStructure(
    new OpenAIChatModel({ model: "gpt-4", temperature: 0 }),
    new ZodStructureDefinition({
      name: "story",
      description: "Kids story with narration.",
      schema: z.object({
        parts: z.array(
          z.object({
            type: z
              .enum(["narration", "dialogue"])
              .describe("Type of story part. Either 'narration' or 'dialogue'."),
            speaker: z
              .string()
              .describe("Speaker of a dialogue (direct speech) part. Must be a single speaker."),
            content: z.string().describe("Content of the story part"),
          })
        ),
      }),
    }),
    [
      OpenAIChatMessage.user(
        [
          "Expand the following story into a longer, narrated audio story for preschoolers.",
          "",
          "The audio story should include interesting dialogue by the main characters.",
          "The language should be understandable by a preschooler.",
          "",
          "Add details and dialoge to make the story parts longer.",
          "Add the speaker to each dialogue part. A dialogue part can only have one speaker.",
          "There must only be one narrator.",
          "Each spoken part must be a dialogue part with a speaker.",
          "",
          "Story:",
          story,
        ].join("\n")
      ),
    ]
  );

~1.5 minutes

Diagram:
Client
Server
audio recording
transcribe topic
event with topic
generate story outline
generate title
event with title
generate image prompt
generate full audio story

### Streaming story parts

ModelFusion streams partially parsed story fragments. The application excludes the last potentially incomplete part, validates the rest, and processes completed parts for narration.

[4:40](https://www.youtube.com/watch?v=KnndhPSVF3E&t=280s) [Code partially parses streamed story fragments and processes complete parts.](https://assets.ai.engineer/sites/aiecode2025/sha256/3210880e367d45daf4eb10facdf20bd094688969edab037546791613708190b1.webp)

“a bear in the forest.”
Ben the Brave Bear and the Lost Cub

Client
Server

audio recording
transcribe topic
event with topic
generate story outline
generate title
event with title
generate image prompt
generate image
event with path
generate full audio story
streaming story part

const audioStoryFragments = await generateAudioStory(story);

for await (const fragment of audioStoryFragments) {
  if (!fragment.isComplete) {
    const parseResult = structuredStorySchema
      .deepPartial()
      .safeParse(fragment.value);

    if (parseResult.success) {
      const partialParts = (parseResult.data.parts ?? [])
        // the last story part might not be complete yet:
        .slice(0, -1);

      // ensure that the remaining story parts are complete:
      const partialPartsParseResult = z
        .array(narratedStoryPartSchema)
        .safeParse(partialParts);

      if (partialPartsParseResult.success) {
        await processNewParts(partialPartsParseResult.data);
      }
    }
  } else {
    await processNewParts(fragment.value.parts);
  }
}

### Choosing a voice for each speaker

Each streamed story part needs a voice. The narrator has a predefined voice, and known speakers reuse theirs; a new speaker triggers a voice description.

[5:01](https://www.youtube.com/watch?v=KnndhPSVF3E&t=301s) [The workflow branches when a streamed story part has a speaker without an assigned voice.](https://assets.ai.engineer/sites/aiecode2025/sha256/fd6778f91815601e937afd659181ad66b9830db1c49da4df930d6f5ee291aeb5.webp)

“a bear in the forest.”
Ben the Brave Bear and the Lost Cub

Client
Server

audio recording
transcribe topic
event with topic
generate story outline
generate title
event with title
generate image prompt
generate image
event with path
image file
retrieve image
generate full audio story
streaming story part
speaker has voice id
no voice for speaker
generate voice description for speaker

### Generating a voice description

When a story speaker has no assigned voice, GPT-3.5 Turbo generates a structured description with gender and age. The slide shows this step taking approximately 2 seconds.

[5:27](https://www.youtube.com/watch?v=KnndhPSVF3E&t=327s) [Code requests a structured voice description for a story speaker using GPT-3.5 Turbo.](https://assets.ai.engineer/sites/aiecode2025/sha256/8cd0067fc1f4275e8f5a0f430bec8b37d90a051ab81abcbd25da6352c971547c.webp)

Voice description: OpenAI gpt-3.5-turbo

const voiceDescription = await generateStructure(
  new OpenAIChatModel({
    model: "gpt-3.5-turbo",
    temperature: 0,
  }),
  new ZodStructureDefinition({
    name: "voice",
    schema: z.object({
      gender: z.string().describe("M for male, F for female"),
      description: z.string().describe("Voice description"),
    }),
  }),
  [
    OpenAIChatMessage.user(
      [
        `## Task`,
        `Generate a voice description for ${speaker} from the following story for an audio book.`,
        "The voice should be appropriate for a preschooler listener.",
        "Include the gender and age in the voice description.",
        "",
        "## Story",
        story,
        "",
        "## Speaker",
        speaker,
        "",
        "## Voice description (incl. age, gender)",
      ].join("\n")
    ),
  ]
);

~2 seconds

Client
Server

audio recording
transcribe topic
event with topic
generate story outline
generate title
event with title
generate image prompt
generate full audio story
streaming story part
speaker has voice id
no voice for speaker
generate voice description for speaker

### Retrieving similar voices

The app searches previously embedded voice descriptions for candidates matching a speaker’s generated description, filtering by gender when specified. The slide shows retrieval taking about 0.35 seconds.

[5:36](https://www.youtube.com/watch?v=KnndhPSVF3E&t=336s) [Code retrieves voice candidates from a vector index using a voice description and a gender filter.](https://assets.ai.engineer/sites/aiecode2025/sha256/103c9f983a877378b8293fd87f406c7e52c1d0cd8f47d726ed5af9668156b542.webp)

Voice retrieval: in-memory vector store and OpenAI text-embedding-ada-002

const potentialVoices = await retrieve(
  new VectorIndexRetriever({
    vectorIndex: voiceIndex,
    embeddingModel: new OpenAITextEmbeddingModel({
      model: "text-embedding-ada-002",
    }),
    maxResults: 5,
    similarityThreshold: 0.2,
    filter: (voice) =>
      ["M", "F"].includes(voiceDescription.gender)
        ? voice.gender === voiceDescription.gender
        : true,
  }),
  voiceDescription.description
);

~0.35 sec

Client
Server

audio recording
transcribe topic
event with topic
generate story outline
generate title
event with title
generate image prompt
generate image
event with path
generate full audio story
streaming story part
speaker has voice id
no voice for speaker
generate voice description for speaker
retrieve voice ids with similar descriptions

Interface text:
“a bear in the forest.”
Ben the Brave Bear and the Lost Cub

### Speech synthesis

The selected voice determines whether LMNT or ElevenLabs synthesizes each story part. The slide shows a synthesis time of 1–3 seconds.

[5:53](https://www.youtube.com/watch?v=KnndhPSVF3E&t=353s) [Workflow diagram with code selecting a speech-synthesis provider for each story part.](https://assets.ai.engineer/sites/aiecode2025/sha256/57a94d5aafb343afbdb448a78ac5b2fb1b64ad02ebabfdd3f66152d7a4621202.webp)

Speech synthesis: LMNT & ElevenLabs

switch (provider) {
  case "lmnt":
    return synthesizeSpeech(
      new LmntSpeechSynthesisModel({ voice: voiceId }),
      content
    );

  case "elevenlabs":
    return synthesizeSpeech(
      new ElevenLabsSpeechSynthesisModel({ voice: voiceId }),
      content
    );
}

1-3 seconds

Client
Server

audio recording
transcribe topic
event with topic
generate story outline
generate title
event with title
generate image prompt
generate image
event with path
[illegible]
generate full audio story
streaming story part
speaker has voice id
no voice for speaker
generate voice description for speaker
retrieve voice ids with similar descriptions
select voice id
generate audio for story part

Interface text:
“a bear in the forest.”
Ben the Brave Bear and the Lost Cub

### Story generation workflow recap

Storyteller updates its loading states as results arrive. The server generates the title, image and audio in parallel, letting listeners start playback while later story parts are still being generated.

[6:35](https://www.youtube.com/watch?v=KnndhPSVF3E&t=395s) [Client loading states above a diagram of parallel story, image, and audio generation.](https://assets.ai.engineer/sites/aiecode2025/sha256/6b8f62b11973f26f5a86923196f2ebb9ac764dc82c3a8e6519f4a5822e6bfb7d.webp)

Client
Server

audio recording
transcribe topic
event with topic
generate story outline
generate title
event with title
generate image prompt
generate image
event with path
image file
retrieve image
generate full audio story
streaming story part
speaker has voice id
no voice for speaker
generate voice description for speaker
retrieve voice ids with similar descriptions
select voice id
generate audio for story part
event with path
audio file
retrieve audio

Interface text:
Story Teller
Automatically generate stories for pre-school kids.
Record topic

Story Teller
Automatically generate stories for pre-school kids.
Recording...

“a bear in the forest.”

“a bear in the forest.”
Ben the Brave Bear and the Lost Cub

“a bear in the forest.”
Ben the Brave Bear and the Lost Cub

“a bear in the forest.”
Ben the Brave Bear and the Lost Cub
0:01 / 0:03
Part 3 / 11

### Storyteller and ModelFusion resources

Storyteller and ModelFusion are available on GitHub. Three examples show illustrated stories with audio playback.

[7:07](https://www.youtube.com/watch?v=KnndhPSVF3E&t=427s) [Project resource names above three illustrated story examples; the presenter overlaps part of the first example.](https://assets.ai.engineer/sites/aiecode2025/sha256/787a1471ea56d80331c1d0b6c330a668d09bb8dcc471e5ceeb2fa3ed8bac148c.webp)

Storyteller: github.com/lgrammel/storyteller
ModelFusion: github.com/lgrammel/modelfusion

"a bear in the forest."
Ben the Brave Bear and the Lost Cub

"Timmy and his dad are building a snowman."
Timmy and the Magic Snowman

"Pirate Jack finding the golden treasure."
Pirate Jack and the Golden Treasure Adventure

Interface text:
[illegible] / 11
0:11 / 0:29
Part 1 / ...
0:06 / 0:14
Part 5 / 9

## Full transcript

[0:00](https://www.youtube.com/watch?v=KnndhPSVF3E&t=0s) [upbeat music] Hey, everyone. I'm presenting Storyteller, an app for generating short audio stories for preschool kids.

[0:20](https://www.youtube.com/watch?v=KnndhPSVF3E&t=20s) Storyteller is implemented using TypeScript and ModelFusion, an AI orchestration library that I've been developing. It generates audio stories that are about two minutes long, and all it needs is a voice input.

[0:34](https://www.youtube.com/watch?v=KnndhPSVF3E&t=34s) Here is an example of the kind of story it generates to give you an idea.

[0:38](https://www.youtube.com/watch?v=KnndhPSVF3E&t=38s) One day, while they were playing, Benny noticed something strange. The forest wasn't as vibrant as before. The leaves were turning brown, and the animals seemed less cheerful. Worried, Benny asked his friends what was wrong.

[0:51](https://www.youtube.com/watch?v=KnndhPSVF3E&t=51s) Friends, why do the trees look so sad? And why are you all so quiet today?

[0:55](https://www.youtube.com/watch?v=KnndhPSVF3E&t=55s) Benny, the forest is in trouble. The trees are dying, and we don't know what to do.

[1:00](https://www.youtube.com/watch?v=KnndhPSVF3E&t=60s) How does this work? Let's dive into the details of the Storyteller application. Storyteller is a client-server application. The client is written using React, and the server is a custom Fastify implementation.

[1:13](https://www.youtube.com/watch?v=KnndhPSVF3E&t=73s) The main challenges were responsiveness, meaning getting results to the user as quickly as possible, uh, quality, and consistency. So when you start Storyteller, it's just a small screen that has a Record Topic button, and once you start pressing it, it starts recording.

[1:32](https://www.youtube.com/watch?v=KnndhPSVF3E&t=92s) Um, the audio, when you release, gets sent to the server as a buffer, and there we transcribe it. For transcription, I'm using OpenAI Whisper. Um, it is really quick for a short topic, one point five seconds, and once it becomes available, an event goes back to the client.

[1:52](https://www.youtube.com/watch?v=KnndhPSVF3E&t=112s) So the client-server communication works through an event stream, server-sent events, [clears throat] that are being sent back.

[2:02](https://www.youtube.com/watch?v=KnndhPSVF3E&t=122s) The event arrives on the client, and the React state updates, updating the screen. Okay, so then the user knows something is going on. In parallel, I start generating the story outline.

[2:14](https://www.youtube.com/watch?v=KnndhPSVF3E&t=134s) For this, I use gpt-3.5-turbo-instruct, which I found to be very fast. So it can generate a story outline in about four seconds, and once we have that, we can start a bunch of other tasks in parallel.

[2:28](https://www.youtube.com/watch?v=KnndhPSVF3E&t=148s) Generating the title, generating the image, and generating and narrating the audio story all happen in parallel. I'll go through those one by one now.

[2:41](https://www.youtube.com/watch?v=KnndhPSVF3E&t=161s) First, the title is generated. For this, OpenAI gpt-3.5-turbo-instruct is used again, giving a really quick result. Once the title is available, it's being sent to the client again as an event and rendered there.

[2:57](https://www.youtube.com/watch?v=KnndhPSVF3E&t=177s) In parallel, the image generation runs. First, uh, there needs to be a prompt to actually generate the image, and here consistency is important. So we pass in the whole story into a GPT-4 prompt that then extracts relevant representative keywords for an image prompt from the story.

[3:17](https://www.youtube.com/watch?v=KnndhPSVF3E&t=197s) That image prompt is passed into Stability AI Stable Diffusion XL,

[3:24](https://www.youtube.com/watch?v=KnndhPSVF3E&t=204s) where an image is generated. The generated image is stored as a virtual file in the server, and then an event is sent to the client with a path to that file.

[3:37](https://www.youtube.com/watch?v=KnndhPSVF3E&t=217s) The client can then, through a regular URL request, just retrieve the image as part of an image tag,

[3:45](https://www.youtube.com/watch?v=KnndhPSVF3E&t=225s) and it shows up in the UI. Generating the full audio story is the most time-consuming piece of the puzzle. Here we have a complex prompt that takes in the story and creates a structure with dialogue and speakers and extends the story.

[4:05](https://www.youtube.com/watch?v=KnndhPSVF3E&t=245s) We use GPT-4 here with a low temperature to retain the story, and the problem is it takes one and a half minutes, which is unacceptably long for an interactive client.

[4:17](https://www.youtube.com/watch?v=KnndhPSVF3E&t=257s) So how can this be solved? The key idea is streaming the structure. That's a little bit more difficult than just streaming characters token by token. Um, we need to always partially parse the structure and then determine if there is a new passage that we can actually, uh, narrate and, uh, synthesize speech for.

[4:41](https://www.youtube.com/watch?v=KnndhPSVF3E&t=281s) ModelFusion takes care of the partial parsing and returns an iterable over fragments of partially parsed results that the application needs to decide what to do with them. Here we determine which story part is finished so we can actually narrate it.

[4:58](https://www.youtube.com/watch?v=KnndhPSVF3E&t=298s) So we narrate each story part as it's getting finished.

[5:03](https://www.youtube.com/watch?v=KnndhPSVF3E&t=303s) For each story part, we need to determine which voice, uh, we use to narrate it. The narrator has a predefined voice, and for all the speakers where we already have voices, we can immediately proceed.

[5:16](https://www.youtube.com/watch?v=KnndhPSVF3E&t=316s) However, when there's a new speaker, we need to figure out which voice to give it.

[5:23](https://www.youtube.com/watch?v=KnndhPSVF3E&t=323s) The first step for this is to generate a voice description for the speaker.

[5:29](https://www.youtube.com/watch?v=KnndhPSVF3E&t=329s) Here's a GPT-3.5 Turbo prompt that gives us a structured result with gender and a voice description, and we then use that, um, for retrieval, where we beforehand embedded all the voices based on their descriptions and now can just retrieve them filtered by gender.

[5:46](https://www.youtube.com/watch?v=KnndhPSVF3E&t=346s) Um, then a voice is selected, making sure there are no speakers with the same voice, and finally, we can generate the audio.

[5:54](https://www.youtube.com/watch?v=KnndhPSVF3E&t=354s) Here for the speech synthesis, LMNT and ElevenLabs are supported. Based on the voices that have been chosen, one of those providers is picked and the audio is synthesized.

[6:07](https://www.youtube.com/watch?v=KnndhPSVF3E&t=367s) Similar to the images, we generate an audio file, and we store it virtually in the server and then send the path to the client, which reconstructs the URL and just retrieves it as a media element.

[6:20](https://www.youtube.com/watch?v=KnndhPSVF3E&t=380s) Once the first audio is completed, the client can then start playing. And while this is ongoing, in the background you're listening, and in the background the server continues to generate more and more parts.

[6:36](https://www.youtube.com/watch?v=KnndhPSVF3E&t=396s) And that's it. So let's recap how the main challenge of responsiveness is addressed here. We have a loading state that has multiple parts that are updated as more results become available.

[6:48](https://www.youtube.com/watch?v=KnndhPSVF3E&t=408s) We use streaming and parallel processing in the back end to make results available as quickly as possible, and you can start listening while the processing is still going on.

[6:58](https://www.youtube.com/watch?v=KnndhPSVF3E&t=418s) And finally, models are being chosen such that the processing time for, like, the generation, say, of the story is minimized.

[7:08](https://www.youtube.com/watch?v=KnndhPSVF3E&t=428s) Cool. I hope you enjoyed my talk. Thank you for listening, and if you wanna find out more, you can find Storyteller and also ModelFusion on GitHub at github.com/lgrammel/storyteller and github.com/lgrammel/modelfusion. [upbeat music]
