Text-to-Speech Data Preparation and Fine-tuning Workshop - Ronan McGovern

How do you make a pretrained text-to-speech model sound like a particular person? Ronan McGovern’s workshop starts with Sesame CSM-1B and builds toward that goal through data preparation and fine-tuning. The Trelis Research workshop repository provides the Colab notebook and slides. CSM is a token-based speech model; Orpheus, from Canopy Labs, is another example of the same broad approach.

The workflow takes a recording, converts it into a voice dataset, fine-tunes with Unsloth, and compares speech before and after training. You can record yourself or use a single-speaker video for which you have permission to adapt the voice; Sesame’s usage guidance requires explicit consent when mimicking a real individual. One notebook covers the process. For a deeper architecture introduction, McGovern also points to an approximately 90-minute companion explanation linked through the repository, including discussion of Moshi.

A text language model predicts the next token, appends it to the input, and repeats. Token-based speech generation extends that loop: supply text and predict audio tokens. A conversational speech model can also consume earlier text and audio, so its next utterance depends on how the preceding conversation sounded as well as what was said. The output is a sequence of discrete audio tokens that can be decoded into speech.

To turn a continuous waveform into discrete tokens, use a codebook: a dictionary of learned vectors. A token selects an entry from that dictionary. The representation needs enough capacity to preserve both acoustic detail and information relevant to meaning. In the introductory training picture, an encoder converts a waveform into tokens, a decoder reconstructs the waveform, and reconstruction error supplies a learning signal to improve the representation. This explains the codec’s purpose, rather than the complete training objective of a modern codec.

One token per audio window is too restrictive for the representation McGovern describes. Multiple codebook tokens let the model represent a window at several levels of detail. CSM uses 32 audio codebook tokens per window: its main transformer predicts token zero across time, while a smaller depth decoder predicts the remaining 31 tokens within each window. The main transformer has one billion parameters. This separates progression through the utterance from completion of each window’s audio representation.

The workshop does not train this architecture or its codec from scratch. It begins with a model that already accepts text and audio and generates speech. McGovern then uses a recording from his own YouTube channel to shift the generated voice toward his own.

Open the notebook in Colab and select a GPU through Change runtime type. The 2025 workshop uses a T4 and describes it as available on Colab’s free tier; availability and YouTube download access are not present-day guarantees. The first notebook stage uses Whisper to make paired audio snippets and transcripts, targeting clips up to 30 seconds long.

McGovern’s demonstration uses 41 rows. He reports that an earlier trial with 20 ten-second snippets was insufficient, and suggests roughly 50 thirty-second snippets as a starting point for an audible effect. These are his small-data heuristics, not a measured minimum dataset size. The resulting pairs will train CSM through Unsloth.

Prepare the source in this order:

1. Choose one speaker. A multi-speaker video needs diarization to separate identities; this basic notebook does not provide it.
2. Select the transcription model. McGovern recommends Whisper turbo as close to large-model quality but faster, and prefers it to small, base, or tiny for this task.
3. Install the extraction and transcription tools. The notebook uses yt-dlp and OpenAI Whisper. McGovern recommends Colab because he has encountered YouTube authentication blocks outside it.
4. Transcribe on the GPU. Use CUDA and save the transcription as a local JSON file. That file contains text, token details, and timestamped segments.

Read the transcript before constructing training examples. McGovern’s recurring correction is the brand name: Whisper may produce the English word Trellis, while his channel is Trelis. Correct the relevant occurrences and re-upload the JSON to Colab. The text paired with the audio should reflect what the speaker actually said, including names the transcriber does not know. This Python version applies that same correction to both the full transcript and individual segment text while leaving their timing fields intact:

import json
from pathlib import Path

source = Path("transcript.json")
transcript = json.loads(source.read_text(encoding="utf-8"))

transcript["text"] = transcript["text"].replace("Trellis", "Trelis")
for segment in transcript["segments"]:
    segment["text"] = segment["text"].replace("Trellis", "Trelis")

Path("transcript.corrected.json").write_text(
    json.dumps(transcript, ensure_ascii=False, indent=2),
    encoding="utf-8",
)

Use the corrected file as the input to dataset construction.

Whisper produces short segments; training rows can combine adjacent segments into longer passages. The notebook accumulates segments toward a configured 30-second maximum and pairs their text with the corresponding source audio. McGovern describes both crossing the threshold and enforcing a maximum, so inspect clips near the boundary rather than assuming the narration specifies a strict clipping rule.

For the optional Hugging Face upload, authenticate and set the organization, repository name, and maximum row duration. Point the construction cell at the extracted audio and corrected JSON, then generate the rows. The run produces 41 clips; setting the upload flag to False keeps the dataset local. McGovern estimates transcription should take a few minutes even for a source video of 30–60 minutes in this workflow.

Inspect the result by reading the text and playing the paired audio. The dataset viewer makes that relationship visible: each row has a transcript and an audio player. Retain the organization and dataset name for loading it during fine-tuning.

A duration limit alone does not produce good linguistic boundaries. The basic implementation can finish a row in the middle of a sentence. McGovern suggests detecting sentence boundaries with NLTK or a regular expression so that rows contain complete sentences or paragraphs where possible. This is an improvement to the data preparation, not a feature already demonstrated by the simple segment accumulator.

Install Unsloth, then load CSM-1B with a configured maximum sequence length. Leaving dtype=None lets the loading path choose the precision: the workshop uses torch.float16 on T4 and describes torch.bfloat16 for supported newer GPUs, including Ampere, Hopper, and Blackwell. The Transformers model class is CsmForConditionalGeneration. Native CSM support was already available in Transformers 4.52.1 before this workshop; current documentation is not a frozen copy of its notebook APIs.

McGovern loads the base model without quantization, describing its few-gigabyte size as fitting within the T4’s roughly 15 GB of available memory. Watch GPU allocation while the shards load. An unexpectedly high reading makes him consider whether the model was loaded twice and whether a runtime restart is needed, but the usage drops and he continues.

Printing the architecture exposes three different responsibilities:

• Backbone: the main transformer that advances through the audio sequence.
• Depth decoder: the smaller transformer that predicts the other 31 codebook tokens for each window.
• Codec: the components that convert between waveforms and discrete audio representations.

Fine-tuning does not update every parameter in those components. The workshop attaches trainable adapters to selected attention and other linear layers, reducing the memory and work required for adaptation.

Before applying adapters, skip ahead to inference. McGovern deliberately tests the sentence “We just finished fine-tuning a text-to-speech model” before any fine-tuning has happened. The notebook offers two input paths: apply a chat template, or prepare tensors more directly from the text and speaker ID. Both lead to text-only, zero-shot generation.

Without reference context, the base model can produce different speaker identities. Nonzero-temperature sampling adds variation, but temperature is not a way to specify the desired person. Voice cloning supplies reference audio at inference time without updating model weights. The notebook takes an utterance and its transcript from the prepared dataset, places them in the conversation, and asks for a new utterance. That context gives the model an example of the voice it should continue.

Load the dataset before running the reference-conditioned call. It must contain audio and text columns. An optional source column identifies speakers, starting at zero; the notebook defaults missing identity information to source=0. That is appropriate for McGovern’s single-speaker video, but it does not separate speakers in mixed audio. The loading code also checks maximum audio length for the trainer that will be configured later. After commenting out a line during the live run, McGovern starts the cloning call.

The first text-only playback uses the fine-tuning sentence, and McGovern identifies the result as a woman’s voice. A second test uses “Sesame is a super cool TTS model which can be fine-tuned with Unsloth.” The contrast between these samples illustrates how variable the unconditioned speaker can be.

He then plays the reference-conditioned version of the second sentence, interrupting and replaying it while comparing the voice. McGovern judges it closer to his own, including some of the drawn-out quality he hears in his speech, but still short of what he hopes to obtain from training. The next experiment therefore changes the weights and repeats the comparison.

Return to the loaded model and attach LoRA adapters to the attention projections—Q, K, V, and O—and the MLP gate, up, and down projections. McGovern chooses lora_alpha=16, which he recommends for a model of roughly this size, and rank r=32. Rank controls the dimensions of the adapter matrices; increasing it gives the update more capacity.

The recording calls its rescaling option rescale_lora and describes it in terms of learning rate. For the documented rank-stabilized option, PEFT’s LoRA reference uses use_rslora: it changes the multiplier on the adapter contribution from α/r to α/√r, rather than directly setting the optimizer learning rate. This distinction should not be treated as confirmation of the notebook’s exact historical argument name. Calling print_trainable_parameters() shows just under 2% of parameters trainable in McGovern’s configuration. Additional modules such as lm_head or embed_tokens could be trained and saved through modules_to_save, but he leaves them unchanged because this adaptation does not change the token inventory.

The raw data is already loaded, with source=0 for the single speaker. Next, determine the maximum audio and text lengths so preprocessing can prepare compatible inputs. The notebook displays an audio length of approximately 700,000 and a text length of 587; McGovern explicitly corrects himself after calling the audio value tokens. Keep these as notebook length values, not durations or a verified token count. Supply the respective lengths through the text and audio keyword arguments, along with the audio sampling rate.

Apply the preprocessing mapping across the examples. Unsloth’s notebook preparation creates the input IDs, attention mask, labels, input values, and cutoffs needed by the multimodal trainer. Inspect the resulting columns and row count before passing the processed dataset, together with the adapter-equipped model, into training.

The planned virtual batch size is eight. McGovern usually prefers 32, but reduces it for this small dataset. He initially describes one epoch over 41 rows as roughly five steps. Against such a short schedule, three warm-up steps would consume much of training, so he suggests reducing warm-up to one. The remaining optimization choices are a learning rate of 2e-4, hardware-appropriate float16 or bfloat16, an 8-bit AdamW optimizer, weight decay, and a constant learning rate after warm-up. Training artifacts go to the configured output directory.

Before training, McGovern reports nearly 7 GB occupied out of roughly 15 GB of available GPU memory. He estimates about ten minutes for training and describes loss falling from approximately 6.34 to about 3.7. A still-declining training loss suggests there may be room to train further, and he expects reducing warm-up to help this short run. The actual batch size is two, with a virtual batch size of eight; because the reported memory use is only about half the GPU, he suggests that an actual batch of four may fit.

Queue the inference cells so the trained model produces the same kinds of saved audio as the base model. This creates four conditions to compare:

Using both axes separates the contribution of training from the contribution of a reference utterance.

The live progress later shows two of 60 steps, which does not match the earlier one-epoch, roughly five-step description. The recording therefore does not establish a single consistent executed schedule. More consequentially, this run has no evaluation dataset: its displayed loss is training loss, not a held-out measure of voice quality. McGovern recommends splitting training and evaluation data and watching whether evaluation loss continues to fall. If it stalls, reconsider the number of epochs or learning rate.

He also recommends monitoring gradient norm, suggesting around or below one as a heuristic. TensorBoard can expose these signals: install it, enable reporting to it, and set logging_dir, for example to logs. That monitoring is discussed but left disabled in the basic demonstration. McGovern points to his longer video and planned follow-ups for more detail.

While training proceeds, choose how to retain the result. A local save writes to Colab’s temporary storage. For a Hub repository, the notebook exposes separate publishing choices for the model and processor; retain both so the saved model has its corresponding input-processing configuration.

There are two useful distribution forms:

• Adapter-only: save the trained LoRA weights as a lightweight artifact that accompanies the base model.
• Merged model: merge the adapter update into the base weights and save the full model, with a 16-bit merge option in the notebook.

Notebook flags control saving and uploading. McGovern is unsure whether the merged helper automatically includes the processor, so check that it is present rather than assuming the merge handled it. To reload the adaptation, return to the loading cell and replace the base-model identifier with the repository you saved. This is model persistence and distribution; the workshop does not set up a hosted inference service.

At completion, McGovern reports a training loss of 3.72 and memory use just under half of the available GPU capacity. He then listens to the fine-tuned model without reference audio. On the first sentence, he judges the sample to sound a little like him, closer to his own voice than the unconditioned baseline.

The pacing is still imperfect. McGovern suggests filtering source segments with excessive pauses, although his edited YouTube videos already remove many pauses. This identifies a separate target from speaker identity: a voice can acquire recognizable characteristics while retaining awkward timing.

In the second fine-tuned, text-only sample, McGovern hears room for improvement in pronunciation: the T in tuned still sounds partly American. He suggests more data could improve it. Fine-tuning has moved the voice in the desired direction without fully reproducing its pronunciation.

Finally, he plays the fine-tuned model with reference audio. McGovern judges this the strongest sample, praising the pronunciation of tuned and a small irregularity he finds natural. On replay, he identifies a remaining mismatch in intonation around super cool, wording he would not usually use himself. These conclusions are his listening judgments on the demonstrated samples; the workshop provides no held-out quantitative voice-similarity score.

For the next experiment, McGovern proposes about 500 thirty-second rows, particularly to improve generation without reference conditioning. That larger dataset is a suggestion, not a tested result here. He retrospectively estimates that a roughly 30-minute source video was enough for this small-data example combining fine-tuning and cloning; source-video length does not establish how much audio was retained in the training rows. The demonstrated path is to use adaptation and reference context together, then improve data coverage and timing where listening reveals weaknesses.

The workshop repository remains the starting point for the notebook and slides. McGovern closes by pointing toward more detailed Trelis videos on data preparation and fine-tuning hyperparameters, and invites questions in the recording’s YouTube comments.