Text-to-Speech Data Preparation and Fine-tuning Workshop - Ronan McGovern

Talk transcript

1. 0:00

   Welcome to this workshop on text-to-speech model fine-tuning. I'm Ronan of the Trelis Research YouTube channel, and today I'll be walking you through data preparation and fine-tuning of Sesame's CSM-1B model.

2. 0:13

   You should be able to follow along. All of the materials are available on the Trelis Research GitHub. If you go to ai-worlds-fair-2025, you'll be able to find the Colab notebook and also the slides that I'm going to go through for this workshop.

3. 0:29

   Now, by the end of this workshop, you should be able to train a text-to-speech model so that it sounds like a specific voice. You should be able, uh, to understand how token-based text-to-speech models work.

4. 0:42

   The one we're going to train today is Sesame CSM-1B, but Orpheus from Canopy Labs is also a token-based text-to-speech model.

5. 0:50

   We'll go through how to create a voice data set. You can record one yourself, uh, but today we're just going to pull audio from a YouTube video. You can pick which video and use that as a basis for the fine-tuning.

6. 1:03

   We'll cover then fine-tuning using Unsloth, which is a library based around Transformers. Many of you might be familiar with it. And last of all, we'll evaluate performance, uh, both before the fine-tuning and after the fine-tuning.

7. 1:17

   Now, before I start to get into the Colab notebook, it's just going to be one Colab note-book, book that covers all of these steps. I want to describe a little bit how these models work very briefly.

8. 1:27

   If you do want to understand in much more detail, uh, I'll point you to a much longer video, about an hour and thirty minutes, where I talk about token-based, uh, text-to-speech models.

9. 1:38

   I also talk a little bit about Moshi. I'll put a link to this in the GitHub repo as well. It helps to start off with reviewing how a token-based text model works, such as, say, the ChatGPT models, uh, from OpenAI, uh, like GPT-4o or Llama series of models.

10. 1:55

    Those models, which you'll be familiar with, most likely involve taking in a string of text and predicting the next, uh, word or token, and then passing that around so that you recursively predict the next word.

11. 2:07

    Now, when we talk about text-to-speech, what we want to do is take in a series of text tokens and predict the next audio token, if you can bear with me as to what an audio token might be.

12. 2:18

    And in fact, we'd like to be able to pass in a history of text and also audio from the earlier conversation and then decode. Recursively, we'd like to produce the next audio token, so we have a string of audio tokens that re-represent the output speech, uh, that we want.

13. 2:33

    So conceptually, this is what we're trying to do. We need a transformer model that instead of taking in only text and outputting text tokens, can input ideally text and audio and output a string of audio tokens.

14. 2:46

    So the question, the first question then is: how do you make tokens for audio? How do you take a wave, a sound wave like this and represent it, uh, in terms of discrete tokens rather than continuous, uh, frequencies or sound waves?

15. 3:01

    And the answer is, you can represent audio or a little piece of audio as a choice from a codebook. A codebook is like a dictionary, and inside the dictionary, there are a series of vectors where a vector represents a given, uh, sound.

16. 3:17

    Now, these codebooks could be fairly large in order to represent a wide variety of sounds, and that could be the acoustics and also the semantics. But it is possible to take sound and represent it in a discrete way, much like text, uh, represents meaning in a discrete way.

17. 3:34

    The way that these codebooks are trained, I won't get into too much detail, but broadly speaking, you need to train, um, an encoder-decoder where you take a sound wave, you then have some kind of transformer that converts it into tokens, and then you have another transformer that will decode it back, uh, into a wave.

18. 3:53

    And the question you ask during training is whether the output here, after encoding it to tokens and decoding it back, you ask the question of whether it matches the input.

19. 4:03

    And to the extent it doesn't match, you will use that difference or the loss to back propagate and update these weights here. So it is possible, giving sufficient data on sound waves, to then train, uh, a codebook so you can represent those, uh, that sound in a discrete form.

20. 4:20

    Now, it turns out if you use that approach, uh, you can represent sound using a codebook, but it turns out insufficient to just have one token for each time step.

21. 4:30

    Actually, what works out better is to have a hierarchical representation, or at least have multiple tokens allowed at the same kind of timestamp or same window, so that you can have a more detailed representation.

22. 4:42

    So this is the idea that's used in most token-based audio models. Instead of just having one token at each window, you will have multiple tokens potentially set up in a hierarchy.

23. 4:51

    So you represent higher level kind of meaning or acoustics, and then you have more granular, more granular, and so you have these multiple layers of representation. Now, there are a few ways to set things up to have different tokens represent sound at a given window.

24. 5:05

    The one we're going to focus on is the Sesame model, uh, where there are thirty-two tokens at every, uh, window that will represent the sound. There is a zeroth token that will be predicted by the main transformer here, but then there's a secondary transformer that's being used to decode out the other thirty-one, uh, tokens of meaning.

25. 5:27

    So whereas here we basically have Sesame decoding out with a main transformer autoregressively these first zeroth tokens, the kind of stacked tokens are being decoded by a second transformer.

26. 5:39

    So actually, Sesame is two models. It's the main one billion parameter model and then a much smaller model that will decode out kind of these hierarchical tokens here. So this is a quick overview of how the model is going to work.

27. 5:53

    We will start off with a pre-trained model that is capable of taking in text and audio and outputting, uh, a stream of audio here, and we'll see how it performs just when we give it some text.

28. 6:02

    We'll see it generate or we'll hear it generate some sound. But then we're going to take a data set And it's, uh, in my case, it's gonna be of one of my YouTube videos with my voice, and we're gonna see if we can make the model sound a little bit more like me by fine-tuning it, uh, on

29. 6:16

    that data. So with that, we will move over to the Colab notebook. As I said, you can find it, uh, on Trelis Research if you look for the AI Worlds Fair 2025 repo.

30. 6:29

    And if you open that up in Colab, you should be able to run it for free using, uh, a T5. So when you open up the notebook, make sure that, uh, you are connected to a GPU.

31. 6:41

    This should be available for free. If you're not connected to a GPU, uh, you can say, uh, Change runtime type, and then you'll want to select, uh, the TPU here.

32. 6:51

    So I am connected, and there are two sections in, in this notebook. If you're following along, you can just open it up in your browser now. There are two sections we'll go through.

33. 7:02

    The first is data generation, where we're gonna pick a YouTube video, we're gonna use Whisper in the Colab notebook to transcribe the YouTube video, and then we're going to convert it into a data set.

34. 7:13

    The data set will have audio snippets of up to 30 seconds, and they will have text, which will be the transcription, uh, using Whisper.

35. 7:23

    Given that data set of snippets, I think I will have maybe around, let's see, 41 rows of data, which is actually enough to have an effect on the quality.

36. 7:33

    I have previously done trainings with maybe shorter snippets of only 10 seconds and 20, 20 of those, and I don't think that's enough data. So I would say if you've got 30-second snippets, probably having roughly 50 of them is enough to start to have an effect.

37. 7:48

    And given that data set, we'll then fine-tune the Unsloth model. Or sorry, we'll fine-tune the CSM model, the Sesame model, using Unsloth, and we will measure performance at the end and see how that fares.

38. 8:03

    So let's start off with the data generation, and you can pick your own YouTube video if you like. I recommend picking a video where you just have one speaker.

39. 8:12

    You could pick a video with two speakers, but you need to do more data processing. You need to do diarization to split that, uh, audio up into multiple speakers, which is not supported in this basic notebook here.

40. 8:23

    So I recommend taking a video where you just have one speaker. This is a, a Trelis video here. You can then pick the model size, the Whisper model size that you want to use for transcription.

41. 8:35

    I recommend using turbo, which is almost as good as large. Um, I think it's better than definitely small, base, and tiny, but it's actually a lot faster than large.

42. 8:44

    So we'll go with turbo. Run the installs. So you'll need white, uh, youtube-dlp. Note that this works well in Colab. If you try and run this notebook outside of Colab, you can run into authentication issues where you'll get blocked from downloading from YouTube, so I recommend running this in Colab.

43. 9:01

    Um, you also need to install Whisper, which is, uh, from OpenAI. And with all of that installed, you should be ready to then load the Whisper model here and run the transcription.

44. 9:12

    And we're gonna use the GPU, so we'll use CUDA here, and it should then save your transcript to a file, to a JSON file, which will be saved here locally.

45. 9:22

    So what you should find is a JSON file, uh, that's saved and, uh, you should be ready to, to then use that file as a basis for the training.

46. 9:31

    Now, when you run the transcription, I'm not gonna run it here 'cause it's, uh, gonna take a little bit of time, and when you have that JSON file, um, you can take a look at that file, which I'll show you now.

47. 9:45

    Okay. So I've opened up the JSON file, and it has a bunch of text and also details around the exact tokens. It also will have the segments of, uh, the transcription.

48. 9:57

    And what I recommend doing is just take a quick read through it. I know you won't be able to see 'cause of the font size here, but take a look through for any words that you think might be misspelled, and what you can do is a quick find/replace on some of those variables.

49. 10:11

    So, for example, often, uh, I'll see the word Trelis with two Ls, 'cause that's how the correct English word is spelt, but the word, uh, Trelis for the name of Trelis Research is one L, and so I'll often just do a find/replace, uh, for Trelis and put in T-R-E-L-I-S, and then I'll replace that throughout.

50. 10:29

    Now, I've already done that, so it's, uh, completed, but if you want to make those fixes, what you can do is do the find/replace and then re-upload the file to Google Colab.

51. 10:40

    Just re-up- upload it here, uh, so that you can read it in and use that corrected transcript then as the basis for your fine-tuning data set. So at this point, what we've done is we've grabbed the audio from YouTube.

52. 10:53

    We have transcribed it. We've corrected it then manually, and we're ready now to split what's a long, uh, transcript consisting of short segments from Whisper. And what we wanna do is combine some of those segments so that we have longer chunks, up to about 30 seconds in length.

53. 11:09

    Uh, we're gonna use a sim- simple algorithm. We're simply gonna stack them until they reach more than 30 seconds, and then we'll move to creating a new row of data.

54. 11:17

    So what we ultimately want is rows of data that are max 30 seconds in length.

55. 11:23

    So that's what we're gonna do here. I'm gonna just log into Hugging Face so that I can push a data set up to Hugging Face. Now, I've done this earlier, so I'm not gonna, uh, log in right now.

56. 11:34

    I'm gonna define the organization. So if you've created an account, you can create an organization. Mine is [REDACTED:username]trelis. I'll create a u- a repo name, and I'll create the maximum length I want for the rows of data.

57. 11:48

    Then I will grab the source audio, which should be saved from running the YouTube extraction earlier, and I'll also specify the path to my JSON file, which could be an updated JSON file with those corrections.

58. 12:00

    Then I'll load that JSON file, and I'm now going to run through basically taking the segments here And adding them together until we get to the maximum duration. And when we're at the maximum duration, we'll call that a row of data, and we'll move then, uh, to the next data set.

59. 12:19

    So with that done, as you can see, uh, here, I've generated forty-one clips. Uh, I've pushed it up to YouTube if you-- Or sorry, pushed it up to Hugging Face.

60. 12:29

    If you don't wanna push it up, just set this to false, and yeah, my data set should now be available.

61. 12:35

    So if you're following along, it will take a bit of time. You might need to pause the video because you need to run the transcription. Should take a few minutes at most, even if you have, like, half an hour or an hour worth of video.

62. 12:47

    And then you should find your data set up here.

63. 12:50

    And you can see my snippets of text, and then you can see the sound. So I could, uh, for example, play some of these sounds and test them out.

64. 12:59

    Now, this is kind of a basic implementation, what I'm showing, but you can see already an opportunity for improvement. Here,

65. 13:07

    right now I'm adding segments of text, but I'm not being careful as to whether the segments are ending. It would be better if I ended every row of data here on a full stop.

66. 13:18

    That could be done by using a library such as NLTK to detect, uh, sentence boundaries, or you could even use just a regular expression. So there is room for improvement of how these data sets are being compiled.

67. 13:31

    Shouldn't make a huge difference, but generally, you want to have kind of paragraphs that are complete within a row of data here. All right, so at this point, hopefully you have your data set, uh, pushed up and you've noted what your org name and your data set name is.

68. 13:45

    So we're ready to move on, uh, to fine-tuning. Now, the first thing for fine-tuning is to install Unsloth. Uh, so I've actually run this, and I have run it just, uh, earlier this morning, so we're gonna be able to follow on with these cells.

69. 14:01

    We're installing, um, Unsloth here, which includes Transformers and a number of packages for loading and fine-tuning models. Next, with Unsloth installed, we're gonna load the base model that we want to train, which is CSM-1B.

70. 14:14

    This is the Sesame model. We're going to load it with a maximum sequence length. This is the maximum tex-text length, which is probably quite a bit longer than thirty seconds, uh, which it should be.

71. 14:25

    We will leave the data type as none. Yeah, the data type, if I'm on a T4, will be, uh, float sixteen. So it'll be float,

72. 14:34

    uh, Torch.floatsixteen on T4, and it'll be, uh, Torch.bfloat sixteen

73. 14:43

    on, um, anything more recent than that, so on Hopper or Blackwell GPUs.

74. 14:50

    Also Ampere as well. The auto model is CSM for conditional generation, so Sesame models are now supported by Transformers, and that's what U-Unsloth is leveraging here. You could quantize the base model, but it's not necessary 'cause CSM-1B is just a few gigabytes in size, and the T4 is fifteen gigabytes, so it will fit easily, uh, within this

75. 15:10

    here. So the model is gonna download, and it will then be loaded into the GPU. When the shards are loading, you should see that, uh, the GPU should start to get full with the shards.

76. 15:21

    Now, this here looks like my GPU usage is kind of high. It might suggest I've loaded the model twice, uh, so I may have to restart my runtime and reload it again.

77. 15:32

    Uh, but we'll see. It's, it's lowering down now, so maybe that's okay.

78. 15:36

    And what you can do then is print out the model architecture just to take a look. And yeah, there's quite a lot in the CSM model. There's

79. 15:47

    the backbone model. This is the main Transformer model here. And then there's the depth decoder, so this is where you're decoding out those thirty-one other tokens. And then there's also the model that allows you to convert from the waveform into tokens and tokens back to the waveform.

80. 16:03

    So you've got the codec model, and there's also the decoder down here as well.

81. 16:08

    So this is the architecture. It's relevant to print the architecture because actually we're not gonna train all of these parameters. We're just gonna train a subset. We're gonna focus on the linear layers.

82. 16:18

    And actually, we're not even gonna train all of the parameters in the linear layers. We're gonna train adapters that plug on. So that's gonna save memory and also make the training faster.

83. 16:27

    So you'll just take note of some of these modules here. These are the attention modules that we're gonna train. These are the linear layers. We're gonna train these as well.

84. 16:35

    Okay, so that's the model printed out, and we're ready now to apply those adapters I just mentioned. So apply little adapters to train and apply them to these modules here.

85. 16:46

    I'm gonna come back though before I apply this because I'm gonna skip rudder because I want to see how the base model is performing without any fine-tuning. So we're gonna skip the data preparation just for now.

86. 16:58

    We're gonna skip the training, and we're gonna run straight to inference just to hear what some of these sounds are like.

87. 17:05

    So, um, yeah, let's start off and just pass in a piece of text here saying we just finished fine-tuning a text-to-speech model. We didn't finish 'cause we're just starting, but let's run this here and see, um, what it sounds like.

88. 17:20

    There's an alternative way of running here. Basically, here we're using a chat template to apply to text that we're passing in, whereas here we're preparing, uh, the tensors in a more raw format just by passing in, um, the speaker ID and text here.

89. 17:35

    So essentially, we're bypassing the chat template, whereas here we're using the chat template. Just two different ways to call the model. So this is just a basic called zero-shot inference.

90. 17:44

    You're passing in text, you get out audio. You're gonna get a random speaker's voice here because, uh, the temperature is non-zero. So you could get female, you could get [REDACTED:gender], you could get deep, you could get high-pitched.

91. 17:55

    Uh, you could really get any speaker. And one way you can avoid that is by doing what's called voice cloning. So with voice cloning, which is different to fine-tuning, with voice cloning, we will pass in a sample, and we will then get the model, uh, to speak.

92. 18:11

    So basically, we pass in audio And then we get the model to generate audio, and the model will tend to generate audio that sounds more like the audio we passed in.

93. 18:20

    This is called voice cloning, and we're just gonna use some of the data set that we already created for this purpose. And yeah, that data set will get passed in, and then we will ask the model to speak this text here, and this should be quite a bit closer to my voice than the first example because we've

94. 18:40

    passed in a sample. Uh, but it still won't be as close as we get after fine-tuning. Now, to be able to run this, I do need to have loaded the data set, so I need to r- load the raw data set.

95. 18:51

    Uh, so for that, I'm actually gonna go up here to the fine-tuning. I've got Unsloth, uh, data prep. I'm not gonna... Yeah, and then I do need to run the loading here.

96. 19:02

    So I will just run the data preparation script here so that I load in my raw data set from my YouTube TTS. Just a note that when we load the data set, we do need to have a data set with an audio column and a text column.

97. 19:17

    Optionally, you can have a source column, and the source would refer to the speaker number, which would start at zero. So if you have two speakers, it could be zero, and then it could be one for a different speaker.

98. 19:27

    Um, if you don't have a speaker column, what this code here is doing is simply assigning a speaker zero. So we'll just, uh, if there's no speaker found, it will use the default of source, which is zero.

99. 19:41

    So it will apply a column effectively of source being equal to zero. Okay, so I'm just loading the data set here

100. 19:49

     because we need to have some data if we're gonna clone. And yeah, the message is showing here that the speaker is being set to zero for each of the rows, which is good 'cause it's a video where I just have one speaker.

101. 20:01

     And now it's just checking the maximum audio length, which we're gonna need when we set up the trainer later. So we configured a trainer to also have, uh, that maximum audio length.

102. 20:12

     But while that's running, uh, we can just go down so that we're ready to run the cloning script, and this is just going to run inference with cloning.

103. 20:23

     And I do have an issue here. I need to comment that out. And the cloning is now underway, so we're basically gonna pass in that, well, a sample of my voice, and then we're going to try and generate text, uh, to say this right here.

104. 20:39

     So in the meantime, I've already generated, um, a sample without the cloning, so let's just play that.

105. 20:46

     We just finished fine-tuning a text-to-speech model-

106. 20:50

     Okay. So yeah, you can hear it's, um, it's a woman's voice. Um, and as I said, it can be unpredictable exactly what voice you get based on temperature. And here's another sample.

107. 21:02

     Sesame is a super cool TTS model which can be fine-tuned with Unsloth.

108. 21:07

     So yeah, you can see that there's a very wide variance in the type of speaker that you're gonna get if you just do zero-shot inference.

109. 21:15

     I have got a sound file now of the cloned audio.

110. 21:19

     Sesame is a super cool TTS-

111. 21:22

     And here is that audio

112. 21:24

     ... model which can be fine-tuned with-

113. 21:27

     And here's that audio.

114. 21:28

     Sesame is a super cool TTS model which can be fine-tuned with Unsloth.

115. 21:36

     So it's not too bad. My voice has quite a bit of drag on it.

116. 21:39

     Sesame is a super cool TTS model which can be fine-tuned with Unsloth. Sesame is a sup-

117. 21:47

     So it's definitely closer to my voice, but not as good as we can get with fine-tuning. So what we're gonna do now is go back up to the start, uh, and we'll continue on after having loaded the model to apply LoRA adapters and then run through the training, and we'll see if the fine-tuned model performs better than

118. 22:03

     just using cloning on its own. So I'm gonna scroll back up here to the point where we loaded the model.

119. 22:11

     The model is, uh, is still loaded. Um, I'll just rerun this to kinda make it more compact. And now we're gonna apply those adapters. So we're gonna target the linear layers, the QKV-o, and then also the MLP linear layers here, gate up and down.

120. 22:28

     We'll use a LoRA alpha of 16. It's a pretty small model. I usually recommend 16 if the, if it's about one billion in size, and we will use, uh, rescale_lora.

121. 22:37

     This means that the learning rate for the LoRA adapters will be scaled according to the size of the adapters, because if you have larger adapters, you need to train them more slowly and vice versa.

122. 22:48

     So we'll go ahead with this. We'll use a rank of 32. That means, um, the rank, kinda like the width or the, the height of the adapter matrices.

123. 22:57

     You can make that larger if you want more granularity, but 32 should be fine. And with that, we will apply the adapters, and we should get a printout of the number of trainable parameters.

124. 23:07

     So yeah, I think we've already applied everything there. Um, it's not actually showing us the trainable parameters, but we could probably do that by just running print trainable_params. And yeah, so print trainable parameters.

125. 23:25

     You can see that we've got about just under 2% of the parameters are trainable. Technically, you could train more parameters, say, by training the embeddings. If you want to do that, you would add in here modules, uh, modules to save, and the ones that you typically would want to save or that you would typically want to train

126. 23:44

     are LM head and LM embed or embed tokens. But, um, because we're not changing the embeddings, we're not changing the tokens here, there's not really any reason to train them, so just applying adapters to the linear layers, uh, should be plenty.

127. 24:00

     Okay, so we have the adapters applied, and we're ready to move to training. We've already loaded the data set, uh, the raw data set. We've set the source to zero because we've just got one speaker.

128. 24:11

     There's a little more formatting we need to do though. And for that formatting, basically preparing the data for the trainer, we do need to know the maximum audio length.

129. 24:19

     We've already got that printed here. It's about 700,000, um, audio steps,

130. 24:27

     uh, audio, uh, tokens in length. Sorry, not tog- tokens, but it's the length, uh, of the audio. Then we need to measure the length of the text, so I'll just run that cell here, and the max is at 587.

131. 24:41

     And we're gonna pass that max text length and the max audio length into the trainer down here. So you can see when we're preparing the inputs, we need to prepare text keyword arguments, and we need to prepare audio keyword arguments.

132. 24:53

     Uh, we need to set a sampling rate, and we do need to set the text length and the audio length right here.

133. 24:59

     So I've got this set and Unsloth is doing the work to prepare the input IDs, the attention mask, the labels, the input values, and the cutoffs. So all this is, uh, set up nicely for us per the original Unsloth notebook, and we just need to apply this mapping now, uh, to our pre-processed examples to get a processed

134. 25:17

     dataset. So I'll go ahead and do that now,

135. 25:20

     and then I'll be able to print out the dataset, which will show the number of rows at the input IDs and, uh, these other columns here. Next, we're ready to train the model so we can pass that processed dataset, uh, into the trainer.

136. 25:34

     You can see that's being passed in here. The model is being passed in here. We've already applied the adapters. We will use, um, a virtual batch size of eight.

137. 25:43

     You probably can go a bit bigger within the T4. Um, I usually would like to have this of 32 in size, so the product being 32, but we don't have that much data, so we won't go with that large of a virtual size.

138. 25:55

     We'll go with eight. With 41 rows, there'll be, uh, five different steps for every epoch of training. We'll just train for one epoch.

139. 26:04

     Um, the warm-up steps of three, this means that we're gonna increase the learning rate slowly. To be honest, three is a bit much if there's only five, uh, in terms of total steps, so you could just reduce that to one maybe.

140. 26:16

     The learning rate, 2E minus four, is probably good for a one billion parameter model. This will automatically decide whether to use Float 16 or Brain Float 16, depending on the TPU that's being used.

141. 26:26

     Uh, we'll use AdamW 8-bit optimizer to reduce memory requirements a little bit. Uh, weight decay prevents overfitting, probably not essential. We will train at a constant learning rate,

142. 26:38

     and we will save our outputs here, uh, to the output directory. So with that, I think we can, uh, set up training, check our current memory stats. So we've got almost 15 gigabytes in the GPU, and nearly seven of them are used up.

143. 26:53

     And we can start our training, and it's gonna take about, uh, 10 minutes if you run the training. And what you'll find is during that training, uh, the loss will fall from roughly 6.34, uh, down to, you know, somewhere around 3.7.

144. 27:10

     It looks like the loss is still falling, so probably there's potential for improvement here, uh, in terms of the quality of the outputs. Um, maybe it'll help a little that I reduced the number of warm-up steps because that means we'll have a bit higher of a learning rate.

145. 27:26

     So I'd expect if I run this now, I probably should, uh, do a little bit better. Let's get the training going.

146. 27:33

     And when the training is done, we'll print out the final training stats. You can see with that batch size that I specified, the virtual size of eight or the actual batch of two, there's only half of the memory being used.

147. 27:43

     So you probably could increase up to a batch size of four without a problem. So I would run this, and then what we'll do is run the inference again, uh, but this time on the fine-tuned model and see how the performance compares.

148. 27:57

     And what we'll do, um, what will happen automatically is we will get the saving of some of these, um, of some of these sound files, so we'll be able to just play back the zero-shot example here.

149. 28:09

     We've got the zero-shot audio and the cloned audio on the base model, but now we're gonna generate, uh, voice samples using the fine-tuned model.

150. 28:18

     So I'll just run this here, run all of these cells so that when the training is done, and then run the, uh, cloning test for when the training is done.

151. 28:29

     Training should be underway here now. If I scroll up, you can see we're at, uh, two out of 60 steps. Note that there's no eval dataset. Ideally, you would split your training set into train and eval, and you would inspect the eval loss, uh, make sure that that's falling.

152. 28:44

     If it stops falling, that means you probably need to stop training. Either you've gone too far in epochs or maybe your learning rate is too high. You also typically want to monitor the Grad Norm, make sure it's around one or less than one.

153. 28:55

     You would do that typically by adding in here, uh, TensorBoard. So you would report to TensorBoard. You also need to set a log directory. So I think it's logging dir that you need to set here.

154. 29:07

     Dir. Typically, you would set it to something like logs.

155. 29:12

     Uh, but I'm gonna leave this because we're just looking at a basic setup for now. But if you do that, you can pip install TensorBoard, and then you can look up, uh, how to monitor the losses.

156. 29:21

     Now you can see that in the longer video I already published. I'll probably publish a more detailed video after this, uh, too, on the us- on the Trelis Research YouTube channel.

157. 29:30

     So training is going on. It looks like our loss is falling, which is good. And while that's happening, I'll just explain once, uh, the training and the inference is done, you have the option to save the model and also push it to Hub.

158. 29:42

     If you just run this cell here, it's gonna save a copy of the model locally, so it'll be in your temporary storage on Colab. If you want to push it to Hub, you need to set one of these to true, uh, both of them to true because you need the model and processor.

159. 29:55

     This will just push the LoRA adapters, uh, which is lightweight. It'll be a smaller repo. But if you want to push the full model, uh, you need to merge it first, and that's where, uh, this comes into play here.

160. 30:06

     You can save the model in merged form. Um, well, this is just saving again in, as LoRA adapters, but if you want to merge it to 16-bit format, you can set these two to true, and you can save the model and also push it to Hub.

161. 30:18

     And I believe these will actually save both the model and the processor as well. So if you want to push to Hub, yeah, you just set this one, uh, here true.

162. 30:27

     It will merge the model and then push it up to Hub. By the way, if you do want to reload, um, a model that you have previously created, you would do that right back up here where we loaded the model.

163. 30:38

     Instead of specifying the base model name, you would specify the, um, the fine-tuned model name, which in my case is going to be trellis my-youtube-tts. Okay, so training is done.

164. 30:50

     We've gotten down to 3.72 as loss and used just under half of the memory available. Next, uh, we'll run inference using this fine-tuned model. So actually, this is run.

165. 31:00

     We should have a sample here. Uh, we should have a second sample here, which we'll listen to now, and then we have a cloned example, which we expect, uh, should be the best.

166. 31:09

     So let's give a listen to the fine-tuned model here. We just finished fine-tuning. Let's have a listen. We just finished fine-tuning

167. 31:17

     a text-to-speech model. It's pretty good. So it sounds a little bit like me, but, um, yeah, definitely it's a [REDACTED:gender] voice, so that randomness of it being any kind of voice has been removed, and it does sound a little bit [REDACTED:origin].

168. 31:33

     You can see the pacing is not quite right. That perhaps would be improved by better filtering the original data set so that I don't have segments from my YouTube video where there's just a lot of pause.

169. 31:44

     Now, there isn't a lot of pause anyway in my YouTube videos typically because I cut those out, but you could better do some filtering on the data if you want to improve that.

170. 31:52

     Let's see a second sample though because there is some randomness. Sesame is a super cool TTS model which can be fine-tuned with Unsloth. So it's a little bit of an [REDACTED:origin] accent.

171. 32:03

     Probably indicates more data is needed to improve the quality here. The T in tuned is still a little bit American, although it's kind of slightly [REDACTED:origin], so some room for improvement here.

172. 32:14

     But what we see now is probably the best chance we have, which is using the cloning plus the fine-tuning. So I'll play this sample now. Sesame is a super cool TTS model which can be fine-tuned with Unsloth.

173. 32:28

     Okay, so that was great. It even has the [REDACTED:origin], uh, tuned. It even kind of makes this little bit of an error here, which actually sounds natural. Let's give it one more listen.

174. 32:36

     Sesame is a super cool TTS model which can be fine-tuned with Unsloth. Yeah. Now, maybe the super cool, that's not something I would say, so maybe it hasn't got my tonat- uh, my intonation correct.

175. 32:49

     But broadly speaking, this, uh, this sounds pretty great.

176. 32:53

     So at this point, we've pretty much finished the workshop. Uh, you should have a sound ideally with the cloning that sounds something more like what you do or what the person in the video you chose sounds like.

177. 33:04

     If you wish, um, you could create more data, aim for maybe 500 rows of 30 seconds, and I think you would see better performance, particularly without the voice cloning.

178. 33:14

     But you can see here that if you combine fine-tuning with cloning, you're able to get pretty good performance even with a relatively small amount of data. Uh, just say I think a video of 30 minutes was enough in this case here.

179. 33:26

     So that rounds up the workshop. As I said at the start, you can get all of the resources on GitHub. That's, uh, TrelisResearch/ai-worlds-fair-2025. I will probably make, uh, I will make future videos on voice that you can check out on the Trelis YouTube channel.

180. 33:41

     Probably some more detailed videos covering in depth, a little more depth, the data preparation and some of the hyperparameters around the fine-tuning itself. Uh, in the meantime, let me know if you've any questions by dropping some of your questions or comments below in the YouTube comments.

181. 33:57

     Cheers, folks.