While my guitar gently speaks

Talk transcript

1. 0:00

   [upbeat music] So I am Todd Fisher. I love the guitar. That's one of my passions in life.

2. 0:16

   Uh, today I wanna talk about, uh, this project I've been working on, uh, for a while here, effectively making my guitar speak. Uh, but of course, I wanna start out with kinda framing it under this awesome premise.

3. 0:28

   You know, we've all been to live performances where, like, your mind would just be blown. It was awesome. Uh, you know, the, the first one I remember, way back, uh, I was in high school.

4. 0:36

   I went to a Slipknot concert, so a little bit heavier music, uh, here in the Bay Area. And I remember at some point, the drummer was drumming a cool drum solo, and his drum set started raising up.

5. 0:46

   And I was like, "Whoa, that's cool," and everyone k- got kind of excited, right? And then halfway through the solo, his drum set started to go like this, and tipping.

6. 0:54

   And his whole drum set was, like, on the wall while he was upside down playing his drum solo, and everyone was just cheering, like, "Whoa, this is so crazy.

7. 1:02

   Mind-blowing." Like, whoa. It was so, so impactful, right? I still rememb- remember it today. Um, fast-forward to probably about a year ago, uh, I went to New York to see the Stranger Things on Broadway, uh, the, The First Shadow, and it was really cool because the whole time I was there, I was watching it, and it felt

8. 1:19

   like it was actually like a, a, a Netflix episode but in real life. And it was just like, the effects that they did there was so, so amazing. I remember even one time at, near the end, there was a scene where somebody, somebody was falling backwards a- and it was, like, a slow motion scene, and it probably

9. 1:35

   about a minute long, the whole scene. But just, like, the fact that, like, they, they were able to produce that in real life, but it looked like it was a post-production slow motion scene, just, just mind-blowing.

10. 1:45

    So I was like, "Whoa, that's cool." So given you, everyone here has probably had some experiences, uh, with live performances in the past, and I think it's really awesome just to, like, see the creativity, uh, that people can do, uh, with leveraging technology in some creative way.

11. 2:00

    And so, you know, I'm thinking about all the projects that we have as, as engineers over the years where it's like, "Hey, that's a cool idea. I'm gonna write this down."

12. 2:06

    I have, like, a big, giant list. Anyone have a list of projects that maybe they'll get to? Yeah. So it all happens, right? Uh, and it's really awesome 'cause the last six months or so with AI, uh, being able to, you know, push us forward with some of these projects, it's just really cool to see where we

13. 2:21

    can take all these projects. So, so part of, uh, my goal today is to inspire you guys to find whatever project you're passionate about. Go start building it because it's super easy now with AI.

14. 2:31

    It's easier with AI. I know sometimes there's caveats. But, but in general, uh, I, I want to have you guys leave this session being inspired to go build whatever that cool thing is for you personally that's gonna, you know, help you to learn stuff or maybe make a difference in the things that you're, you're trying to figure

15. 2:47

    out. Uh, so I wanna start off with framing this kind of in the realm of the guitar. So the guitar, 'cause the guitar, guitar's been around for several years, several thousands year, or hundreds of years, sorry.

16. 2:58

    Um, and at some point, somebody said, "Hey, I'm gonna put a pickup on a guitar. I'm gonna plug it into a speaker," and now we have rock and roll.

17. 3:05

    We're gonna make it super loud. That was pretty awesome. Uh, and then of course, we have a lot of people getting these stomp boxes, these, these effects pedals. We put them all together.

18. 3:13

    We have really awesome sounds. You know, really famous people have a lot of these. Uh, and then at some point, Peter- Peter Frampton, uh, came along and said, "Hey, what if we get the sound of the guitar, put it through an actual hose, like a physical hose, put it in my mouth, and then basically play the guitar

19. 3:29

    sound in my mouth a- and, and form some sort of words?" So that's pretty awesome. That, that's known as the talk box. Uh, and then in the last couple decades, we have a lot of progress made with software emulation.

20. 3:40

    Uh, you think of, you know, Pro Tools. You think of Logic, of, uh, Fruity Loops. Uh, uh, there's a lot of software, even to the point where all the effects are in software now.

21. 3:48

    There is the argument made, hot take, uh, that maybe you don't need all those physical effects pedals anymore. So, so there's that question, right? And then of course, looking forward, it's what is our next evolution of the guitar, uh, with AI in the picture?

22. 4:03

    So with that said, I wanna take you back several years ago, at Halloween time. I was passing out candy, bunch of trick-or-treaters. You know, it got kind of boring, so I was like, "Hey, what if I go and actually bring my guitar out with my amp and just play?"

23. 4:15

    Uh, so for the past, what, 10 or plus years or so, I've actually been playing guitar on Halloween passing out candy, and it's kinda fun. Uh, and then fast-forward probably three years ago, I decided to dress up as Eddie Munson.

24. 4:27

    Anyone know Eddie Munson? Stranger Things guitar player guy. So pretty awesome guy, so I dressed up as him, and I was like, "Yeah, this is gonna be so fun.

25. 4:34

    I could play the music with him," whatever, and you kind of go the... Oh, hold on.

26. 4:42

    Hold on, wait a minute. [guitar strumming] Technical difficulties. [guitar strumming] All right, there it goes. [upbeat music]

27. 4:56

    So a bunch of little Stranger Things awesomeness, right? Uh, but figured out, "Hey, what if there was, like, some more stuff I could put i- in this whole experience?"

28. 5:03

    And so I decided to go and build a little app that would actually paint the Stranger Things alphabet on my garage door just because it's fun, right? Uh, and so let me just show you a quick example of how this worked.

29. 5:14

    So effectively, make sure I have the mic permission. Whenever I play a note, [guitar strumming]

30. 5:22

    it would go and communicate whatever lights, and I even had it to where I could actually, uh, set custom messages such as Happy Halloween, all that fun stuff. It was, it was overall pretty fun, uh, just to kinda mess with that, right?

31. 5:33

    So [guitar strumming] once again, you know, whatever I type in there, I could actually spell whatever a- and, and very much a, a, a cool nod to Stranger Things. But it kinda got me thinking, like, "What is the next evolution of this project?"

32. 5:46

    And I settled on this idea of, like, hey, how hard would it be to make my guitar speak?

33. 5:51

    It sounds easy. Maybe, maybe not. So, so today I want to share kind of my journey, uh, in this process, uh, and where kind of I'm at today. Uh, so looking around the different tools, there's a framework out there called JUCE, uh, really awesome for anyone building audio software, look into JUCE.

34. 6:06

    It's, it's pretty good. Um, and then, of course, there's a, a number of plug-ins, uh, uh, sorry, plug-in formats out there, uh, for your, your digital audio workstation. And for, for those that are not aware, uh, your DAW is effectively your IDE, but for musicians and, and music producers.

35. 6:22

    And then I, I started with some text-to-speech stuff with Piper and some built-in Apple stuff, and then some other really fun, uh, digital signal processing. Uh, so with that said, my first stab here was I want to get raw text, so I can just type in whatever text I want, uh, push it through the text-to-speech, uh, get

36. 6:39

    the audio clip, and then whenever I play a note on the guitar, I want to go and play that back. So let's go see how that works. So switching over to my Logic Pro here,

37. 6:50

    and this is the plug-in I made. So it's just like any other plug-in in Logic, where you just plop it in there. It's just, you know, chaining all the effects together.

38. 6:59

    And so this is what I came up with.

39. 7:02

    Developers.

40. 7:04

    So it's playing-

41. 7:04

    Developers.

42. 7:06

    Pretty awesome, right?

43. 7:07

    Developers. Developers.

44. 7:08

    Oh, it kinda-

45. 7:08

    Developers

46. 7:09

    ... kinda reminds me of something, right?

47. 7:10

    Developers.

48. 7:11

    Start clapping, everyone. Start clapping.

49. 7:13

    Developers.

50. 7:14

    Developers.

51. 7:15

    Developers.

52. 7:16

    Developers.

53. 7:16

    Developers.

54. 7:16

    Developers.

55. 7:17

    Developers.

56. 7:18

    Awesome. Thank you. That, that's pretty awesome. Uh, you guys are great. Um, so I got it to where it's playing an actual audio file. That's pretty awesome, but it turns out, you know, in English or in any languages for that matter, there's more than one word.

57. 7:31

    So the next kind of evolution-

58. 7:32

    Developers

59. 7:32

    Oh, it's a little bit chatty. Uh, the next evolution there is let's actually go and slice it per word now.

60. 7:39

    So I got to the point where it's playing. Now it's gonna slice per word.

61. 7:43

    Look at me. I can speak. Look-

62. 7:50

    Oh

63. 7:50

    ... at me. I can speak.

64. 7:53

    So now it's speaking words, and that's pretty awesome, right? Uh, but it turns out that, uh, as we speak, there's a number of challenges in how we automatically slice words.

65. 8:03

    And so I looked into this thing called energy gap segmentation. The general idea here is if you look in, uh, any waveform over here, we see that here's a bunch of, of words that we're speaking, right?

66. 8:14

    Uh, the idea there is there's typically silence in between words, so let's just cut it whenever the decibels are, are very much close to zero, right? Uh, but the issue with that is there's actually sometimes when, you know, as I'm speaking right now, for example, uh, there's actually no silence in between some of my words, so it

67. 8:30

    gets a little bit challenging to where it's not 100% foolproof, right? So beyond that, I looked into this thing called sonority peak syllabi- syllabifier. That's a hard word to say.

68. 8:40

    Uh, effectively identifying the syllables of the audio signal and, uh, identifying that there's vowels in here. Vowels typically lead to syllables. That's kind of the idea. So I said, "Okay, let's take the sonority peak, uh, add it to the, uh, the energy gap, and figure out if we could just make that work automatically."

69. 8:58

    So with that said, I made it kind of work, so let's just play this one.

70. 9:02

    Thank you for letting me be-

71. 9:05

    Oh. Oh

72. 9:06

    ... here. It feels so good to get out of one's guitar case

73. 9:19

    once in a while.

74. 9:23

    So there you go. It's working pretty well. Not quite a- as good as I want to, so long story short, uh, I settled on just the, uh, ability to go and actually I could drag this and manually edit some of these, uh, uh, segments in here.

75. 9:36

    So worked okay, right? Um, but moving on, you know, uh, with the spirit of evolving the thought, evolving the, the project, right? It's like, "Okay, we're, we're having the AI say stuff.

76. 9:46

    That's great, but what if we could actually make it sing?" Let's take it to the next step 'cause this is music. Why not, right? Uh, so looked into pitch detection, and for those that are not aware, uh, as you hear any, any noise out there, there's typically multiple frequencies going on at any given time.

77. 10:02

    You know, you think of when you play the, the C key on the piano, uh, there is definitely a fundamental frequency or the, the kind of the, the one that we identify as the note, but there's a bunch of other frequencies.

78. 10:12

    So we needed a way to go and figure out, like, how do we detect that fundamental frequency? So when I p- press something on the guitar, how do I translate that into an actual note?

79. 10:21

    Uh, and then I found this, uh, YIN pitch algorithm. Basically what it does is it detects the pitch. Uh, I won't get into all the details, but look it up.

80. 10:29

    It's kind of a, a fun, uh, really cool way of detecting the pitch. Uh, but effectively what I do is I play the guitar, uh, I detect the, the pitch.

81. 10:39

    I, I make what's effectively a pitch sawtooth or AKA a synthesized note. Uh, so for those that are not familiar with how audio works on the, the computer, you think of all electronic music, all that stuff, uh, that is basically, uh, a synthesized note.

82. 10:53

    We have ADSR, which effectively are the levers to figure out how to actually make the note sound in different ways. Uh, and then so basically we get the pitched notes or the sy- synthesized note.

83. 11:04

    Uh, we then push it through, uh, the, the voice clip effectively. So think of the talk box. Uh, we're kind of filling up the cavity of the talk, uh, of the voice that is, and we push it through a vocoder, and it should sing.

84. 11:16

    So that's kind of the idea, right? So with that said, let's go ahead and jam out a little bit because I have a guitar and it's fun. So

85. 11:24

    let me get back to Logic here. Uh, so I'm gonna play some chords. Uh, I originally was gonna, uh, play a song that is very much related to the talk, or the, the title of my talk, uh, While My Guitar Gently Speaks.

86. 11:41

    But because this is gonna be posted online, I don't wanna muddy up the waters with any copyright things. So I will play some chords that may or may not sound similar to a famous song.

87. 11:51

    So-[guitar playing]

88. 12:31

    So that's the backing track, so let's go ahead and have some AI speaking on top of it. [guitar playing]

89. 12:41

    I'm here at this-

90. 12:47

    Oh. Hold on, technical difficulties. Let's try that again.

91. 12:52

    Doo, doo-doo doo doo. Live demos, always the best. [guitar playing]

92. 12:54

    I'm here at this con- fence. [laughs] [guitar playing]

93. 13:09

    All right, technical difficulties again, so sorry for that. But let's just run through it and see what happens. [guitar playing]

94. 13:19

    I'm con- fence that speak. I'm a guitar that can speak,

95. 13:30

    so...

96. 13:32

    Awesome. So there you go. S- some bugs to work out, but overall, we are able to, to say words on top of this. And for what it's worth, I'm trying to mix the, uh, the t- the synthesized note, uh, with this clarity lever right here.

97. 13:44

    So bal-- Or mixing the, the synthesized note with the actual voice that the AI is giving. Uh-

98. 13:50

    Can-

99. 13:50

    So, so that worked pretty good. Um, but there's also this other question, you know. Given that, uh, when you speak, it's typically conversational, what if I could take this to the next level, and what if we actually had this microphone right here where I could speak into the mic, it would then respond on the guitar?

100. 14:08

     Uh, so the way that I c- accomplish this is speak into the mic, use, uh, speech-to-text, so Whisper. Uh, put that into raw text, run a local model on my computer.

101. 14:18

     Uh, it doesn't really matter what LLM, just any local model, have a conversation coming, and then from that output, go and plop it on the guitar. So let's see how this works.

102. 14:31

     Hmm. Anyone have a question you wanna ask my guitar?

103. 14:39

     Anybody? What is reality? Let's do it. What is reality?

104. 14:50

     So it's thinking, going through the LLM things,

105. 14:54

     and let's see what it says. [guitar playing]

106. 14:56

     That's quite a question to start with. [laughs] How does your music help you understand that elusive concept of reality right now?

107. 15:18

     So yeah, very existential. I like that, so thank you for that suggestion there. S- so there you go. We have a choppy version, but, but it is working, so, so that's a win, right?

108. 15:26

     Uh, awesome, and thanks for the claps there. Um,

109. 15:30

     but really like, it's not quite singing yet, right? So moving on in this project, it's like, how do I actually make it sing, right? Uh, I found a lot of, uh, open source options out there as far as, uh, uh, recordings or, or samples, if you will.

110. 15:43

     Uh, there's a thing called Vocalset out there. They basically recorded a bunch of singers, and I could use those audio files to go and do some fun stuff with them.

111. 15:50

     And effectively, I took those samples. I used, uh, a project called World, which helps with things like pitch shifting and some other things, and I'm able to then shift the, the pitches of everyone singing on those audio clips and map it to the guitar.

112. 16:05

     Uh, this is a very heavy process, and so I can't really do that live. I had to effectively pre-bake that, and then once it's already pre-baked, I could then go and jam out with it.

113. 16:14

     So with that said, uh, let me go ahead and show that example here.

114. 16:19

     Uh... So, uh, because it takes so long, I actually just started with the vowel sounds. So there's somebody singing all... Each of the five vowels, and so we'll see how that sounds, right? [singing]

115. 16:41

     Kinda fun, kinda weird, but overall, it's working. It's, it's closer to singing, right? Uh, so let's go ahead and throw that on top of all the chords over here, and let's see if we can make it sound decently well.

116. 16:53

     Oh. So we'll just play these chords again. [singing] [guitar playing] [laughs]

117. 17:15

     So not quite your opera singer, but getting closer, and, and so I, I think there's something awesome there. Uh, so effectively, once again, that is, uh, going through, uh, the whole synthesize process, uh, putting a sample, shifting the, the pitch shift of the sample, and then effectively mapping each fret or each, each note on the guitar to

118. 17:32

     one of those samples that are pre-baked effectively, right? Um, so with that said, you know, where do I wanna take this next? Uh, very much wanna get into more of, of the, you know, AI heavier options.

119. 17:44

     Uh, if anyone ha- has any other ideas of how to make the guitar sing, feel free to come up to me after. I think it's pretty awesome. Uh, but more importantly, you know, everyone here, uh, going back to the, the charge to go and, and build some awesome, you know, side project, some passion project of yours.

120. 17:59

     Like, what is that for you? Go and build awesome things because nowadays with AI, we could build so many really cool things, and time is typically not the, the big time suck that, that it once was, right?

121. 18:09

     So with that said, uh, be awesome. Uh, or, or be good to each other, and thank you very much. [applause] [upbeat music]