# While my guitar gently speaks

Todd Fisher

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/E_Txocq-Lrw-while-my-guitar-gently-speaks) · [Video](https://www.youtube.com/watch?v=E_Txocq-Lrw)

## While My Guitar Gently Speaks

A guitar-controlled voice starts with triggering a speech clip, then grows through word segmentation, pitch detection, a vocoder, local conversation and precomputed singing samples.

No audio-programming background is required; the article introduces the signal-processing concepts used in the demonstrations.

### Making the audience wonder how it works

[0:36](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=36s)

At a Slipknot concert Todd Fisher attended in high school, the drummer’s platform began to rise during a solo. Then it tilted, leaving the drummer playing upside down as the audience cheered. Years later, at *Stranger Things: The First Shadow* on Broadway, Fisher watched a performer fall backward in what appeared to be slow motion. An effect familiar from post-production was happening on a live stage. Both experiences left him with the same fascination: technology could make a performance do something the audience did not expect.

That is the starting point for making a guitar speak. Fisher loves the instrument, and he has the familiar engineer’s backlog of ideas waiting for spare time. AI assistance makes some of those projects easier to pursue—easier, he qualifies, rather than effortlessly easy. The opportunity is to take a personal curiosity far enough that it becomes something people can experience.

![Black slide reading “Live performances rock” and “Leverage technology in a creative way,” with a lightbulb icon and the guitarist inset.](https://ai.engineer/api/talk-essay-media/4fa304c63509f73e143deb1dc12b8213a798b26f3d7235f52e8dde94218a1cae)

“Live performances rock” expands into a call to use technology creatively.

### A guitar already has a history of transformations

[2:47](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=167s)

Fisher frames the project as another step in the guitar’s technological evolution. Pickups and amplification made it louder. Effects pedals let musicians chain transformations together to create distinctive sounds. The talk box, illustrated by Peter Frampton, took a particularly physical approach: route the guitar’s sound through a hose into the player’s mouth, then use mouth shapes to articulate that sound into something resembling words.

Software moved many sound transformations into digital audio workstations such as Pro Tools, Logic and Fruity Loops. Whether software removes the need for physical pedals is a matter of preference; for this project, the useful consequence is that another transformation can live in the same software environment. What could AI add to the instrument next?

### First, make notes spell a message

[4:03](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=243s)

The practical setting was Halloween. Fisher began bringing his guitar and amplifier outside while handing out candy, turning it into a recurring performance. Later, dressing as Eddie Munson—the guitar player from *Stranger Things*—gave that performance a theme. After adding the music, he wanted something more of the show’s world in the experience.

He built an app to display the *Stranger Things* alphabet on his garage door. In the browser demonstration, he enables microphone access and plays notes to trigger the alphabet’s lights. A custom-message field lets him enter text such as Happy Halloween and spell it through the interaction. The guitar is already doing more than producing sound: it is driving a second medium.

![Browser demo with three rows of painted letters, colored string lights, a bright light near I, and controls at the upper right.](https://ai.engineer/api/talk-essay-media/6341443e3128728c4ffce7eaccd150b486fba3898bd2c496d9c706aad9c5fa81)

The alphabet-wall demo shows a brightly illuminated light above the letters.

Once typed messages could become illuminated letters, the next question followed naturally: how hard would it be to make the guitar speak them?

### Turn a guitar note into a speech trigger

[5:51](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=351s)

The audio implementation starts with [JUCE](https://github.com/juce-framework/JUCE). A digital audio workstation, or **DAW**, plays a role analogous to an IDE for musicians and producers: it hosts the work and the plug-ins that process it. Fisher experiments with [**Piper**](https://github.com/rhasspy/piper) and Apple’s built-in speech capabilities, then uses a deliberately small pipeline:

1. Enter the text to be spoken.
2. Convert that text into a speech audio clip.
3. Play the clip when a guitar note triggers it.

In Logic Pro, Fisher inserts his plug-in into the effects chain like another audio effect. The first demonstration repeatedly plays the word “Developers,” becoming a rhythmic call for the audience to clap along. This establishes a working connection between the instrument and speech playback, but the unit being triggered is still an entire clip.

### Speech does not arrive neatly divided into words

[7:18](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=438s)

Repeating one word is entertaining, but a sentence needs finer control. Fisher’s next version slices speech so that guitar playing can advance through its fragments. The demonstration now says, “Look at me. I can speak.” That exposes a harder problem: deciding where one spoken word ends and the next begins.

The first heuristic is **energy-gap segmentation**: inspect the waveform for low-energy pauses and use them as candidate cut points. This works when a speaker leaves gaps, but connected speech can run words together without silence. The useful signal is low amplitude or energy, not a decibel reading near zero.

Fisher then adds a **sonority-peak syllabifier**, looking for vowel-related peaks as clues to syllables. The two heuristics supply different evidence:

| Method | Boundary clue | Limitation |
| --- | --- | --- |
| Energy gaps | Pauses in the waveform | Adjacent words may have no pause |
| Sonority peaks | Vowel-related syllable cues | Syllable cues do not directly give word boundaries |

Combining them gives the slicer more information than silence alone. The slide’s sonority diagram, labeled with the letters in *cats run fast*, illustrates why the internal structure of speech matters when choosing cuts.

![Slide with two bullets about phoneme alignment and timing, above a sonority diagram labeled with the letters in “cats run fast.”](https://ai.engineer/api/talk-essay-media/628c0f45b423ab8f992b08e4c4ae0e4830f875a3c2538fec3796a57f6292077a)

“Syllables are tricky!” illustrates phoneme alignment and sonority peaks.

A longer demonstration thanks the audience for letting the guitar be there and jokes about getting out of its guitar case. It works well enough to show progress, but the automatic boundaries still need correction. Fisher adds draggable segment boundaries so he can repair the cuts manually. **Automatic segmentation supplies a starting point; editable boundaries make the result usable for performance.**

### Give the voice the guitar’s pitch

[9:36](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=576s)

The guitar can now control when speech happens. To move toward singing, it also needs to control pitch. A musical note contains multiple frequencies; the target is the **fundamental frequency** associated with the perceived note. Fisher uses the **YIN pitch algorithm** to estimate that fundamental from the guitar signal.

The detected pitch drives a synthesized sawtooth note. ADSR—attack, decay, sustain and release—controls shape the note’s amplitude envelope. Fisher then combines that pitched sound with the voice clip through a **vocoder**. The synthesized note supplies the pitched sound, while the speech supplies its vocal shaping. This returns to the talk-box idea, replacing the physical mouth-and-hose arrangement with audio processing.

To demonstrate it musically, Fisher first plays a backing chord part. He avoids explicitly performing the song suggested by the talk’s title because the recording will be posted online. With the backing part in place, he adds the guitar-controlled voice. Several attempts encounter technical difficulties and produce fragmented words, although a phrase identifying itself as a guitar that can speak comes through.

The result has remaining bugs, and the mix itself is part of the experiment. A **clarity control** blends the synthesized note with the original AI-generated voice. That gives the performer a way to retain more of the spoken voice while introducing the pitched component; it does not, by itself, resolve the fragmented articulation heard in the demonstration.

### Ask the guitar a question

[13:50](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=830s)

Once generated text can reach the guitar, conversation becomes another input path. Fisher adds a microphone and [**Whisper**](https://github.com/openai/whisper) to turn spoken input into text. A local language model produces a response, and that response feeds the existing guitar speech pipeline. He does not require a particular local LLM: its role is to generate the next conversational turn.

An audience suggestion supplies the question: “What is reality?” Fisher speaks it into the microphone, waits for the model, then plays the response through the guitar. The voice acknowledges the scale of the question and asks how the listener’s music helps them understand reality. The output is choppy, as Fisher points out, but the full interaction works: a spoken question becomes a generated reply delivered through the instrument.

### Start with singing, then prepare it for playback

[15:30](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=930s)

Conversation extends what the guitar can say, but Fisher still does not consider the result singing. His next approach changes the source material. Instead of starting with synthesized speech, he takes recordings of singers from [VocalSet](https://zenodo.org/records/1203819). He uses [WORLD](https://github.com/mmorise/World) to process the recordings, including shifting their pitches so they can be mapped to guitar notes.

In Fisher’s implementation, preparing those samples is too computationally heavy to do live, so he performs the work ahead of time. The slide separates grain analysis from resynthesis at each target pitch. **Precomputation moves the expensive transformation out of the performance path:** the prepared audio is ready before the musician needs to play it. This is a constraint of his sample-preparation setup, not a claim that WORLD universally requires offline operation.

![Slide titled “This is a heavy process,” with side-by-side diagrams for analyzing a grain and resynthesizing it at each target pitch.](https://ai.engineer/api/talk-essay-media/9b2f81c7f31efc1ac343f2fc65cdefea3b61a058728d6819f3919e57686a8ae6)

WORLD processing separates grain analysis from pitch-specific resynthesis performed ahead of playback.

Because preparation takes time, Fisher begins with five vowel sounds. He demonstrates the prepared vocals on their own, then plays them over the backing chords. The sound is closer to singing, though he characterizes it as progress rather than an opera-quality result.

The performance mapping is now concrete: each guitar note or fret selects a corresponding sample whose pitch has already been prepared. The instrument controls playback from that bank rather than requiring a fresh pitch transformation for every note. Fisher wants to explore more AI-intensive approaches next, and invites ideas for making the guitar sing better.

That open question is part of the project’s appeal. A Halloween performance grew into an alphabet display, then speech playback, conversation and an experiment in singing. There is still work to do, but each version gives Fisher something he can play. His closing invitation is to choose an equally personal project and build it: AI assistance can reduce the development effort enough to bring an idea off the backlog and into the world.

## Resources

- [JUCE](https://github.com/juce-framework/JUCE) — C++ framework for building audio applications and plug-ins, with examples and build instructions.
- [Piper original repository](https://github.com/rhasspy/piper) — Original local neural text-to-speech project, with a pointer to its successor.
- [Piper current development](https://github.com/OHF-Voice/piper1-gpl) — Local speech synthesis engine with installation instructions, voices, and command-line and programming interfaces.
- [Whisper](https://github.com/openai/whisper) — OpenAI's speech recognition implementation for converting recorded speech into text.
- [VocalSet singing voice dataset](https://zenodo.org/records/1203819) — Professional singing recordings covering five vowels, varied vocal techniques, scales, arpeggios, and excerpts.
- [WORLD](https://github.com/mmorise/World) — Speech analysis and synthesis software exposing fundamental frequency, spectral envelope, and aperiodicity parameters.
- [WORLD's original system paper](https://www.jstage.jst.go.jp/article/transinf/E99.D/7/E99.D_2015EDP7457/_article) — Original research describing WORLD's speech synthesis architecture and evaluation for real-time applications.

## Slides

### Live performances rock

An upside-down drum solo and a stage scene that looked like slow motion frame the speaker’s interest in creative uses of technology in live performances.

[0:31](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=31s) [Dark slide reading “Live performances rock.”](https://assets.ai.engineer/sites/aiecode2025/sha256/0b27c9052b4c80c90fc5fa2ae719a72c9b0620ddac5b7865e29e7bb06dc5aa32.webp)

Live performances rock

Interface text:
app
localhost:3030/2
Relaunch to update
1m

[1:53](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=113s) [The live performances slide adds a light bulb and the message “Leverage technology in a creative way.”](https://assets.ai.engineer/sites/aiecode2025/sha256/bbdbc7b173a7c5f7da6dca88177184eb74c2a2fddf26471f01577baa043ea6ce.webp)

Live performances rock

💡 Leverage technology in a creative way

Interface text:
app
localhost:3030/2?clicks=1
Relaunch to update
1m

### Unused project domains

A cartoon shows an engineer renewing 112 unused domains. The speaker hopes AI will help engineers start the projects they have saved for later.

[2:01](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=121s) [Cartoon of a developer renewing many unused domain names.](https://assets.ai.engineer/sites/aiecode2025/sha256/e1380c49856721bc8b71d045c6535556255751810a3a744a5809b308819182a5.webp)

Another year of renewing 112 domain names I will never use

...coolproject.dev,
ideasignup.io,
coders.life,
side-project-name.com,
the-next-big-thing.ai,
justin-case.net,
what-was-i-thinking.org,
never-used-this.app...

CONFIRM RENEWAL
(112 DOMAINS)
for $1,450.32

Interface text:
While My Guitar Gently Spea[illegible]
app
localhost:3030/3
Relaunch to update

### Halloween guitar

A glowing guitar-themed pumpkin sets the scene as the speaker recalls bringing his guitar and amp outside to play while handing out Halloween candy, a tradition he has kept for roughly 10 years or more.

[4:02](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=242s) [A guitar appears inside an illuminated carved pumpkin outdoors.](https://assets.ai.engineer/sites/aiecode2025/sha256/7acf41d9bd4c094fda736b2e19ec12cc8fea3bec862f2e40df2d84909fd26aaa.webp)

Interface text:
app
localhost:3030/5
Relaunch to update

### Costumed guitarist

After years of playing guitar while handing out Halloween candy, the speaker dressed as Eddie Munson, the Stranger Things guitarist.

[4:24](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=264s) [A costumed guitarist stands with an electric guitar against a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/d33f0cb9f3218d258ba746f50bb35ffc5a5c125138b77069edd36f71bbeb2129.webp)

Interface text:
While My Guitar Gently Spea[illegible]
app
localhost:3030/6
Relaunch to update

### Guitar audio setup

Logic Pro shows several guitar tracks with Dublin Delay selected as the speaker works through technical difficulties. Guitar strumming gives way to music.

[4:40](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=280s) [Logic Pro displays guitar tracks and effects.](https://assets.ai.engineer/sites/aiecode2025/sha256/b617c1d46f907efec62a05facb33d0ea6c62bbb9d70771468ceee1dc2b04fb25.webp)

Tracks:
Mic
Guitar Speak
Todd Clean Echoes
Dublin Delay
American Metal
Wah I Am

Dublin Delay effects:
Noise Gate
Pedalboard
Amp
Channel EQ
Compressor

Interface text:
Guitar Speak.logicx - Tracks
Region: [illegible] #02
Track: Dublin Delay
Edit
Functions
View
Timeline: 1 5 9 13 17 21 25 29 33 37 41
120 KEEP TEMPO
4/4
Cmaj
Setting
EQ
Input 1
Audio FX
Mastering
Sends
Stereo Out
Group
Read

### Halloween guitar and projected lights

A guitarist plays beside the Stranger Things alphabet projected onto a garage door, an effect the speaker built with a small app.

[5:03](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=303s) [A guitarist plays outside beside a projected alphabet display.](https://assets.ai.engineer/sites/aiecode2025/sha256/e38cec08f0c6a988f3aca6ab05d0cf2f4b6d45d9e1c6c8210472b20dffe3fc33.webp)

A B C D E F G H
I J K L M N O P Q
R S T U V W X Y Z

activeviz.com/stranger-things-lights ↗

Interface text:
While My Guitar Gently Spea…
app
localhost:3030/7
Relaunch to update
3:30

### Alphabet lights demo

Guitar notes advance the alphabet lights through a custom message. The demo shows “Happy Halloween” with H lit on the wall.

[5:12](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=312s) [An alphabet wall with a lit letter and custom-message controls.](https://assets.ai.engineer/sites/aiecode2025/sha256/324a2fa6a7d146f318b43730b8a46bad56ec9eca8dbdc29ff37d67d4c204c202.webp)

A B C D E F G H
I J K L M N O P Q
R S T U V W X Y Z

Mode:
Custom Message

All Letters (A-Z)
Notes Only (A-G)
Custom Message

Custom Message:
Happy Halloween
15 characters

Display your own message, advancing on audio detection

H
Stop Listening

Interface text:
While My Guitar Gently Spea[illegible]
app
localhost:5173/stranger-things-lights
Relaunch to update

### What if my guitar could speak?

The alphabet-lights project prompts a new question: how hard would it be to make the guitar speak?

[5:42](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=342s) [A slide asks what if the guitar could speak.](https://assets.ai.engineer/sites/aiecode2025/sha256/b3d3f76ac5f6e8c531ab60e9c4c23b55bb02676d61e3d50ce6a6c642e9793ffe.webp)

What if my guitar could
speak?

Interface text:
While My Guitar Gently Spea…
app
localhost:3030/8
Relaunch to update

### Tools for the job

The toolkit pairs JUCE with text-to-speech and DSP; AUv2, VST3, and AAX plugin formats can share the same source.

[5:57](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=357s) [A slide lists JUCE, plugin formats, text-to-speech tools, and digital signal processing techniques.](https://assets.ai.engineer/sites/aiecode2025/sha256/8a3fe91d4a4e608f048da4a51c0993bfac273d8b54f89a1a3eb9f08ea537418f.webp)

Tools for the job

• JUCE has the most production miles in live audio. "No allocations on the audio thread" is built into its idioms. C++ FTW!
• Plugin formats are free. AUv2, VST3, AAX – same source.
• TTS – Piper (local neural), Apple AVSpeechSynthesizer, prebaked WAV.
• DSP – 24-band channel vocoder, YIN pitch detection, PolyBLEP saw.

Interface text:
localhost:3030/9
app
Relaunch to update

### Guitar triggers a word

The proposed flow converts typed text to an audio clip once, then plays that clip whenever the guitar is plucked.

[6:31](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=391s) [Diagram showing text converted to an audio clip, then played when a guitar is plucked.](https://assets.ai.engineer/sites/aiecode2025/sha256/451888cf9af14f8df8c83079a56a09f8a3982521b970506a12b3883a0e412197.webp)

Goal: Guitar triggers saying a word

Offline / bake this once
Raw text → TTS → Audio clip

Live / repeat per pluck
Guitar pluck → Play audio clip → 🔊

Scene 1

Interface text:
localhost:3030/10
Relaunch to update
Google Chrome

### Single-word plugin demo

In Logic Pro, the presenter demonstrates Guitar Speak playing the audio file “Developers” repeatedly as he plays the guitar. The plugin displays the word above its waveform.

[7:01](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=421s) [Guitar Speak plugin in Logic Pro displays the word Developers above an audio waveform.](https://assets.ai.engineer/sites/aiecode2025/sha256/87f0d1c5db1280e24d47d9847c4c2f9f70a0047b24dffc2d9ae0e96893fa4760.webp)

a word

Guitar Speak
Untitled
Side Chain: Input 2
Compare  Copy  Paste  Undo  Redo
View: Editor
44100 Hz | 512 samples | 11.6 ms | [illegible] | Gate (0.00) | Limit off | Log | MIDI
In  -inf dB  Out  -inf dB
Scene
1 multi syllable words
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Developers
Rewind
1 / 1
V Bypass voc...  N Noise carrier  S Sibilance off  P Pitch sing  M Sing  Ph Phoneme  K Knobs  O Scope
VOCODER
Makeup 4.00
Pitched floor 0.30
Sibilance 0.30
Clarity 0.90
Sensitivity -50.0 dB
Mic gain 12.0 dB
(no pitch)
Latch  Advance  Syllable
MIC SC  -54 dB
Waveform + slice boundaries
Import  Auto-slice  Snap  Load  Save
Settings
Q&A
Developers  Say
Guitar Speak

Interface text:
Guitar Speak.logicx - Tracks
Region: Guitar Speak
Track: Guitar Speak
Mic
Guitar Speak
Todd Clean Echoes
Dublin Delay
American Metal
Wah I Am
Noise Gate
Amp
Channel EQ
Compressor
Gtr Spe...
Audio FX
Mastering
Stereo Out
Scene 1

### Say multiple words

The audio clip is sliced into words ahead of time, so each guitar pluck plays the next word.

[7:34](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=454s) [Diagram adds per-word slicing so each guitar pluck plays the next word.](https://assets.ai.engineer/sites/aiecode2025/sha256/f99e86c038687dae59b1d7054f49174c31fda11ea1c4c7f1edeaf9eb30d7bd09.webp)

Goal: Say multiple words

Offline / bake this once
Raw text → TTS → Audio clip
↓
Slice per word
↓
Live / repeat per pluck
Guitar pluck → Play next word

Scene 2

Interface text:
While My Guitar Gently Spea…
app
localhost:3030/11
Relaunch to update
Logic Pro

### Multiple-word plugin demo

Guitar Speak plays a phrase sliced into words. The plugin shows the waveform’s slice boundaries and highlights “speak” as the demo reaches that word.

[7:39](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=459s) [Guitar Speak plugin displays a sliced waveform and highlights the word speak.](https://assets.ai.engineer/sites/aiecode2025/sha256/3da72b23dd63ce540c163a05ade6b8ce61f73557f75a019a3824f505b0410f7e.webp)

Guitar Speak
Untitled
Side Chain: Input 2
Compare  Copy  Paste  Undo  Redo
View: Editor
44100 Hz | 512 samples | 11.6 ms | [illegible] | Gate (0.00) | Limit off | Log | MIDI
In  -114.7 dB    Out  -118.1 dB
Scene 2  Text to Speech
0  1  2  3  4  5  6  7  8  9  10  11  12  13  14
can  speak
Rewind
6 / 6
V Bypass voc...  N Noise carrier  S Sibilance off  P Pitch sing  M Sing  Ph Phoneme  K Knobs  O Scope
VOCODER
Makeup  4.00
Pitched floor  0.30
Sibilance  0.30
Clarity  0.95
Sensitivity  -50.0 dB
Mic gain  12.0 dB
Latch  Advance  Syllable
MIC SC  -40 dB
Waveform + slice boundaries
Import  Auto-slice  Snap  Load  Save
Settings
Q&A
Look at me, I can speak
Say
Guitar Speak

Interface text:
Guitar Speak.logicx - Tracks
Region: Guitar Speak
Track: Guitar Speak
Edit  Functions  View
Mic
Guitar Speak
Todd Clean Echoes
Dublin Delay
American Metal
Wah I Am
Noise Gate
Amp
Channel EQ
Compressor
Gtr Spe...
Stereo Out
Scene 2
Relaunch to update

### Auto slice the audio clip

WordAligner finds silences where the peak envelope falls below 15% of peak, then uses the N-1 longest gaps to cut N word segments. Some words have no silence between them.

[7:56](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=476s) [A flowchart shows how silence in a TTS audio clip is used to find word segment boundaries.](https://assets.ai.engineer/sites/aiecode2025/sha256/ef48b635d042127c41b2bb84ee83874c7b30fcde47ddea2175a497fe66f2c5fa.webp)

Auto slice the audio clip

We get one blob of PCM back from TTS – no timestamps. To play one word per pluck, we have to find the cuts ourselves. ( Enter: WordAligner )

TTS PCM
(one blob)
→ Peak envelope
30 ms release
→ Find silences
env < 15% peak
→ Keep N-1 longest
→ cut points
→ N word segments

💡 Energy-gap segmentation

Interface text:
While My Guitar Gently Spea…
app
localhost:3030/12
Relaunch to update
Scene 0

### Audio slicing demonstration

Guitar Speak shows a waveform with slice boundaries. The speaker explains that silence can mark word cuts, but some words have no silence between them, so the method is not foolproof.

[8:09](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=489s) [The Guitar Speak plugin displays an audio waveform with slice boundaries over a music production workspace.](https://assets.ai.engineer/sites/aiecode2025/sha256/536f16898a021813160ce4af96a334290839858562c05eea7c9022c6d3e77f14.webp)

Guitar Speak

Untitled
Side Chain: Input 2
Compare  Copy  Paste  Undo  Redo
View: Editor

44100 Hz | 512 samples | 11.6 ms | [illegible] | Gate (0.00) | Limit off | Log | MIDI
In -66.0 dB  Out -inf dB

Scene
0 Intro
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
(pluck a note to speak)
Rewind
29 words

V Bypass voc...  N Noise carrier  S Sibilance off  P Pitch sing  M Sing  Ph Phoneme  K Knobs  O Scope
VOCODER
Makeup 4.00
Pitched floor 0.30
Sibilance 0.30
Clarity 1.00
Sensitivity -50.0 dB
Mic gain 12.0 dB
(no pitch)
Latch  Advance  Syllable

MIC SC
Waveform + slice boundaries
Import  Auto-slice  Snap  Load  Save
Settings
Q&A
thank you for let-ting me be here. It feels so good to get out of my gui-tar case once in a while. No thanks
[illegible]
Say
Guitar Speak

Interface text:
Guitar Speak.logicx - Tracks
Region: Guitar Speak
Track: Guitar Speak
Mic
Guitar Speak
Todd Clean Echoes
Dublin Delay
American Metal
Wah I Am
Noise Gate
Amp
Channel EQ
Compressor
Gtr Spe...
Stereo Out
Scene 0
[illegible] per pluck, we have to find the
N word segments
Relaunch to update

### Syllables are tricky!

A sonority-peak syllabifier uses peaks to identify syllables, but espeak-ng’s phoneme labels have uniform fake durations, so their positions still need to be estimated from audio.

[8:35](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=515s) [A slide describes phoneme alignment and shows a sonority-peak diagram for syllable segmentation.](https://assets.ai.engineer/sites/aiecode2025/sha256/07312a5b09263c8bdd0036ee4258cdc5342b73c7eb2e5ffd359b242892a8ff0d.webp)

Syllables are tricky!

• Phoneme alignment – espeak-ng labels + sonority-peak syllabifier
• espeak gives the sounds, but not the when – uniform fake durations, so we still have to guess each syllable’s position from the audio

SONORITY VALUE (ACOUSTIC LOUDNESS)
Y-axis ticks: 1, 2, 3, 4
Highest
(Lowest

SYLLABLE NUCLEUS (PEAK)
A=Vowels (5)
A=Vowels (5)
R=Liquids
R=Liquids (4)
R=Liquids
S=Fricatives
S=Fricatives (2)
C=Stops (1)
C=Stops (1)
C=Stoptes (1)

THE ALGORITHM DETECTS A PEAK-TO-TROUGH PATTERN. A DROP TO 0 (SPACE) INITIATES A NEW SEGMENTATION.

C A T S R U N F A S T
ALGORITHMIC BOUNDARY MARKER (VALUE = 0)
OUTPUT: [CATS] [RUN] [FAST]

Interface text:
While My Guitar Gently Spea…
app
localhost:3030/13
Relaunch to update

### Syllable slicing demonstration

Guitar Speak displays syllable slices on a waveform. After playing a phrase, the presenter explains that he can drag the segments to correct slicing that still falls short of his goal.

[8:58](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=538s) [The Guitar Speak plugin shows a waveform with slice boundaries over the syllables slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/432252a96473134bec8034508aff1809b2f7df449b2f1372a46e22dab44cd7f6.webp)

Syllables are tricky!
• Phoneme align[illegible]
• espeak gives th[illegible] we still have to guess each syllable’s positi[illegible]

Guitar Speak
Untitled
Side Chain: Input 2
Compare  Copy  Paste  Undo  Redo
View: Editor
44100 Hz | 512 samples | 11.6 ms | I:4->mono -> O:2 | Gate 0.00
In  -inf dB  Out  -inf dB
Scene 0 Intro
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Rewind
29 words
(pluck a note to speak)
V Bypass voc...  N Noise carrier  S Sibilance off  P Pitch sing  M Sing  Ph Phoneme  K Knobs  O Scope
VOCODER
Makeup 4.00
Pitched floor 0.30
Sibilance 0.30
Clarity 1.00
Sensitivity -50.0 dB
Mic gain 12.0 dB
(no pitch)
Latch  Advance  Syllable
MIC SC
Waveform + slice boundaries
Import  Auto-slice  Snap  Load  Save
Settings
Q&A
thank you for let-ting me be here. It feels so good to get out of my gui-tar case once in a while. No thanks
Say
Guitar Speak

Partially obscured diagram:
[illegible]RITHM DETECTS
[illegible]ROUGH PATTERN.
[illegible] (SPACE) INITIATES
[illegible]GMENTATION.
[illegible]tives (2)
[illegible]=Stoptes (1)
S  T
[illegible] = 0)

Interface text:
While My Guitar Gently Spea[illegible]
app
localhost:3030/13
Relaunch to update

### What if my guitar could sing?

After making the guitar speak, the speaker turns to pitch detection as a step toward making it sing.

[9:37](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=577s) [A slide asks whether the guitar could sing and introduces pitch detection.](https://assets.ai.engineer/sites/aiecode2025/sha256/365316fe364a9e45cd17d655bb755f872c640d268218d1168bc2b6ced1b30f54.webp)

What if my guitar could
sing?

Next Stop: Pitch detection.

Interface text:
While My Guitar Gently Spea…
app
localhost:3030/14
Relaunch to update

### Fundamental frequency and harmonics

A complex tone combines a fundamental with higher harmonics. The speaker wants to detect the fundamental to identify a guitar note; the diagram associates the other harmonics with timbre.

[9:54](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=594s) [A diagram compares a fundamental frequency with higher harmonics using waveforms and frequency spectra.](https://assets.ai.engineer/sites/aiecode2025/sha256/2044a2cc99d2b5556a42833add3786b7305ab8413c5717d50284143c5fb0e5be.webp)

Combined Complex Tone

F0 (Fundamental Frequency)
e.g., at 200 Hz

2nd Harmonic (2x F₀)
400 Hz

3rd Harmonic (3x F₀)
600 Hz

**Higher Harmonics (4F₀, 5F₀, 6F₀...)

F0 (Fundamental)
Highest Amplitude (Dominant Pitch)
2F₀
3F₀
NxF₀
Reduced Amplitude

F0 (Fundamental - Pitch)
Harmonics (Timbre)
2F₀  3F₀  4F₀  ...
Full Frequency Spectrum of the Complex Tone

F₀
Harmonics

THE DOMINANT F₀ SETS THE PITCH, WHILE OTHER FREQUENCIES (HARMONICS) DETERMINE THE TIMBRE.

Interface text:
While My Guitar Gently Spea[illegible]
app
localhost:3030/15
Relaunch to update

### The vocoder marries pitch and voice

YIN detects the guitar’s pitch to make a sawtooth carrier. A 24-band vocoder shapes it with a spoken word’s voice envelopes, aiming to sing the word at the guitar’s pitch.

[10:31](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=631s) [Diagram showing guitar-derived pitch and a voice clip feeding a 24-band vocoder.](https://assets.ai.engineer/sites/aiecode2025/sha256/1d23b03c2dac5762a29dc71ccf4256b0ee1f90c5b07e718ee439779a46c6c8b5.webp)

The vocoder marries pitch + voice

Carrier = pitch
Guitar → YIN f₀ → Pitched saw

Modulator = words
Voice clip
one spoken word

24-band vocoder
carrier bands × voice envelopes

word, sung
at our pitch

It's not a blender – it's a stencil. The voice is the shape; the guitar is the paint.

...or: the voice works a 24-band EQ on my guitar, hundreds of times a second

ADSR – Attack · Decay · Sustain · Release

Interface text:
While My Guitar Gently Spea...
app
localhost:3030/17
Relaunch to update

### Guitar-Lead speech demo

The presenter attempts AI speech over a recorded guitar backing, but technical difficulties interrupt the demo. He describes Clarity as mixing the synthesized note with the AI voice.

[12:35](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=755s) [Logic displays a backing clip and the Guitar Speak plugin's Guitar-Lead scene with a sliced speech waveform.](https://assets.ai.engineer/sites/aiecode2025/sha256/4673296160ef097a0498b1f9b05e72a7926d82cb7ede22843023fdc45a5b3e08.webp)

Guitar Speak

Scene
10 Speak v2 — Guitar-Lead
(pluck to speak v2)
0 / 37

V Bypass voc...
N Noise carrier
S Sibilance off
P Pitch sing
M Sing
Ph Phoneme
K Knobs
O Scope

VOCODER
Makeup: 4.00
Pitched floor: 0.30
Sibilance: 0.30
Clarity: 0.60
Sensitivity: -50.0 dB
Mic gain: 12.0 dB

Latch
Advance
Syllable

MIC SC
Waveform + slice boundaries
Import  Auto-slice  Snap  Load  Save

Settings
Q&A
I'm here at this conference, a conference that's peak. I'm a guitar that can speak. So thank you for [illegible]
Say

Scene 10

Interface text:
Guitar Speak.logicx - Tracks
Region: Guitar Speak
Track: Guitar Speak
Mic
Guitar Speak
Todd Clean Echoes
Dublin Delay
American Metal
Wah I Am
Todd Clean Echoes #16

Untitled
Side Chain: Input 2
Compare  Copy  Paste  Undo  Redo
View: Editor
44100 Hz | 512 samples | 11.6 ms
Gate  Limit  Log  MIDI
In -59.9 dB  Out -83.6 dB
Rewind

Noise Gate
Amp
Channel EQ
Compressor
Gtr Spe...
Audio FX
Mastering
Sends
Stereo Out
Group
Read
Relaunch to update

## Full transcript

[0:00](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=0s) [upbeat music] So I am Todd Fisher. I love the guitar. That's one of my passions in life.

[0:16](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=16s) Uh, today I wanna talk about, uh, this project I've been working on, uh, for a while here, effectively making my guitar speak. Uh, but of course, I wanna start out with kinda framing it under this awesome premise.

[0:28](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=28s) You know, we've all been to live performances where, like, your mind would just be blown. It was awesome. Uh, you know, the, the first one I remember, way back, uh, I was in high school.

[0:36](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=36s) I went to a Slipknot concert, so a little bit heavier music, uh, here in the Bay Area. And I remember at some point, the drummer was drumming a cool drum solo, and his drum set started raising up.

[0:46](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=46s) And I was like, "Whoa, that's cool," and everyone k- got kind of excited, right? And then halfway through the solo, his drum set started to go like this, and tipping.

[0:54](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=54s) And his whole drum set was, like, on the wall while he was upside down playing his drum solo, and everyone was just cheering, like, "Whoa, this is so crazy.

[1:02](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=62s) Mind-blowing." Like, whoa. It was so, so impactful, right? I still rememb- remember it today. Um, fast-forward to probably about a year ago, uh, I went to New York to see the Stranger Things on Broadway, uh, the, The First Shadow, and it was really cool because the whole time I was there, I was watching it, and it felt

[1:19](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=79s) like it was actually like a, a, a Netflix episode but in real life. And it was just like, the effects that they did there was so, so amazing. I remember even one time at, near the end, there was a scene where somebody, somebody was falling backwards a- and it was, like, a slow motion scene, and it probably

[1:35](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=95s) about a minute long, the whole scene. But just, like, the fact that, like, they, they were able to produce that in real life, but it looked like it was a post-production slow motion scene, just, just mind-blowing.

[1:45](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=105s) So I was like, "Whoa, that's cool." So given you, everyone here has probably had some experiences, uh, with live performances in the past, and I think it's really awesome just to, like, see the creativity, uh, that people can do, uh, with leveraging technology in some creative way.

[2:00](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=120s) And so, you know, I'm thinking about all the projects that we have as, as engineers over the years where it's like, "Hey, that's a cool idea. I'm gonna write this down."

[2:06](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=126s) I have, like, a big, giant list. Anyone have a list of projects that maybe they'll get to? Yeah. So it all happens, right? Uh, and it's really awesome 'cause the last six months or so with AI, uh, being able to, you know, push us forward with some of these projects, it's just really cool to see where we

[2:21](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=141s) can take all these projects. So, so part of, uh, my goal today is to inspire you guys to find whatever project you're passionate about. Go start building it because it's super easy now with AI.

[2:31](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=151s) It's easier with AI. I know sometimes there's caveats. But, but in general, uh, I, I want to have you guys leave this session being inspired to go build whatever that cool thing is for you personally that's gonna, you know, help you to learn stuff or maybe make a difference in the things that you're, you're trying to figure

[2:47](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=167s) out. Uh, so I wanna start off with framing this kind of in the realm of the guitar. So the guitar, 'cause the guitar, guitar's been around for several years, several thousands year, or hundreds of years, sorry.

[2:58](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=178s) Um, and at some point, somebody said, "Hey, I'm gonna put a pickup on a guitar. I'm gonna plug it into a speaker," and now we have rock and roll.

[3:05](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=185s) We're gonna make it super loud. That was pretty awesome. Uh, and then of course, we have a lot of people getting these stomp boxes, these, these effects pedals. We put them all together.

[3:13](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=193s) We have really awesome sounds. You know, really famous people have a lot of these. Uh, and then at some point, Peter- Peter Frampton, uh, came along and said, "Hey, what if we get the sound of the guitar, put it through an actual hose, like a physical hose, put it in my mouth, and then basically play the guitar

[3:29](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=209s) sound in my mouth a- and, and form some sort of words?" So that's pretty awesome. That, that's known as the talk box. Uh, and then in the last couple decades, we have a lot of progress made with software emulation.

[3:40](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=220s) Uh, you think of, you know, Pro Tools. You think of Logic, of, uh, Fruity Loops. Uh, uh, there's a lot of software, even to the point where all the effects are in software now.

[3:48](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=228s) There is the argument made, hot take, uh, that maybe you don't need all those physical effects pedals anymore. So, so there's that question, right? And then of course, looking forward, it's what is our next evolution of the guitar, uh, with AI in the picture?

[4:03](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=243s) So with that said, I wanna take you back several years ago, at Halloween time. I was passing out candy, bunch of trick-or-treaters. You know, it got kind of boring, so I was like, "Hey, what if I go and actually bring my guitar out with my amp and just play?"

[4:15](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=255s) Uh, so for the past, what, 10 or plus years or so, I've actually been playing guitar on Halloween passing out candy, and it's kinda fun. Uh, and then fast-forward probably three years ago, I decided to dress up as Eddie Munson.

[4:27](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=267s) Anyone know Eddie Munson? Stranger Things guitar player guy. So pretty awesome guy, so I dressed up as him, and I was like, "Yeah, this is gonna be so fun.

[4:34](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=274s) I could play the music with him," whatever, and you kind of go the... Oh, hold on.

[4:42](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=282s) Hold on, wait a minute. [guitar strumming] Technical difficulties. [guitar strumming] All right, there it goes. [upbeat music]

[4:56](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=296s) So a bunch of little Stranger Things awesomeness, right? Uh, but figured out, "Hey, what if there was, like, some more stuff I could put i- in this whole experience?"

[5:03](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=303s) And so I decided to go and build a little app that would actually paint the Stranger Things alphabet on my garage door just because it's fun, right? Uh, and so let me just show you a quick example of how this worked.

[5:14](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=314s) So effectively, make sure I have the mic permission. Whenever I play a note, [guitar strumming]

[5:22](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=322s) it would go and communicate whatever lights, and I even had it to where I could actually, uh, set custom messages such as Happy Halloween, all that fun stuff. It was, it was overall pretty fun, uh, just to kinda mess with that, right?

[5:33](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=333s) So [guitar strumming] once again, you know, whatever I type in there, I could actually spell whatever a- and, and very much a, a, a cool nod to Stranger Things. But it kinda got me thinking, like, "What is the next evolution of this project?"

[5:46](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=346s) And I settled on this idea of, like, hey, how hard would it be to make my guitar speak?

[5:51](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=351s) It sounds easy. Maybe, maybe not. So, so today I want to share kind of my journey, uh, in this process, uh, and where kind of I'm at today. Uh, so looking around the different tools, there's a framework out there called JUCE, uh, really awesome for anyone building audio software, look into JUCE.

[6:06](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=366s) It's, it's pretty good. Um, and then, of course, there's a, a number of plug-ins, uh, uh, sorry, plug-in formats out there, uh, for your, your digital audio workstation. And for, for those that are not aware, uh, your DAW is effectively your IDE, but for musicians and, and music producers.

[6:22](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=382s) And then I, I started with some text-to-speech stuff with Piper and some built-in Apple stuff, and then some other really fun, uh, digital signal processing. Uh, so with that said, my first stab here was I want to get raw text, so I can just type in whatever text I want, uh, push it through the text-to-speech, uh, get

[6:39](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=399s) the audio clip, and then whenever I play a note on the guitar, I want to go and play that back. So let's go see how that works. So switching over to my Logic Pro here,

[6:50](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=410s) and this is the plug-in I made. So it's just like any other plug-in in Logic, where you just plop it in there. It's just, you know, chaining all the effects together.

[6:59](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=419s) And so this is what I came up with.

[7:02](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=422s) Developers.

[7:04](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=424s) So it's playing-

[7:04](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=424s) Developers.

[7:06](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=426s) Pretty awesome, right?

[7:07](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=427s) Developers. Developers.

[7:08](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=428s) Oh, it kinda-

[7:08](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=428s) Developers

[7:09](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=429s) ... kinda reminds me of something, right?

[7:10](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=430s) Developers.

[7:11](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=431s) Start clapping, everyone. Start clapping.

[7:13](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=433s) Developers.

[7:14](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=434s) Developers.

[7:15](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=435s) Developers.

[7:16](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=436s) Developers.

[7:16](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=436s) Developers.

[7:16](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=436s) Developers.

[7:17](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=437s) Developers.

[7:18](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=438s) Awesome. Thank you. That, that's pretty awesome. Uh, you guys are great. Um, so I got it to where it's playing an actual audio file. That's pretty awesome, but it turns out, you know, in English or in any languages for that matter, there's more than one word.

[7:31](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=451s) So the next kind of evolution-

[7:32](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=452s) Developers

[7:32](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=452s) Oh, it's a little bit chatty. Uh, the next evolution there is let's actually go and slice it per word now.

[7:39](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=459s) So I got to the point where it's playing. Now it's gonna slice per word.

[7:43](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=463s) Look at me. I can speak. Look-

[7:50](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=470s) Oh

[7:50](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=470s) ... at me. I can speak.

[7:53](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=473s) So now it's speaking words, and that's pretty awesome, right? Uh, but it turns out that, uh, as we speak, there's a number of challenges in how we automatically slice words.

[8:03](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=483s) And so I looked into this thing called energy gap segmentation. The general idea here is if you look in, uh, any waveform over here, we see that here's a bunch of, of words that we're speaking, right?

[8:14](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=494s) Uh, the idea there is there's typically silence in between words, so let's just cut it whenever the decibels are, are very much close to zero, right? Uh, but the issue with that is there's actually sometimes when, you know, as I'm speaking right now, for example, uh, there's actually no silence in between some of my words, so it

[8:30](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=510s) gets a little bit challenging to where it's not 100% foolproof, right? So beyond that, I looked into this thing called sonority peak syllabi- syllabifier. That's a hard word to say.

[8:40](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=520s) Uh, effectively identifying the syllables of the audio signal and, uh, identifying that there's vowels in here. Vowels typically lead to syllables. That's kind of the idea. So I said, "Okay, let's take the sonority peak, uh, add it to the, uh, the energy gap, and figure out if we could just make that work automatically."

[8:58](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=538s) So with that said, I made it kind of work, so let's just play this one.

[9:02](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=542s) Thank you for letting me be-

[9:05](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=545s) Oh. Oh

[9:06](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=546s) ... here. It feels so good to get out of one's guitar case

[9:19](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=559s) once in a while.

[9:23](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=563s) So there you go. It's working pretty well. Not quite a- as good as I want to, so long story short, uh, I settled on just the, uh, ability to go and actually I could drag this and manually edit some of these, uh, uh, segments in here.

[9:36](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=576s) So worked okay, right? Um, but moving on, you know, uh, with the spirit of evolving the thought, evolving the, the project, right? It's like, "Okay, we're, we're having the AI say stuff.

[9:46](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=586s) That's great, but what if we could actually make it sing?" Let's take it to the next step 'cause this is music. Why not, right? Uh, so looked into pitch detection, and for those that are not aware, uh, as you hear any, any noise out there, there's typically multiple frequencies going on at any given time.

[10:02](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=602s) You know, you think of when you play the, the C key on the piano, uh, there is definitely a fundamental frequency or the, the kind of the, the one that we identify as the note, but there's a bunch of other frequencies.

[10:12](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=612s) So we needed a way to go and figure out, like, how do we detect that fundamental frequency? So when I p- press something on the guitar, how do I translate that into an actual note?

[10:21](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=621s) Uh, and then I found this, uh, YIN pitch algorithm. Basically what it does is it detects the pitch. Uh, I won't get into all the details, but look it up.

[10:29](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=629s) It's kind of a, a fun, uh, really cool way of detecting the pitch. Uh, but effectively what I do is I play the guitar, uh, I detect the, the pitch.

[10:39](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=639s) I, I make what's effectively a pitched sawtooth or AKA a synthesized note. Uh, so for those that are not familiar with how audio works on the, the computer, you think of all electronic music, all that stuff, uh, that is basically, uh, a synthesized note.

[10:53](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=653s) We have ADSR, which effectively are the levers to figure out how to actually make the note sound in different ways. Uh, and then so basically we get the pitched notes or the sy- synthesized note.

[11:04](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=664s) Uh, we then push it through, uh, the, the voice clip effectively. So think of the talk box. Uh, we're kind of filling up the cavity of the talk, uh, of the voice that is, and we push it through a vocoder, and it should sing.

[11:16](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=676s) So that's kind of the idea, right? So with that said, let's go ahead and jam out a little bit because I have a guitar and it's fun. So

[11:24](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=684s) let me get back to Logic here. Uh, so I'm gonna play some chords. Uh, I originally was gonna, uh, play a song that is very much related to the talk, or the, the title of my talk, uh, While My Guitar Gently Speaks.

[11:41](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=701s) But because this is gonna be posted online, I don't wanna muddy up the waters with any copyright things. So I will play some chords that may or may not sound similar to a famous song.

[11:51](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=711s) So-[guitar playing]

[12:31](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=751s) So that's the backing track, so let's go ahead and have some AI speaking on top of it. [guitar playing]

[12:41](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=761s) I'm here at this-

[12:47](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=767s) Oh. Hold on, technical difficulties. Let's try that again.

[12:52](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=772s) Doo, doo-doo doo doo. Live demos, always the best. [guitar playing]

[12:54](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=774s) I'm here at this con- fence. [laughs] [guitar playing]

[13:09](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=789s) All right, technical difficulties again, so sorry for that. But let's just run through it and see what happens. [guitar playing]

[13:19](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=799s) I'm con- fence that speak. I'm a guitar that can speak,

[13:30](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=810s) so...

[13:32](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=812s) Awesome. So there you go. S- some bugs to work out, but overall, we are able to, to say words on top of this. And for what it's worth, I'm trying to mix the, uh, the t- the synthesized note, uh, with this clarity lever right here.

[13:44](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=824s) So bal-- Or mixing the, the synthesized note with the actual voice that the AI is giving. Uh-

[13:50](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=830s) Can-

[13:50](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=830s) So, so that worked pretty good. Um, but there's also this other question, you know. Given that, uh, when you speak, it's typically conversational, what if I could take this to the next level, and what if we actually had this microphone right here where I could speak into the mic, it would then respond on the guitar?

[14:08](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=848s) Uh, so the way that I c- accomplish this is speak into the mic, use, uh, speech-to-text, so Whisper. Uh, put that into raw text, run a local model on my computer.

[14:18](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=858s) Uh, it doesn't really matter what LLM, just any local model, have a conversation coming, and then from that output, go and plop it on the guitar. So let's see how this works.

[14:31](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=871s) Hmm. Anyone have a question you wanna ask my guitar?

[14:39](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=879s) Anybody? What is reality? Let's do it. What is reality?

[14:50](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=890s) So it's thinking, going through the LLM things,

[14:54](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=894s) and let's see what it says. [guitar playing]

[14:56](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=896s) That's quite a question to start with. [laughs] How does your music help you understand that elusive concept of reality right now?

[15:18](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=918s) So yeah, very existential. I like that, so thank you for that suggestion there. S- so there you go. We have a choppy version, but, but it is working, so, so that's a win, right?

[15:26](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=926s) Uh, awesome, and thanks for the claps there. Um,

[15:30](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=930s) but really like, it's not quite singing yet, right? So moving on in this project, it's like, how do I actually make it sing, right? Uh, I found a lot of, uh, open source options out there as far as, uh, uh, recordings or, or samples, if you will.

[15:43](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=943s) Uh, there's a thing called VocalSet out there. They basically recorded a bunch of singers, and I could use those audio files to go and do some fun stuff with them.

[15:50](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=950s) And effectively, I took those samples. I used, uh, a project called WORLD, which helps with things like pitch shifting and some other things, and I'm able to then shift the, the pitches of everyone singing on those audio clips and map it to the guitar.

[16:05](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=965s) Uh, this is a very heavy process, and so I can't really do that live. I had to effectively pre-bake that, and then once it's already pre-baked, I could then go and jam out with it.

[16:14](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=974s) So with that said, uh, let me go ahead and show that example here.

[16:19](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=979s) Uh... So, uh, because it takes so long, I actually just started with the vowel sounds. So there's somebody singing all... Each of the five vowels, and so we'll see how that sounds, right? [singing]

[16:41](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=1001s) Kinda fun, kinda weird, but overall, it's working. It's, it's closer to singing, right? Uh, so let's go ahead and throw that on top of all the chords over here, and let's see if we can make it sound decently well.

[16:53](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=1013s) Oh. So we'll just play these chords again. [singing] [guitar playing] [laughs]

[17:15](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=1035s) So not quite your opera singer, but getting closer, and, and so I, I think there's something awesome there. Uh, so effectively, once again, that is, uh, going through, uh, the whole synthesize process, uh, putting a sample, shifting the, the pitch shift of the sample, and then effectively mapping each fret or each, each note on the guitar to

[17:32](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=1052s) one of those samples that are pre-baked effectively, right? Um, so with that said, you know, where do I wanna take this next? Uh, very much wanna get into more of, of the, you know, AI heavier options.

[17:44](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=1064s) Uh, if anyone ha- has any other ideas of how to make the guitar sing, feel free to come up to me after. I think it's pretty awesome. Uh, but more importantly, you know, everyone here, uh, going back to the, the charge to go and, and build some awesome, you know, side project, some passion project of yours.

[17:59](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=1079s) Like, what is that for you? Go and build awesome things because nowadays with AI, we could build so many really cool things, and time is typically not the, the big time suck that, that it once was, right?

[18:09](https://www.youtube.com/watch?v=E_Txocq-Lrw&t=1089s) So with that said, uh, be awesome. Uh, or, or be good to each other, and thank you very much. [applause] [upbeat music]
