# The End of Awkward AI Transcriptions

Travis Bartley · Myungjong Kim · Byungjoong · Jaehan

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/aDj9sY2RoG8-end-awkward-ai-transcriptions) · [Video](https://www.youtube.com/watch?v=aDj9sY2RoG8)

## The End of Awkward AI Transcriptions

Reliable speech recognition depends on matching encoders, decoders, speaker models, and customization tools to the audio and deployment constraints of each application.

Basic familiarity with neural networks and inference is useful; ASR means automatic speech recognition.

### What must a speech system handle?

[0:32](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=32s)

Will a recognition system work as well on a noisy telephone call as it does on clean speech? Will it recognize medical terminology, handle a speaker switching languages, and still respond quickly enough for a conversation? These are the opening engineering questions behind NVIDIA Riva, an enterprise platform covering speech recognition, synthesis, and translation. Low latency and efficient inference matter throughout, including on embedded devices.

The requirements fall into four connected categories:

- **Robustness:** Noise, telephony audio quality, and environmental interference change what reaches the recognizer.
- **Coverage:** Medical, entertainment, and call-center applications need different vocabulary and data. Language coverage also includes dialects and code switching, not just a list of supported languages.
- **Personalization:** Target-speaker recognition, uncommon-word boosting, and finite-state text normalization let customers influence whose speech is recognized and how the result appears.
- **Deployment:** Speed competes with accuracy, while broad task coverage can compete with efficiency.

**The model choice follows the application’s constraints.** A versatile model is useful only if its resource requirements and response time fit the product.

![The Developer’s Challenge slide shows a photo of two people beside four categories of speech application requirements and trade-offs.](https://ai.engineer/api/talk-essay-media/7650a1a9fcc35aeeed5bc70c974dc7b23db610b79a7ac778547e5c3dd7190f51)

Speech application trade-offs: robustness, coverage, personalization, and deployment.

### Choose the decoder for the workload

[2:42](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=162s)

**CTC** remains useful because its non-autoregressive decoding supports fast inference. The streaming approach described here runs recognition over chunks of audio. When that approach does not deliver the desired accuracy, **RNN-T** and NVIDIA’s **TDT** variant provide autoregressive streaming: they combine an audio encoder’s representations with an internal language-modeling component.

When streaming is less important, an **attention encoder-decoder** can prioritize accuracy and accommodate several tasks. **Whisper** is a familiar reference point for this architecture. A Transformer decoder can use prompt changes to select speech recognition, translation, timestamp prediction, or language identification.

| Decoder approach | Decoding behavior | Emphasis in the talk |
| --- | --- | --- |
| CTC | Non-autoregressive | Fast recognition over audio chunks |
| RNN-T / TDT | Autoregressive transducer | Streaming with language context |
| Attention encoder-decoder | Attention-based sequence generation | Accuracy and multiple tasks |

These are workload choices, not a universal accuracy ranking. The relevant question is whether the application needs immediate partial results, richer task control, or a particular balance of both.

### Shorten the sequence before decoding

[4:26](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=266s)

Across these decoder choices, the common encoder is [**Fast Conformer**](https://research.nvidia.com/labs/conv-ai/blogs/2023/2023-06-07-fast-conformer/). Its central change in this walkthrough is additional temporal subsampling: conventional 40 ms encoder steps become 80 ms steps. That gives downstream computation a shorter sequence to process. The 80 ms figure describes encoder-frame spacing, not end-to-end recognition latency.

Shorter encoded sequences reduce training memory pressure and the number of temporal positions processed during inference. The presenters report quicker convergence with less training data, but give no quantities or experimental conditions. Their practical reason for sharing this encoder across decoder families is that the computational savings remain useful whether the output is produced by CTC, a transducer, or an attention decoder.

### Parakeet and Canary divide the work

[5:27](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=327s)

The portfolio puts these architecture choices into two families. [**Riva Parakeet**](https://docs.nvidia.com/nemo-framework/user-guide/25.04/nemotoolkit/asr/models.html) groups CTC and TDT offerings around fast, efficient streaming recognition. The presentation associates Parakeet with recognition, translation, and target-speaker applications. That broad application framing should not be read as translation support in every Parakeet checkpoint: the architecture documentation describes Parakeet as an ASR family and **Riva Canary** as supporting recognition and translation.

Canary emphasizes accuracy and multitask modeling, while still pursuing speed. Fast Conformer is shared by both families; it is not exclusive to Canary. Together, the families let a deployment choose among speed, accuracy, and task coverage instead of forcing every workload through one model. **Variety is part of the system design:** the goal is a model that fits the need, rather than one model expected to satisfy every need.

### Connect speaker timestamps to transcript tokens

[6:57](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=417s)

Recognizing words does not establish who said them. [**Sortformer**](https://arxiv.org/abs/2409.06656) extends the system with an end-to-end neural diarizer that assigns speaker order by arrival time: the speaker who appears first comes first. This ordering bridges diarization, which identifies speaker activity over time, and ASR, which must associate speaker tokens with recognized words.

A small Python example makes the ordering rule concrete. Given illustrative first-speech times, the output assigns speaker tokens in arrival order rather than in the arbitrary order of the input labels:

```python
first_speech_seconds = {
    "voice_b": 2.4,
    "voice_a": 0.6,
}

arrival_order = sorted(
    first_speech_seconds,
    key=first_speech_seconds.get,
)
speaker_tokens = {
    voice: f"speaker_{index}"
    for index, voice in enumerate(arrival_order)
}

assert speaker_tokens == {
    "voice_a": "speaker_0",
    "voice_b": "speaker_1",
}
```

The neural diarizer supplies speaker activity; sorting alone does not detect voices or recognize their words. Its role here is to illustrate a consistent convention for connecting speaker identities to tokens.

The joint architecture fuses ASR encoder embeddings and Sortformer embeddings through a **speaker kernel**. This gives the recognizer speaker information alongside acoustic information, addressing who spoke what and when. Fine-tuning uses an objective described as similar to ordinary ASR training.

Optional query audio changes the task. With a query, the model can perform target-speaker ASR; without it, the model can perform multi-speaker ASR. The diagram shows the ASR encoder and Sortformer feeding the speaker kernel, a linear projection, and an RNNT decoder, with separate target-speaker and multi-speaker output examples.

![Diagram connects a Fast Conformer ASR encoder and Sortformer through a speaker kernel and linear projection to an RNNT decoder, with TS-ASR and MS-ASR outputs and optional query audio.](https://ai.engineer/api/talk-essay-media/641c600fdaf5f70b9634fa8ad3cea1b55d9b93e04fc18c028095dc5c086445ac)

Multi/target-speaker ASR architecture with Sortformer and optional query audio.

The same component pairing can also operate as a conventional cascade rather than a parallel joint model. That distinction separates the architecture’s integration possibilities from a requirement to deploy every component as one end-to-end system.

### Make recognized speech usable

[8:26](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=506s)

The recognizer sits inside a larger pipeline. Supporting models address distinct problems, from selecting speech regions to producing readable text:

| Component | Function |
| --- | --- |
| Voice activity detection | Detect speech segments for noise robustness |
| External n-gram language model | Rescore recognition hypotheses |
| Inverse text normalization | Convert spoken forms to written forms |
| Punctuation and capitalization | Restore readable sentence formatting |
| Speaker diarization | Identify speakers across a conversation |

For VAD, the presentation names **DistilBERTaVAD** and **MarbleNet**-based models. External n-gram language models add another place to improve accuracy and adapt recognition to a customer’s vocabulary.

Normalization has two directions: text normalization converts written text into spoken form, while **inverse text normalization**, or ITN, converts spoken-form transcripts into readable written forms. Riva uses weighted finite-state transducers for ITN. **BERT**-based punctuation and capitalization models then provide sentence formatting. Sortformer supplies speaker diarization; at the time of the presentation, cascade support is described alongside upcoming joint end-to-end models.

### Recognize lyrics over music

[9:50](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=590s)

At presentation time, Travis reports that NVIDIA supplies a majority of the Hugging Face Open ASR leaderboard’s top five models. He attributes the result to customization and model variety. It is a dated leaderboard claim, not evidence that customization alone caused the ranking, and it leads into a more concrete test: recognizing a song.

The demo plays rapid rap vocals over musical accompaniment. The browser interface shows an audio waveform, playback controls, and timestamped transcript rows under a Riva Parakeet V2 demo. This is a useful qualitative example because lyrical delivery and background music move the input away from clean conversational speech.

![Slide titled The End of Awkward Transcriptions contains a browser demo with a partially green audio waveform, playback controls, and a timestamped transcription table.](https://ai.engineer/api/talk-essay-media/35ebbf6097fb0e77b09d1bc10378a63cac03e2c9bb38732adca72f74af40b087)

Riva Parakeet V2 lyric transcription demo showing an audio waveform and transcript rows.

Travis judges the transcription accurate in the noisy setting. No word error rate, comparison baseline, or detailed test configuration is supplied for the song, so the result demonstrates a particular successful example rather than a measured robustness guarantee.

### Build coverage into the training data

[10:53](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=653s)

The training discussion returns to fundamentals rather than introducing a special recipe. Robust sourcing, multilingual coverage, and dialect sensitivity begin with language documentation: understand the language well enough to decide which kinds of data the training set must contain. Open-source data contributes variety and domain shifts; proprietary data supplies a complementary emphasis on quality.

**Pseudo-labeling** expands the available transcripts. The team uses leading models available for commercial use to generate text from audio, allowing new training data to benefit from both community advances and NVIDIA’s own releases. The generated transcript becomes a training label, so improvements in the models producing those labels can feed into subsequent model development.

### Keep training fed and validation broad

[12:01](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=721s)

Training uses the open-source NeMo toolkit. The operational focus is GPU utilization, data bucketing, and high-speed loading through [Lhotse](https://github.com/lhotse-speech/lhotse). These tools address the practical problem of supplying enough audio efficiently across different training settings. Most of the team’s data resides in object storage, which helps it move between cluster environments.

Validation mirrors the breadth of the training sources: it combines open-source and proprietary datasets and checks bias, domain behavior, and language coverage before release. A recognizer intended for varied customer environments needs evaluation beyond a single clean benchmark; otherwise, broad training coverage can remain an assumption rather than something tested.

### Serve streaming and offline workloads

[13:22](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=802s)

After training, models move into Riva through **NVIDIA NIM**. TensorRT optimizations and Triton Inference Server provide the inference foundation. A gRPC microservice interface supports both low-latency streaming and high-throughput offline processing, while containers package the serving system for deployment.

The presenters describe scaling Riva to hundreds of parallel streams, without specifying the hardware, model, or latency target. Deployment targets include on-premises infrastructure, cloud environments, edge systems, and embedded platforms, serving applications such as contact centers, consumer products, and video conferencing. NIM supplies prebuilt containers, industry-standard APIs, custom-model support, and optimized inference engines.

### Adapt the layer responsible for the error

[14:27](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=867s)

Customer requirements return at deployment time. Medical terminology and restaurant menu names require domain knowledge; telephone audio and noisy contact centers impose acoustic constraints. Those problems do not all belong to the same layer, so Riva exposes several adaptation points:

- **Acoustic models:** Fine-tune Parakeet- or Canary-based models for the relevant audio and recognition task.
- **External language models:** Adapt the n-gram model to domain language.
- **Output formatting:** Customize punctuation and inverse text normalization to produce the written forms the application needs.

The point of this separation is to give vocabulary, acoustics, and presentation their own places to improve.

**Word boosting** adds a more focused option for product names, jargon, and vocabulary that matters in the current context. It biases decoding toward selected terms rather than guaranteeing that they will be recognized correctly. Stronger boosts can also introduce false positives, so the [customization guidance](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-basics-customization-riva.html) treats boosting as a tradeoff rather than a universal correction.

### Find a model, then deploy or fine-tune it

[15:32](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=932s)

The starting point is the [NVIDIA speech model catalog](https://build.nvidia.com/explore/speech), where Riva models are offered through NIM. The catalog now includes later offerings as well, so model capabilities and API options should be read for the selected release rather than assumed from this presentation.

The closing resources point to a Riva quickstart for deployment, the developer forum for implementation questions, and a NeMo fine-tuning guide for adaptation. That leaves two practical paths from the architecture walkthrough: deploy an available model when it meets the application’s requirements, or customize the relevant parts of the speech pipeline when vocabulary, audio conditions, or output formatting demand more.

## Resources

- [Explore NVIDIA Speech Models](https://build.nvidia.com/explore/speech) — Browse NVIDIA speech models and their hosted API or self-hosted deployment options.
- [How Fast Conformer Reduces Speech Encoder Computation](https://research.nvidia.com/labs/conv-ai/blogs/2023/2023-06-07-fast-conformer/) — NVIDIA's explanation of temporal downsampling and other architectural changes in Fast Conformer.
- [Sortformer Research Paper](https://arxiv.org/abs/2409.06656) — Explains arrival-order speaker supervision and connecting diarization timestamps to speaker tokens in ASR.
- [META-CAT: Multi-speaker and Target-speaker ASR](https://arxiv.org/html/2409.12352v1) — Research coauthored by Myungjong Kim on injecting speaker supervision into ASR and conditioning transcription on query audio.
- [NVIDIA Speech Models in June 2025](https://developer.nvidia.com/blog/nvidia-speech-ai-models-deliver-industry-leading-accuracy-and-performance/) — A contemporary overview of Parakeet and Canary, their reported leaderboard positions, and a song-transcription example.
- [NeMo ASR Model Architectures](https://docs.nvidia.com/nemo-framework/user-guide/25.04/nemotoolkit/asr/models.html) — Architecture documentation for Parakeet, Canary, FastConformer, and their decoding approaches.
- [Customize Riva Speech Recognition](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-basics-customization-riva.html) — Practical guidance for word boosting, language-model adaptation, acoustic fine-tuning, and inverse text normalization.
- [Lhotse](https://github.com/lhotse-speech/lhotse) — Tools for preparing and loading speech data for machine-learning training.

## Slides

### The End of Awkward AI Transcriptions

Travis M. Bartley, Jinhan Wang and Myungjong Kim introduce NVIDIA’s talk on AI transcription at AI Engineer World’s Fair 2025.

[0:00](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=0s) [Title slide for an AI Engineer World’s Fair 2025 presentation.](https://assets.ai.engineer/sites/aiecode2025/sha256/99e406ec237c69b443ba651237d2ec8e4a5c69dadae34453b019610ab56ed0e3.webp)

The End of Awkward AI Transcriptions

AI Engineer World’s Fair 2025

Travis M. Bartley
Jinhan Wang
Myungjong Kim

### What We Will Cover Today

The talk follows NVIDIA Speech AI models from architectures and development through deployment and customization.

[0:10](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=10s) [Agenda covering NVIDIA Speech AI, architectures, development, deployment, customization, and closing thoughts.](https://assets.ai.engineer/sites/aiecode2025/sha256/97632c5b4b52163e52eff0278cb1115937eb963472b4d8382d0790383718dfa2.webp)

What We Will Cover Today

• NVIDIA Speech AI Overview
• Model Architectures & Innovations
• Model Development Process
• Deployment & Customization
• Closing Thoughts

Illustration text:
AI VOICE
AI MODEL
[illegible]

NVIDIA

### Why NVIDIA Riva

Riva brings speech recognition, text-to-speech and translation services together with NVIDIA, custom and community models, with a focus on deployment, inference and customization.

[0:31](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=31s) [Diagram of Riva deployment, inference, customization, and speech AI services.](https://assets.ai.engineer/sites/aiecode2025/sha256/719b08ee4210c7345f25592cd88df453d072f0e68e13febbc8d2b8bc35c8b6d3.webp)

Why NVIDIA Riva
Foundational Platform for Enterprise-scale Speech-Driven Multilingual AI Agents

Deploy Anywhere
State-of-the-Art
Support for NVIDIA Models & Community
Optimized Inference
Model Customization

ASR
Riva Parakeet, Canary

TTS
Riva Magpie & Zero-Shot

NMT
Riva Translate

NVIDIA Models
Custom Models
Community Models

Riva Collection of Speech AI Microservices

### The Developer’s Challenge

Speech applications must handle noisy audio and varied domains and languages, while adapting to user needs. Deployment adds choices between speed and accuracy, and between variety and efficiency.

[1:09](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=69s) [Speech application trade-offs in robustness, coverage, personalization, and deployment.](https://assets.ai.engineer/sites/aiecode2025/sha256/aae5f89ca83924ab9f921c853e12f1cb5a96c0f663f4f4793b512fbcf62e8a0b.webp)

The Developer’s Challenge
Trade-offs in Real-World Speech Applications

• Robustness
  • Noisy environments
  • Sound quality
• Coverage
  • Domain drift
  • Languages
    • Multilingual
    • Monolingual
    • Dialect
    • Code-switching
• Personalization
  • Target speaker or general use
  • Word boosting
  • Normalization
• Deployment
  • Speed vs. Accuracy
  • Variety vs. Efficiency

NVIDIA

### Connectionist Temporal Classification (CTC)

CTC merges repeated tokens and removes blanks to turn the illustrated sequence into “hello.” The speaker highlights its speed for streaming inference on chunks.

[2:40](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=160s) [Diagram showing repeated tokens and blanks collapsed into the word hello.](https://assets.ai.engineer/sites/aiecode2025/sha256/3609e2748f75be8d0a8f94d13f67edb201e3df1faf0f6074f0ec643db73f3500.webp)

Connectionist Temporal Classification (CTC)
Efficient alignment of input and output sequences for real-time applications

For an input, like speech

Predict a sequence of tokens
h e e ε l ε l l o

Merge repeats, drop ε
h e l l o

Final output
h e l l o

How CTC Collapsing Works
Sequence Modeling with CTC

Figure from: Hannun, Awni. "Sequence Modeling with CTC." Distill, 2017. https://distill.pub/2017/ctc/

NVIDIA

### Transducer (RNN-T/TDT)

The transducer combines audio encoded from xₜ with a prediction based on previous tokens, then passes the joint representation through softmax. The speaker describes this as an autoregressive streaming setup.

[3:13](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=193s) [Diagram connecting an encoder and prediction network to a joint network and softmax.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b3f4057b11140f9be9ee2d3d46f34f98f45f8d5aada59afe72ff22b92d1d739.webp)

Transducer (RNN-T/TDT)
Advanced sequence modeling architecture that predicts the token and its duration, skipping redundant computations for optimal efficiency

P(yₜ,ᵤ | x₍₁:ₜ₎, y₍₁:ᵤ₋₁₎)
Softmax
hₜ,ᵤ
Joint Network
hᵖʳᵉᵤ
hᵉⁿᶜₜ
Prediction Network
Encoder
yᵤ₋₁
xₜ

Figure adapted from: Guo, Jinxi, et al. "Efficient minimum word error rate training of RNN-Transducer for end-to-end speech recognition." arXiv:2007.13802, 2020
NVIDIA

### Attention-based Encoder/Decoder

The encoder feeds an attention-based decoder that produces output probabilities. The speaker says prompt changes let one model handle speech recognition, translation, timestamp prediction and language identification.

[3:44](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=224s) [Transformer encoder and decoder diagram with attention, feed-forward layers, and output probabilities.](https://assets.ai.engineer/sites/aiecode2025/sha256/23529404712b04e119d5e0150b9cb03347e22b13c3bbfb7009c9ae2894e765cb.webp)

Attention-based Encoder/Decoder (AED)
Context-aware decoding for complex transcription tasks

Encoder path:
Inputs
Input Embedding
Positional Encoding
Multi-Head Attention
Add & Norm
Feed Forward
Add & Norm
N×

Decoder path:
Outputs (shifted right)
Output Embedding
Positional Encoding
Masked Multi-Head Attention
Add & Norm
Multi-Head Attention
Add & Norm
Feed Forward
Add & Norm
N×
Linear
Softmax
Output Probabilities

Figure from: Vaswani et al. ”Attention Is All You Need.” Advances in Neural Information Processing Systems 30 (NeurIPS 2017), arXiv:1706.03762. https://arxiv.org/pdf/1706.03762

NVIDIA

### FastConformer

FastConformer adds a subsampling step, moving its main layers from 40 ms to 80 ms intervals. The speaker says this reduces training memory and speeds inference.

[4:24](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=264s) [Stacked diagrams compare Conformer processing at 40 ms with FastConformer processing at 80 ms.](https://assets.ai.engineer/sites/aiecode2025/sha256/67dbccacc69c9dd4fcb7bd6f4835255e68e4aaf8c5d8b258fb4c8b4c7017574b.webp)

FastConformer
The Engine Under Hood

Conformer
40ms rate
40ms rate
40ms rate
40ms rate
40ms rate
40ms rate
40ms rate
40ms rate
20ms rate
10ms rate

FastConformer
80ms
80ms
80ms
80ms
80ms
80ms
80ms
40ms rate
20ms rate
10ms rate

Source: NVIDIA Research. (2023, June 7). Fast-Conformer: Fast and Accurate Speech Recognition with Conformer-RNNT. https://research.nvidia.com/labs/conv-ai/blogs/2023/2023-06-07-fast-conformer/

NVIDIA

### NVIDIA Riva Model Families

Parakeet uses CTC/TDT FastConformer for streaming ASR/AST; Canary uses AED FastConformer for accuracy-focused offline ASR/AST and multitask modeling.

[5:26](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=326s) [Parakeet and Canary illustrations accompany descriptions of their streaming and offline model families.](https://assets.ai.engineer/sites/aiecode2025/sha256/4de9c88193f88c406a996217c0d54ec9990b73406c956a9618acbdb557a1bd4b.webp)

NVIDIA Riva Model Families
Parakeet and Canary

Riva Parakeet
CTC/TDT FastConformer, streaming-optimized for ASR/AST

Riva Canary
AED FastConformer, accuracy-optimized for offline ASR/AST and multitask modeling

Images generated by DALL-E
NVIDIA

### Summary

CTC gets two checks for speed, while RNN-T and AED get two for accuracy. AED alone gets a multitasking check. The speaker favors a variety of models to meet different needs.

[6:21](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=381s) [Table compares RNN-T, AED, and CTC on speed, multitasking, and accuracy.](https://assets.ai.engineer/sites/aiecode2025/sha256/e6b413e9f8614ccf96205f4fa9c10c5a9363624a3617e9331fa7c4fcb961cc70.webp)

Summary

                Fast    Multitask    High Accuracy
RNN-T           ☑       ☒            ☑ ☑
AED             ☒       ☑            ☑ ☑
CTC             ☑ ☑     ☒            ☑

NVIDIA

### Multi/Target-Speaker ASR

A speaker kernel fuses ASR encoder and Sortformer embeddings before the RNNT decoder. An optional query audio input selects target-speaker transcription; without it, the model transcribes multiple speakers.

[6:53](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=413s) [Diagram connects a FastConformer encoder and Sortformer to an RNNT decoder for target-speaker and multi-speaker transcription.](https://assets.ai.engineer/sites/aiecode2025/sha256/b08b39969719bbe9d6fd118a69fbbf2f68509b5f0862e86b0e2b239b69e654d5.webp)

Multi/Target-Speaker ASR
Speaker-informed E2E MS / TS-ASR with Sortformer

Linear Projection 🔥
Speaker Kernel
Fast Conformer ASR Encoder 🔥
Sortformer ❄
RNNT Decoder 🔥

Query Audio (Optional)
Multitalker Audio Recording

TS-ASR
y eah that’s true hmm that makes sense

MS-ASR
<spk0> yeah <spk0> that's <spk0> true
<spk1> I <spk1> want <spk1> month ...
<spk2> think <spk0> hmm <spk0> that
<spk0> makes <spk0> sense
<spk3> well <spk3> because ...

Figure from: Wang, Jinhan, et al. "META-CAT: Speaker-Informed Speech Embeddings via Meta Information Concatenation for Multi-talker ASR." arXiv preprint arXiv:2409.12352, 2024. https://arxiv.org/abs/2409.12352

NVIDIA

### Additional Models We Offer

VAD detects speech segments, an external language model rescores transcripts, and normalization, punctuation, and diarization make the output easier to read and attribute to speakers.

[8:22](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=502s) [List of voice activity detection, language modeling, text normalization, punctuation and capitalization, and speaker diarization models.](https://assets.ai.engineer/sites/aiecode2025/sha256/419398826247a3f2b3c64f6b256d46194c916971613e30a50d87c1ed8af17b8b.webp)

Additional Models We Offer
Better Accuracy/Customization/Readability

• Voice Activity Detection (VAD)
  • Detects speech segments for better noise-robustness
  • SileroVAD and MarbleNet based VAD models
• External Language Model (LM)
  • Re-scores ASR transcription for better accuracy/customization
  • n-gram based language models
• Inverse Text Normalization (ITN)/Text Normalization (TN)
  • Converts spoken forms to written forms in text for better readability
  • WFST-based ITN model
• Punctuation and Capitalization (PnC)
  • Adds punctuation and capitalization to the transcription for better readability
  • BERT-based PnC models
• Speaker Diarization
  • Identifies multiple speakers in conversation
  • Sortformer-based model

NVIDIA

### Hugging Face ASR Leaderboard

The leaderboard ranks models by average word error rate, with lower scores ranked higher. NVIDIA models occupy three of the top five places shown.

[9:45](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=585s) [A speech recognition leaderboard comparing models by error rate and other metrics.](https://assets.ai.engineer/sites/aiecode2025/sha256/6ad76c57d0c2d29fc9df7b3ae0f74c0abd50166908458f5f20b384371e899603.webp)

Hugging Face ASR Leaderboard

Leaderboard
Speech Recognition

The 🤗 Open ASR Leaderboard ranks and evaluates speech recognition models on the Hugging Face Hub.
We report the Average WER (lower the better) and RTFx (higher the better). Models are ranked based on their Average WER, from lowest to highest. Check the Metrics tab to understand how the models are evaluated.
If you want results for a model that is not listed here, you can submit a request for it to be included.
The leaderboard currently focuses on English speech recognition, and will be expanded to multilingual evaluation in later versions.

Leaderboard | Metrics | Request a model here! | About

model | Average WER | RTFx | License | AMI | Earnings22 | Gigaspeech | LS Clean | LS Other | SPGISpeech
nvidia/parakeet-tdt-0.6b-v2 | 6.05 | 3386.82 | Open | 11.16 | 11.15 | 9.74 | 1.69 | 3.19 | 2.17
microsoft/Phi-4-multimodal-instruct | 6.14 | 62.12 | Open | 11.45 | 10.5 | 9.77 | 1.67 | 3.82 | 3.11
nvidia/canary-1b-flash | 6.35 | 1045.75 | Open | 13.11 | 12.77 | 9.85 | 1.48 | 2.87 | 1.95
nvidia/canary-1b | 6.5 | 235.34 | Open | 13.9 | 12.19 | 10.12 | 1.48 | 2.93 | 2.06
nyrahealth/CrisperWhisper | 6.67 | 84.05 | Open | 8.71 | 12.89 | 10.24 | 1.82 | 4 | 2.7
elevenlabs/scribe_v1 | 6.88 | NA | Proprietary | 14.43 | 12.14 | 9.66 | 1.79 | 3.31 | 3.3
nvidia/parakeet-tdt-1.1b | 7.01 | 2390.61 | Open | 15.87 | 14.49 | 9.52 | 1.4 | 2.6 | 3.16
assemblyai/assembly_best | 7.03 | NA | Proprietary | 15.64 | 13.54 | 9.5 | 1.74 | 3.11 | 1.81
revai/fusion | 7.12 | NA | Proprietary | 10.93 | 12.09 | 9.41 | 2.88 | 6.23 | 4.05
nvidia/canary-180m-flash | 7.12 | 1233.58 | Open | 14.86 | 12.33 | 10.51 | 1.73 | 4.35 | 2.26
nvidia/parakeet-rnnt-1.1b | 7.12 | 2053.15 | Open | 17.01 | 13.94 | 9.89 | 1.45 | 2.5 | 2.93
[partially obscured row: illegible]

Show proprietary models

Interface text:
13 of 24
NVIDIA

### The End of Awkard Transcriptions

A Parakeet V2 demo plays a song alongside timestamped transcription segments. The interface lets viewers play individual rows and download the transcript as a CSV.

[10:18](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=618s) [A lyric transcription demo showing an audio waveform, transcription controls, and timestamped transcript segments.](https://assets.ai.engineer/sites/aiecode2025/sha256/f997f807e6e88ba96912f73ed4a08135617f2d2d09df4ec05662740b8631faeb.webp)

The End of Awkard Transcriptions
Demo of Lyric Transcription by Riva Parakeet V2

Upload Audio File
2:54 / 9:14
Example Audio Files (Click to Load)
[illegible].mp3
Transcribe Uploaded File

Transcription Results (Click row to play segment)
Download Transcript (CSV)
Transcription Segments
Start (s) | End (s) | Segment
164.88 | 171.68 | The speech here is coming from a rapper, one of our employees, John Delabana, and it is really fantastic.
171.92 | 172.80 | Take a look.
174.96 | 176.96 | Living for the now, long as time allows.
177.12 | 178.72 | I'ma keep on switching different styles.
178.80 | 180.08 | Keep creative on a cloud.
180.32 | 183.92 | Sweat is on my brow because I'm running all these tracks just to keep them running back.
184.00 | 185.36 | You know, the drill to cordon back.
185.44 | 186.72 | And I've been practicing my crap.
186.80 | 188.00 | Dedicate this play to Kobe.
188.24 | 190.80 | What could be a bigger legacy than making it a story?

NVIDIA

Interface text:
huggingface.co/spaces/nvidia/parakeet-tdt-0.6b-v2
1x

### Model Development Process

A right-pointing arrow maps five stages from data sourcing through training and infrastructure to validation and robustness.

[10:52](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=652s) [A five-stage model development process from data sourcing through validation and robustness.](https://assets.ai.engineer/sites/aiecode2025/sha256/e2549096eb468b5b421b18add2bb2002ad4b7448d4d1c58df868f211502d1d57.webp)

Model Development Process
From Data Sourcing to Real-World Robust Models

Data Sourcing
Training Pipeline
Data Infrastructure
Training Methodologies
Validation & Robustness

NVIDIA

### Data Sourcing

Open-source data supplies breadth, while proprietary data supports fine-tuning. Source text and pseudo-labels are validated with commercial models using two-pass agreement; minimal post-processing preserves flexibility for developers.

[11:10](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=670s) [Slide describing speech data sources, labels, and post-processing.](https://assets.ai.engineer/sites/aiecode2025/sha256/20cd8a24a2f6086d917e792d71a1b0465f3784e0aedc575dcf11032da9a53b5d.webp)

Data Sourcing

• ASR Primary Sourcing
  • Multilingual Coverage, Dialect Sensitive
• Open Source and Proprietary data
  • Open Source -> breadth and robustness
  • Proprietary -> Finetune
• Labels
  • Mixture of Pseudolabeling plus source text
  • Validate both with SOTA Commercial models - 2 pass agreement
  • Additional filtering stages for AST
• Post-processing
  • Minimal postprocessing. Majority of source preserved for end-stage developer
  • Allows for quick adjustment to niche caches (diacritization, ITN, PnC)

NVIDIA

### Training

NeMo supports training with GPU optimization and data bucketing, while Lhotse speeds audio loading and storage infrastructure moves data between clusters. The slide contrasts unprompted ASR with prompted AST.

[11:59](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=719s) [Slide describing software infrastructure, data infrastructure, and training methodology.](https://assets.ai.engineer/sites/aiecode2025/sha256/a43d290da8dedf643c9bb2e276108a34fcb37b1d26dfb8ee3ea5f6c68d2643da.webp)

Training

• Software Infrastructure
  • NVIDIA NeMo Research toolkit - shared being MLOps and research teams for easy transfer of new architectures and training methodologies
  • Native optimization techniques (bucketing, GPU usage optimization, weighted sampling) maximizes training performance.
  • Pytorch-Lightning based infrastructure allows easy monitoring of jobs and compatibility with major monitoring systems (WanDB, Tensorboard)
• Data infrastructure
  • Lhotse backed data processing minimizes latency in audio data i/O operations
  • AIStore/MSC infrastructure allows quick movement of data between cluster instances
• Training methodology
  • Prompted vs. Unprompted Models
  • ASR -> Unprompted (Implicit task identification)
  • AST -> Prompted (Target Language Identification)
  • Supported by NeMo for major prompting systems from LLM

NVIDIA

### Validation

NVIDIA mixes open-source and proprietary data for validation. Its out-of-domain evaluation suite tests dialect variation, environment quality, and speech duration for internal and external model comparisons.

[12:49](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=769s) [Slide describing benchmark coverage and robustness evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/2576c39fbfd133b97bd8a883998f3ad099e3f4d59b306fb7e9223d9ec087334c.webp)

Validation

• Also mix of open-source + proprietary data
  • We validate on all major open-source benchmarks
• Robustness evaluation
  • Extensive out of domain evaluation suite
  • Focus on dialect variation, environment quality, duration of speech
  • Used for both internal and external model comparisons

NVIDIA

### NVIDIA NIM Optimized Inference Microservices

NVIDIA NIM packages Riva inference with standard APIs, custom-model support and optimized engines. The speaker describes low-latency streaming and high-throughput offline use.

[13:17](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=797s) [Diagram of the NVIDIA NIM software stack alongside deployment and enterprise benefits.](https://assets.ai.engineer/sites/aiecode2025/sha256/669a8a7d4e116517725a80c5855c6774bff87e88d62b427f6c10c395450a78d3.webp)

NVIDIA NIM Optimized Inference Microservices
Accelerated runtime for speech AI

NVIDIA NIM
Prebuilt container
Industry standard APIs
Support for custom models
Domain specific code
Optimized inference engines

Deploy anywhere and maintain control of generative AI applications and data
Simplified development of AI application that can run in enterprise environments
Day 0 support for all generative AI models providing choice across the ecosystem
Improved TCO with best latency and throughput running on accelerated infrastructure
Best accuracy for enterprise by enabling tuning with proprietary data sources
Enterprise software with feature branches, validation and support

NVIDIA

### Customization Across NVIDIA Riva ASR Pipeline

Riva offers customization throughout speech recognition: word boosting and speech hints at input, model tuning and custom vocabulary in the pipeline, and rules for formatting the final text.

[14:25](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=865s) [Speech-to-text pipeline with customization options for word boosting, model tuning, vocabulary, and normalization.](https://assets.ai.engineer/sites/aiecode2025/sha256/fad21a59fb61218c370e8cf76c89d6e019fd87c83a668cbffed8c09a484aa459.webp)

Customization Across NVIDIA Riva ASR Pipeline
Accurate | Customizable | Real-time

Word Boosting* for:
• Product Names
• Jargon
• Context-Specific Knowledge

Speech → RIVA ASR PIPELINE → Text
Voice Activity Detection → Feature Extraction → Acoustic Model → Decoder & N-Gram Language Model → Punctuation & Capitalization Model → Inverse Text Normalization Model

Text Normalization
• “at 10:00” => “at ten o’clock”

Speech Hints* for:
• Priming Model to Expect Specific Information Based on Conversation Flow
  (Credit Card Info, Order Number, Home Address, etc.)

Fine-Tuning* or Retraining for:
• Different Accents
• Noisy Environments
• Specific Pronunciation

Retraining LM*

Fine-Tuning or Custom Model*

Adding Custom Pronunciation & Vocabulary for:
• Domain-Specific Language
• Lingo

Rule Modification*
• “at ten o’clock” => “at 10:00”
• “confirmation number one tee four oh seven en” => “Conf #1T407N”

Customization during:
Training (server-side) – Need to rebuild/restart server
Deployment
Inference (client-side) – No need to rebuild/start Riva server

* Generic ASR Services offer black box with rigid & selectively limited customization

NVIDIA

### Experience NVIDIA NIM

The NVIDIA NIM speech catalog groups models for speech recognition and text-to-speech. The speaker points to it as a place to find available Riva models.

[15:29](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=929s) [NVIDIA NIM speech catalog showing speech recognition and text-to-speech models, with a QR code.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a9fb184c50fc7dd3f42d732dea321c2a466662c93356733fd04c4629d42b30e.webp)

Experience NVIDIA NIM
https://build.nvidia.com/explore/speech

Speech

Automatic Speech Recognition (ASR)
Connect generative AI models to speech by transcribing spoken audio to text.

RUN ANYWHERE
nvidia
parakeet-1.1b-rnnt-multilingual-asr
High accuracy and optimized performance for transcription in 25 languages.
asr  multilingual  nvidia nim  +2

RUN ANYWHERE
nvidia
parakeet-ctc-1.1b-asr
Record-setting accuracy and performance for English transcription.
asr  english  nvidia nim  +3

RUN ANYWHERE
nvidia
parakeet-ctc-0.6b-asr
State-of-the-art accuracy and speed for English transcriptions.
asr  batch  english  fast  +4

RUN ANYWHERE
nvidia
canary-1b-asr
Multi-lingual model supporting speech-to-text recognition and translation.
asr  ast  multilingual  nvidia nim  +5

RUN ANYWHERE
openai
whisper-la[illegible]
[illegible]
asr  ast

Convert Text to Speech (TTS)
Voice generative AI models by converting written text to spoken audio.

PREVIEW
nvidia
magpie-tts-zeroshot
Expressive and engaging text-to-speech, generated from a short audio sample.
nvidia nim  nvidia riva  tts  +1

RUN ANYWHERE
nvidia
magpie-tts-multilingual
Natural and expressive voices in multiple languages. For voice agents and brand ambassadors.
nvidia nim  nvidia riva  tts  +2

RUN ANYWHERE
nvidia
fastpitch-hifigan-tts
Expressive and engaging English voices for Q&A assistants, brand ambassadors, and service robots.
nvidia nim  text-to-speech

Neural Machine Translation (NMT) & Audio Speech Translation (AST)
Create multilingual generative AI models by translating speech and text between languages.

Interface text:
DISCOVER
Models
Reasoning
Vision
Visual Design
Retrieval
Speech
Biology
Simulation
Climate & Weather
Safety & Moderation
Industries
Automotive
Gaming
Healthcare
Industrial
Robotics
NVIDIA

### How to get started with NVIDIA Riva

Six QR codes point to Riva resources, from the quick start guide and developer forum to tutorials, best practices, and a NeMo guide for fine-tuning models.

[15:54](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=954s) [Six QR codes link to Riva guides, the developer forum, the NGC catalog and GitHub tutorials.](https://assets.ai.engineer/sites/aiecode2025/sha256/2cce977ded405a21010a3ec766daee2f853c9487b29cab3a0b30ac7cb66bf8c7.webp)

How to get started with NVIDIA Riva

Quick Start Guide
Developers Forum
NeMo Framework Fine-Tuning Guide
Assets - NGC Catalog
GitHub Tutorials
Best Practices Guide

NVIDIA

## Full transcript

[0:00](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=0s) Hi, everyone. I'm Travis.

[0:03](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=3s) Hello, I'm Byungjoong.

[0:04](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=4s) I'm Jaehan.

[0:06](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=6s) And we're here to talk to you today about the end of awkward AI transcripts from NVIDIA.

[0:12](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=12s) So what we're gonna go over today is basically how we develop models at NVIDIA Speech, um, for NVIDIA Speech AI. We're gonna go over the model architectures, um, our main approach to how we develop models for our customer base, um, the entire development process, and our focus on deployment and customization.

[0:32](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=32s) So who are we? Well, we're with NVIDIA Riva. We focus on enterprise-level s- um, speech AI model deployment. We cover speech translation, text-to-speech development, speech recognition, speech translation. Basically everything so that, um, we can at an enterprise level allow customers to provide the best possible conversational AI that is possible.

[0:56](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=56s) And we... Our focus is generally on low latency, highly efficient models that, uh, can be used, um, on embedded devices.

[1:06](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=66s) So, um, at NVIDIA there's generally, like, four categories we focus on when we're thinking about model development. First is robustness. Are our models going to work in noisy environments a- as well as they work in clean environments?

[1:20](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=80s) What type of sound quality are we trying to meet? Are we going to worry about telephony? Are we going to worry about a bunch of environmental contamination factors? Um, we also consider ourselves with coverage.

[1:32](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=92s) What are the domains that our customer base wants? Is it gonna be medical? Is it going to be entertainment? Is it going to be, um, just straight up call center based?

[1:42](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=102s) What is the language demands that we need? Do we wanna focus on monolingual development or multilingual development? How will dialect play into our model cases? And is there a pro- problem of code switching that we should accommodate for in development?

[1:57](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=117s) We also focus on personalization. We want to make sure that our customers are able to have the exact model that they need for their needs. Sometimes we gotta focus on target speaker ASR.

[2:09](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=129s) Sometimes we gotta focus on word boosting for uncommon vocabulary. Sometimes we just get down and dirty with old-fashioned text normalization FST models just to make sure that you're getting the exact output you want.

[2:21](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=141s) And then finally, we focus on deployment cases. How does speed and accuracy trade off here? Which is more important, or do we want a middle ground of the two?

[2:30](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=150s) Um, do we want our models to be, have h- high variety, or are we gonna focus on efficiency and kinda leave like, um, the Swiss ar- army knives in, in our back pocket?

[2:42](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=162s) So what type of models do we use to accomplish this? Well, if you're from a speech AI background, a lot of this is really familiar stuff. First off, we still rely on the old-fashioned CTC models, 'cause we keep finding that non-autoregression, um, decoding cases is really optimal if you need high speed inference, especially in streaming environments where

[3:04](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=184s) you can just run CTC over a, um, chunk data set, use that as a streaming model, and you can move on with your, um, next project.

[3:14](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=194s) But of course, we still like high accuracy. So when we find that just simply non-autoregression is not, um, a ideal, then we s- rely on the old-fashioned RNN-T or, um, you use TDT models, which is a variant, um, developed at NVIDIA.

[3:29](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=209s) Here we've used the audio output of an encoder with an internal LM so that we can have aut- autoregressive streaming setups, um, for deployment. Then sometimes you want even more accuracy.

[3:43](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=223s) Sometimes you don't want to worry about streaming, in which case we also offer, um, attention encoder-decoder setups. We are all familiar with them at this point. Whisper, ChatGPT, LLMs, and yeah, we offer those here.

[3:59](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=239s) We find that they're really good if you want a highly accurate model, um, with less focus on alignment, and more importantly, accommodate a lot of ta- tasks within a single model.

[4:10](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=250s) We find, um, auto, um, we find transformer decoders really great when you need to incorporate speech translation, timestamp prediction, language identification, and speech recognition all within a single model with simple prompt changes.

[4:26](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=266s) But across all of our different, um, decoding platforms, we have one unifying tool, which is that our fundamental architecture is the Fast Conformer. Through a lot of empirical trials, we found that the original Conformer model can be even greatly, um, subsampled so that instead of just the conventional forty millisecond time s- step compression, you can switch...

[4:48](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=288s) You can add in a next, another step of subsampling and get eighty millisecond compression. And we find that along with allowing you to have very small audio inputs and thus really lightens your memory load on training, it also makes training a lot more efficient because you can get way quicker convergence with, um, n- not as much data

[5:10](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=310s) as needed. Further, it allows really fast inference because now you're chunking all your data into just eighty millisecond time steps. And so we found that this is a really strong architecture that we have at the backbone of all of our previous m- um, decoding offerings.

[5:27](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=327s) And this allows us to split our model offerings into two options. On the left we have Riva Parakeet, which is focused on our streaming speech recognition cases, and this is where we put our CTC and our TDT models, and the focus is on streaming.

[5:44](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=344s) If you want speech recognition, speech translation, target speaker ASR, you're gonna run it through a K- Parakeet model so that you can have very fast, um, and efficient, um, recognition Meanwhile, on our other option, we have Riva Canary, which is where we put our Fast Conformer models.

[6:05](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=365s) This is where we focus on accuracy and mul- and multitask modeling. This is where we're going to get the best accuracy as possible while not really caring that much about speed, though we do push for strong speed.

[6:23](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=383s) And so be- being able to cover all this, we get a really, really comprehensive toolkit in which we can offer to our customers, um, a mixture of fast multitasking or high-accuracy models.

[6:36](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=396s) And most important is that our focus on we have the model to meet the need, as opposed to an idea that one model fits all. We focus on variety and coverage rather than, um, unifying and, um, keeping everything under one hood.

[6:57](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=417s) Given the success of the Parakeet ASR model, it can be easily extended to the multi-speaker and target-speaker, uh, scenario with the integration of the diarization model Sortformer. The Sortformer is a end-to-end neural diarizer that following the arrival time sorting principle.

[7:16](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=436s) In the simply words, it's standing for who comes first. So the Sortformer behaves as a bridge between the speaker timestamp from the diarization and the speaker token that can be recognized by the ASR model.

[7:29](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=449s) In our mo- in our model architecture development principle, we fuse the ASR encoder embedding and the Sortformer embedding through the speaker kernel in providing the full-fledged who spoke what and when problem.

[7:43](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=463s) And this objective, the, this model can be fine-tuned with a really simple objective similar to the any ASR model training, uh, perspective. And also, by feeding the model, or feeding the model with the optional query audio or not, we can make the model to conduct the target-speaker ASR task or the sing- or single multi-speaker, uh, ASR

[8:06](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=486s) task. And moreover, our unified model architecture consists of ASR encoder and Sortformer, but it can not only be applied in the parallel joint manner, but can also be used as the cascade system, just like conventional multi-speaker system.

[8:26](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=506s) Yeah, uh, we have additional models we offer to improve further accuracy, customization, and readability. Um, the voice activity detection, VAD, uh, detects speech segments for better noise robustness, and we offer DistilBERTaVAD and the MarbleNet-based VAD models.

[8:47](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=527s) The external language model, uh, rescores ASR transcription for better accuracy and customization. We have n-gram-based language models, uh, in our, uh, the Riva ASR pipelines. Text normalization and the inverse text normalization convert spoken terms to, uh, written forms in text for better readability, and we have, uh, the

[9:11](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=551s) WFST-based ITN models. Punctuation capitalization, PNC, uh, adds punctuation and capitalization to the transcription for better readability, and, uh, BERT-based PNC models is supported.

[9:27](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=567s) Uh, finally, the speaker diarization identifies, uh, multiple speakers in conversation, uh, and we have the Sortformer-based speaker diarization models, uh, in Cascade model, as well as the, uh, upcoming for the, uh, end-to-end, uh, models.

[9:50](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=590s) And so, um, given this level of customization, it doesn't, like, really surprise that, like, it pays off, um, when it comes to, um, rankings. On the Hugging Face Open ASR leaderboard, the majority of the top five models do come from NVIDIA, and all of it does is come down to this approach on a focus on customization and

[10:11](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=611s) variety. But, you know, rankings aren't everything. So let's try out a demo.

[10:19](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=619s) So let's say you're trying to run a recognition system, but you have a... but it, it's a song you're going over. Can our models work?

[10:27](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=627s) Living for the now, long as time allows. I'ma keep on switching different styles, keep creative on a cloud. Sweat is on my brow 'cause I'm running on these tracks just to keep the running back, you know the drill, the quarterback.

[10:37](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=637s) And I've been practicing my craft, dedicate this play to Kobe. What could be a bigger legacy than making it as-

[10:44](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=644s) Yeah, seems to work pretty well. Accurate transcription even in a noisy setting.

[10:53](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=653s) Okay, so that's our architectures, that's our approach to development. So how does it actually go into training? Is there anything special we do with training to, um, uh, to meet this demand?

[11:04](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=664s) And the answer is honestly not really. We really focus on to met the fundamentals when it comes to data development. Like many labs, um, our focus is on robustness when it comes to sourcing data, multilingual coverage, and a, a, an ear for dialect sensitivity.

[11:22](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=682s) We try and get as much language documentation as possible before, um, so that we know exactly what type of data spans we wanna get. We incorporate both open source and proprietary data, with the former allowing us to focus on variety and domain shift, while the latter lets us focus on high-quality NC data.

[11:44](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=704s) We incorporate pseudo-labeling, where we use, um, the top-of-the-line models that are available for commercial use, and we take transcripts from them, allowing us to benefit from further developments in the community and further developments from our own releases.

[12:01](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=721s) And then for training, again, a lot of it is just standard available stuff. Um, the NeMo research, um, toolkit that is used for our model training is an open source available, um, library available for anyone in the community to use.

[12:19](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=739s) It's, um, it's tools for GPU maximalization, data bucketing, um, high-speed data loading through the Lhotse backend, all available for just general use. And so we just focus on this approach where we can maximize our data, we can maximize the, um, speed in which we can ingest data across different settings.

[12:41](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=761s) Most of our data is in stored on a object store infrastructure in which we can quickly migrate between different cluster settings.

[12:50](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=770s) And then for validation, just as much as with training, we focus on a nice mixture of coverage in both open source and proprietary data. We make sure that by the time data be-- by the time our models actually reach end users, we've gone through as much, um, bias and domain testing as possible across all possible language categories,

[13:14](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=794s) just to make sure that our models are as robust as possible.

[13:22](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=802s) The trained model, uh, is now deployed to, uh, NVIDIA Riva through NVIDIA NIM for low latency and high throughput inference. The high performance, uh, inference is powered by NVIDIA TensorRT optimizations, um, and the NVIDIA Triton Inference Server.

[13:41](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=821s) And it is available for gRPC-based microservice for low latency streaming, as well as the high throughput offline use cases. The NVIDIA Riva is fully containerized, and it can, uh, easily scale to hundreds of the, um, the parallel streams, and, uh, it can be run on-prem in any cloud at the edge or, uh, embedded platforms to

[14:05](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=845s) support a variety of the applications, including contact centers, consumer applications, and video conferencing.

[14:15](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=855s) NVIDIA NIM offers prebuilt container industry-standard API support for custom models, uh, and optimized inference engines.

[14:27](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=867s) So one of the pain point in real scenarios from customers is, um, customization, because every use case, um, uh, every application requires domain knowledge, including the medical terms in medical use case, um, menu names in food ordering, and, um, some of the acoustic conditions, telephony, uh, and noisy environment in contact

[14:52](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=892s) centers. So NVIDIA, uh, Riva offers, um, the, uh, customization features at every stages. So, uh, we can fine-tune our acoustic model from the Parakeet-based model, Canary-based models, and then we can fine-tune the n-gram, um, external language model, punctuation model, and the inverse text normalization models.

[15:17](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=917s) And also we offer the word boosting to, uh, recognize, uh, better on some of the, uh, product names, jargon, and context-specific, uh, knowledges.

[15:32](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=932s) So, um, we are offering our, uh, the Riva models in NVIDIA NIM. So, um, you can, um, visit, uh, our website, build.nvidia.com/explore/speech.

[15:47](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=947s) Then you can find more, um, the available, uh, Riva models, uh, in the website.

[15:56](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=956s) Here is the more thing, uh, how to get started with NVIDIA Riva, uh, quick starter guide, uh, developers pro-- uh, developers forum, and, uh, uh, the fine-tuning guide, how to fine-tune our models in the, in, uh, NeMo frameworks.

[16:19](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=979s) Yeah, so that's it. Thank you.

[16:22](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=982s) Thank you, everyone.

[16:22](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=982s) Thank you
