Speaker Separation

Diarize multi-speaker audio and fan out one task per detected speaker so that downstream stages can score each speaker’s audio independently. NeMo Curator ships two diarization stages built on NVIDIA’s SortFormer family. Both target up to 4 speakers per file; choose based on whether your workload is offline batch curation or streaming/online.

Understanding Diarization

What Diarization Does

Diarization answers “who spoke when?” — it segments an audio stream into per-speaker regions, identifying that speaker A talks 0.0–3.5s, speaker B talks 3.5–7.0s, speaker A returns at 7.0–9.0s, and so on. The output is one AudioTask per speaker, each containing only that speaker’s audio.

This unlocks per-speaker filtering: pipelines can rerun VAD, UTMOS, SIGMOS, and the band filter separately on each speaker, dropping individual low-quality speakers without losing the rest of the recording.

Choosing a Stage

For most curation pipelines, SpeakerSeparationStage (offline) is the right choice. Use the streaming variant only when you need bounded latency or RTTM output.

Offline Speaker Separation

Step 1: Configure the Stage

from nemo_curator.stages.audio.segmentation.speaker_separation import SpeakerSeparationStage

speaker_sep = SpeakerSeparationStage(
    model_path="nvidia/diar_sortformer_4spk-v1",
    exclude_overlaps=True,
    min_duration=0.8,
    gap_threshold=0.1,
    buffer_time=0.5,
)
pipeline.add_stage(speaker_sep)

The stage produces a fan-out list of AudioTask objects, one per detected speaker, each carrying:

• speaker_id — speaker identifier (0, 1, 2, …)
• num_speakers — total speakers found in this file
• duration_sec — duration of this speaker’s audio
• waveform — that speaker’s torch tensor with overlapping regions removed (when exclude_overlaps=True)

GPU is required (Resources(cpus=1.0, gpus=1.0) by default).

Step 2: Tune Overlap and Gap Handling

Speaker overlap regions (where multiple speakers talk simultaneously) and short gaps between same-speaker turns affect output quality:

SpeakerSeparationStage Parameters

Streaming Speaker Diarization

When to Use Streaming

The streaming variant (InferenceSortformerStage) is purpose-built for two use cases:

1. Online / chunked workloads — bounded latency requirements that can’t tolerate waiting for the full utterance.
2. RTTM output — downstream tooling (Kaldi, ESPnet, evaluation harnesses) consumes RTTM-format diarization output.

For pure offline curation, SpeakerSeparationStage is faster and more accurate.

Step 1: Configure the Stage

from nemo_curator.stages.audio.inference.speaker_diarization.sortformer import InferenceSortformerStage

streaming = InferenceSortformerStage(
    model_name="nvidia/diar_streaming_sortformer_4spk-v2.1",
    rttm_out_dir="./rttm",
    chunk_len=340,            # ~30.4 seconds latency in 80 ms frames
    inference_batch_size=1,
)
pipeline.add_stage(streaming)

This stage does not fan out per speaker — instead it writes a diar_segments list onto the input AudioTask. Use it as a metadata-enriching stage; downstream code consumes the diar_segments field directly.

Step 2: Tune Latency

chunk_len controls latency vs accuracy:

Streaming-mode evaluation on CallHome-eng0 (139 files) at the default settings: 6.2% macro DER, 6.0% weighted DER at a 0.25-second collar.

InferenceSortformerStage Parameters

Default resource allocation: Resources(cpus=1.0, gpu_memory_gb=8.0).

Complete Speaker Separation Pipeline

A pipeline that diarizes, then runs per-speaker quality filters:

from nemo_curator.pipeline import Pipeline
from nemo_curator.backends.xenna import XennaExecutor
from nemo_curator.stages.audio.preprocessing.mono_conversion import MonoConversionStage
from nemo_curator.stages.audio.segmentation.vad_segmentation import VADSegmentationStage
from nemo_curator.stages.audio.preprocessing.concatenation import SegmentConcatenationStage
from nemo_curator.stages.audio.segmentation.speaker_separation import SpeakerSeparationStage
from nemo_curator.stages.audio.filtering.utmos import UTMOSFilterStage
from nemo_curator.stages.audio.postprocessing.timestamp_mapper import TimestampMapperStage
from nemo_curator.stages.audio.io.convert import AudioToDocumentStage
from nemo_curator.stages.text.io.writer import JsonlWriter

pipeline = Pipeline(name="speaker_diarization")

# 1. Normalize and segment
pipeline.add_stage(MonoConversionStage(output_sample_rate=48000))
pipeline.add_stage(VADSegmentationStage(min_duration_sec=2.0))

# 2. Concatenate surviving segments per file
pipeline.add_stage(SegmentConcatenationStage(silence_duration_sec=0.5))

# 3. Diarize and fan out per speaker
pipeline.add_stage(SpeakerSeparationStage(exclude_overlaps=True))

# 4. Per-speaker quality filter
pipeline.add_stage(UTMOSFilterStage(mos_threshold=3.5))

# 5. Resolve final timestamps
pipeline.add_stage(
    TimestampMapperStage(
        passthrough_keys=["speaker_id", "num_speakers", "utmos_mos"]
    )
)

# 6. Export
pipeline.add_stage(AudioToDocumentStage())
pipeline.add_stage(JsonlWriter(path="./per_speaker_audio"))

executor = XennaExecutor()
pipeline.run(executor)

Best Practices

• Use offline mode unless you specifically need streaming: SpeakerSeparationStage is faster and more accurate than InferenceSortformerStage for batch curation.
• Run VAD + concat before diarization: feeding diarization a clean concatenated speech-only waveform (no long silences) is cheaper and more reliable than feeding raw audio.
• Pair with per-speaker quality filters: place the filtering chain (VAD → Band → UTMOS → SIGMOS) after speaker separation so each speaker’s audio is scored independently. Bad speakers get dropped; good speakers from the same file are kept.
• Mind the 4-speaker model limit: both stages target up to 4 speakers per file. Files with more speakers will likely produce degraded diarization.
• Don’t enable exclude_overlaps=False for training data: overlapping speech is hard for downstream models; only disable when explicitly preserving natural conversation.

Related Topics

• Preprocessing Stages — SegmentConcatenationStage and TimestampMapperStage are typically paired with speaker separation.
• VAD Segmentation — typical upstream stage producing the segments fed into diarization.
• AudioDataFilterStage Composite — bundles offline speaker separation with per-speaker filters into the standard pipeline.