Concepts
Learn about the core components and concepts introduced by NeMo Curator.
Modality Concepts
Learn about working with specific modalities using NeMo Curator.
Learn about text data curation, covering data loading and processing (filtering, classification, deduplication).
Image Curation ConceptsExplore key concepts for image data curation, including scalable loading, processing (embedding, classification, filtering), and dataset export.
Video Curation ConceptsDiscover video data curation concepts, such as distributed processing, pipeline stages, execution modes, and efficient data flow.
Audio Curation ConceptsLearn about speech data curation, ASR inference, quality assessment, and audio-text integration workflows.
Universal Concepts
Core concepts that apply across all modalities in NeMo Curator.
Comprehensive overview of deduplication techniques across text, image, and video modalities including exact, fuzzy, and semantic approaches.
Resource AllocationHow NeMo Curator allocates CPUs, GPUs, and memory across pipeline stages for optimal hardware utilization.
StreamingHow streaming execution processes data in batches for constant memory usage and higher GPU utilization.
Auto-BalancingHow the executor automatically balances resources across pipeline stages to eliminate bottlenecks.
ThroughputScale from a single GPU to multi-node clusters with near-linear scaling.