Deduplication

For fine-grained control, break semantic deduplication into separate stages:

from nemo_curator.stages.deduplication.id_generator import create_id_generator_actor
from nemo_curator.stages.text.embedders.vllm import VLLMEmbeddingModelStage
from nemo_curator.stages.deduplication.semantic import SemanticDeduplicationWorkflow

# 1. Create ID generator
create_id_generator_actor()

# 2. Generate embeddings separately (using vLLM)
embedding_pipeline = Pipeline(
    stages=[
        ParquetReader(file_paths=input_path, _generate_ids=True),
        VLLMEmbeddingModelStage(
            model_identifier="google/embeddinggemma-300m",
            text_field="text",
            embedding_field="embeddings",
        ),
        ParquetWriter(path=embedding_output_path, fields=["_curator_dedup_id", "embeddings"])
    ]
)
embedding_out = embedding_pipeline.run()

# 3. Run clustering and pairwise similarity
semantic_workflow = SemanticDeduplicationWorkflow(
    input_path=embedding_output_path,
    output_path=semantic_workflow_path,
    n_clusters=100,
    id_field="_curator_dedup_id",
    embedding_field="embeddings",
    eps=None  # Skip duplicate identification for analysis
)
result = semantic_workflow.run()

# 4. Analyze results and choose eps parameter
# 5. Identify and remove duplicates

This approach enables analysis of intermediate results and fine-grained control.

When assign_id=True (IDs auto-assigned):

• Duplicate IDs file contains _curator_dedup_id column
• Set ids_to_remove_duplicate_id_field="_curator_dedup_id"
• id_generator_path is required

When assign_id=False (using existing IDs):

• Duplicate IDs file contains the column specified by id_field (e.g., "id")
• Set ids_to_remove_duplicate_id_field to match your id_field value
• id_generator_path not required

Exact Deduplication:

• Removing identical copies of documents
• Fast initial deduplication pass
• Datasets with numerous exact duplicates
• When speed is more important than detecting near-duplicates

Fuzzy Deduplication:

• Removing near-duplicate documents with minor formatting differences
• Detecting documents with small edits or typos
• Fast deduplication when exact matching misses numerous duplicates
• When speed is important but some near-duplicate detection is needed

Semantic Deduplication:

• Removing semantically similar content (paraphrases, translations)
• Large, diverse web-scale datasets
• When meaning-based deduplication is more important than speed
• Advanced use cases requiring embedding-based similarity detection

You can combine deduplication methods for comprehensive duplicate removal:

1. Exact → Fuzzy → Semantic: Start with fastest methods, then apply more sophisticated methods
2. Exact → Semantic: Use exact for quick wins, then semantic for meaning-based duplicates
3. Fuzzy → Semantic: Use fuzzy for near-duplicates, then semantic for paraphrases

Run each method independently, then combine duplicate IDs before removal.

GPU Acceleration

All deduplication workflows require GPU acceleration:

• Exact: Ray backend with GPU support for MD5 hashing operations
• Fuzzy: Ray backend with GPU support for MinHash computation and LSH operations
• Semantic: GPU required for embedding generation (transformer models), K-means clustering, and pairwise similarity computation

GPU acceleration provides significant speedup for large datasets through parallel processing.

Hardware Requirements

• GPU: Required for all workflows (Ray with GPU support for exact/fuzzy, GPU for semantic)
• Memory: GPU memory requirements scale with dataset size, batch sizes, and embedding dimensions
• Executors: Can use various executors (XennaExecutor, RayDataExecutor) with GPU support

Backend Setup

For optimal performance with large datasets, configure Ray backend:

from nemo_curator.core.client import RayClient

client = RayClient(
    num_cpus=64,    # Adjust based on available cores
    num_gpus=4      # Should be roughly 2x the memory of embeddings
)
client.start()

try:
    result = workflow.run()
finally:
    client.stop()

For TB-scale datasets, consider distributed GPU clusters with Ray.

ID Generator for Large-Scale Operations

For large-scale duplicate removal, persist the ID Generator for consistent document tracking:

from nemo_curator.stages.deduplication.id_generator import (
    create_id_generator_actor,
    write_id_generator_to_disk,
    kill_id_generator_actor
)

create_id_generator_actor()
id_generator_path = "semantic_id_generator.json"
write_id_generator_to_disk(id_generator_path)
kill_id_generator_actor()

# Use saved ID generator in removal workflow
removal_workflow = TextDuplicatesRemovalWorkflow(
    input_path=input_path,
    ids_to_remove_path=duplicates_path,
    output_path=output_path,
    id_generator_path=id_generator_path,
    # ... other parameters
)

The ID Generator ensures consistent IDs across workflow stages.