NVIDIA NeMo Curator provides specialized filters for assessing and filtering code snippets and programming files. These filters help ensure that code included in your training dataset meets quality standards and doesn’t contain problematic patterns. Code filtering addresses specific challenges related to programming content, including code quality assessment, detection of non-code content mislabeled as code, identification of embedded data structures or boilerplate, language-specific filtering considerations, and token efficiency for code. These filters are particularly important when preparing datasets for code language models or programming assistants.

How It Works

Code filtering evaluates programming content based on measurable attributes that correlate with code quality and usability for model training. The filters analyze various aspects of code:

1. Structure Analysis: Examines lines of code, indentation patterns, and overall file organization
2. Comment Analysis: Measures the ratio of comments to executable code to identify well-documented code versus automatically generated or tutorial content
3. Content Verification: Ensures files actually contain code rather than data, configuration, or misclassified content
4. Language-Specific Patterns: Applies different criteria based on programming language conventions
5. Token Efficiency: Evaluates how efficiently the code can be tokenized for model training

These filters can be applied individually or in combination to create comprehensive quality assessment pipelines. Each filter typically computes a score or makes a binary decision based on configurable thresholds that can be adjusted to match specific requirements.

Usage

Here’s an example of applying code filters to a dataset:

from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.io.reader import JsonlReader
from nemo_curator.stages.text.io.writer import JsonlWriter
from nemo_curator.stages.text.filters import ScoreFilter
from nemo_curator.stages.text.filters.heuristic.code import (
    PythonCommentToCodeFilter,
    NumberOfLinesOfCodeFilter,
    AlphaFilter
)

# Create pipeline
pipeline = Pipeline(name="code_quality_filtering")

# Load your code dataset
reader = JsonlReader(
    file_paths="code_data/*.jsonl",
    fields=["content", "id"]  # Specify fields to read
)
pipeline.add_stage(reader)

# Add filter stages for code quality
pipeline.add_stage(ScoreFilter(
    filter_obj=PythonCommentToCodeFilter(
        min_comment_to_code_ratio=0.01,
        max_comment_to_code_ratio=0.8
    ),
    text_field="content",
    score_field="comment_ratio"
))
pipeline.add_stage(ScoreFilter(
    filter_obj=NumberOfLinesOfCodeFilter(min_lines=5, max_lines=1000),
    text_field="content",
    score_field="line_count"
))
pipeline.add_stage(ScoreFilter(
    filter_obj=AlphaFilter(min_alpha_ratio=0.3),
    text_field="content",
    score_field="alpha_ratio"
))

# Add output stage
writer = JsonlWriter(path="filtered_code/")
pipeline.add_stage(writer)

# Execute pipeline
results = pipeline.run()

Available Code Filters

NeMo Curator offers several specialized filters for code content:

Comment Analysis Filters

The comment-to-code ratio is an important metric for code quality. Low comment ratios may indicate poor documentation, while high comment ratios might suggest automatically generated code or tutorials. These ratios should be adjusted based on specific programming languages:

# For Python files with docstrings
python_filter = ScoreFilter(
    filter_obj=PythonCommentToCodeFilter(
        min_comment_to_code_ratio=0.05,  # At least 5% comments
        max_comment_to_code_ratio=0.7    # At most 70% comments
    ),
    text_field="content"
)

# For other languages
cpp_filter = ScoreFilter(
    filter_obj=GeneralCommentToCodeFilter(
        language="text/x-c++",  # MIME type for C++
        min_comment_to_code_ratio=0.02,
        max_comment_to_code_ratio=0.6
    ),
    text_field="content"
)

The GeneralCommentToCodeFilter supports various language MIME types:

• text/x-c++ for C++
• text/x-java for Java
• text/javascript for JavaScript
• text/x-ruby for Ruby
• text/x-csharp for C#
• text/x-c for C
• text/x-asm for Assembly

Code Structure Filters

Code structure filters help identify problematic patterns:

# Filter for reasonable line counts
line_filter = ScoreFilter(
    filter_obj=NumberOfLinesOfCodeFilter(
        min_lines=5,     # Filter out tiny snippets
        max_lines=2000   # Filter out extremely long files
    ),
    text_field="content"
)

# Filter for alphabetic content (avoid large data blobs)
alpha_filter = ScoreFilter(
    filter_obj=AlphaFilter(min_alpha_ratio=0.3),  # At least 30% alphabetic chars
    text_field="content"
)

The TokenizerFertilityFilter helps ensure code has efficient token encoding:

# Filter for token efficiency
# Note: path_to_tokenizer is required
tokenization_filter = ScoreFilter(
    filter_obj=TokenizerFertilityFilter(
        path_to_tokenizer="/path/to/code_tokenizer.model",  # Required parameter
        min_char_to_token_ratio=2.5  # Each token encodes at least 2.5 chars on average
    ),
    text_field="content"
)

This filter helps avoid content that has poor token efficiency, which can impact model training.

File Format Filters

Language-Specific Considerations

Different programming languages have different conventions and characteristics. The PerExtensionFilter applies customized filtering based on file extension:

# Apply language-specific filters
python_specific = ScoreFilter(
    filter_obj=PerExtensionFilter(
        lang="python",
        extension=".py",
        metadata_file="code_meta.csv"  # Contains language-specific thresholds
    ),
    text_field="content"
)

The metadata file can specify different thresholds for metrics like:

• Average line length
• Comment ratio
• Empty line ratio
• Alphabetic content ratio

Best Practices for Code Filtering

When filtering code datasets, consider these best practices:

1. Language-specific configurations: Adjust thresholds based on the programming language

   from nemo_curator.stages.text.filters import ScoreFilter
   from nemo_curator.stages.text.filters.heuristic.code import PythonCommentToCodeFilter, GeneralCommentToCodeFilter
   
   # Python tends to have more comments than C
   python_comment_filter = ScoreFilter(
       filter_obj=PythonCommentToCodeFilter(min_comment_to_code_ratio=0.05),
       text_field="content"
   )
   c_comment_filter = ScoreFilter(
       filter_obj=GeneralCommentToCodeFilter(language="text/x-c", min_comment_to_code_ratio=0.02),
       text_field="content"
   )

2. Preserve code structure: Ensure filters don’t inadvertently remove valid coding patterns

   from nemo_curator.stages.text.filters import ScoreFilter
   from nemo_curator.stages.text.filters.heuristic.code import GeneralCommentToCodeFilter
   
   # Some languages naturally have low comment ratios
   assembly_filter = ScoreFilter(
       filter_obj=GeneralCommentToCodeFilter(
           language="text/x-asm",
           min_comment_to_code_ratio=0.001  # Very low minimum for assembly
       ),
       text_field="content"
   )

3. Combine with language detection: Verify file extensions match content

   # First check if the content is actually Python using FastText language ID
   from nemo_curator.stages.text.filters.fasttext import FastTextLangId
   from nemo_curator.pipeline import Pipeline
   from nemo_curator.stages.text.filters import ScoreFilter
   
   # Create pipeline for Python code filtering with language detection
   pipeline = Pipeline(name="python_code_filtering")
   
   # Add language detection stage
   pipeline.add_stage(ScoreFilter(
       filter_obj=FastTextLangId(
           model_path="/path/to/lid.176.bin",  # Download from fasttext.cc
           min_langid_score=0.8
       ),
       text_field="content",
       score_field="language"
   ))
   
   # Then apply Python-specific filters
   pipeline.add_stage(ScoreFilter(
       filter_obj=PythonCommentToCodeFilter(),
       text_field="content"
   ))

4. Avoid over-filtering: Track rejection rates and adjust thresholds as needed

   # Track filter statistics by running individual filters and measuring results
   from nemo_curator.stages.text.io.reader import JsonlReader
   
   # Load dataset for testing
   reader = JsonlReader(file_paths="test_data/*.jsonl")
   
   # Test individual filters to measure rejection rates
   filters_to_test = {
       "python_comment": PythonCommentToCodeFilter(),
       "line_count": NumberOfLinesOfCodeFilter(min_lines=5, max_lines=1000),
       "alpha_content": AlphaFilter(min_alpha_ratio=0.3)
   }
   
   # Note: Actual statistics collection would require running the pipeline
   # and analyzing the results to determine optimal thresholds

Use Cases

from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.filters import ScoreFilter
from nemo_curator.stages.text.filters.heuristic.code import NumberOfLinesOfCodeFilter, XMLHeaderFilter, GeneralCommentToCodeFilter

# Create pipeline to filter non-functional code snippets
pipeline = Pipeline(name="code_cleaning")

# Remove extremely short files
pipeline.add_stage(ScoreFilter(
    filter_obj=NumberOfLinesOfCodeFilter(min_lines=3),
    text_field="content"
))

# Remove files with XML preamble (misidentified as code)
pipeline.add_stage(ScoreFilter(
    filter_obj=XMLHeaderFilter(),
    text_field="content"
))

# Ensure reasonable comment-to-code ratio
pipeline.add_stage(ScoreFilter(
    filter_obj=GeneralCommentToCodeFilter(language="text/x-c++"),
    text_field="content"
))

from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.filters import ScoreFilter
from nemo_curator.stages.text.filters.heuristic.code import AlphaFilter, TokenizerFertilityFilter, HTMLBoilerplateFilter

# Create pipeline for training data preparation
pipeline = Pipeline(name="training_data_prep")

# Ensure enough alphabetic content (not just symbols or data)
pipeline.add_stage(ScoreFilter(
    filter_obj=AlphaFilter(min_alpha_ratio=0.3),
    text_field="content"
))

# Check token efficiency
pipeline.add_stage(ScoreFilter(
    filter_obj=TokenizerFertilityFilter(path_to_tokenizer="tokenizer.model"),
    text_field="content"
))

# Remove HTML with mostly boilerplate
pipeline.add_stage(ScoreFilter(
    filter_obj=HTMLBoilerplateFilter(min_lang_content_ratio=0.3),
    text_field="content"
))

By applying these specialized code filters, you can improve the quality of code in your training datasets, leading to better model performance for code-related tasks.