NVIDIA NeMo Curator Enhances Non-English Dataset Preparation For LLM Training

NVIDIA NeMo Curator Enhances Non-English Dataset Preparation for LLM Training

Data curation is critical for developing effective and fair large language models (LLMs). High-quality, diverse training data directly impacts LLM performance by addressing issues like bias, inconsistencies, and redundancy. NVIDIA has recently announced the open-source release of the NVIDIA NeMo Curator, a data curation library designed to enhance LLM training accuracy through scalable and efficient dataset preparation.

Importance of Data Curation

When training localized multilingual LLMs, particularly for low-resourced languages, web-crawled data such as OSCAR is vital. However, this data often contains noise, irrelevant content, duplicates, and formatting issues. Effective data curation is essential to mitigate these problems and ensure high-quality LLM performance. The NeMo Curator offers a customizable and modular interface that simplifies pipeline expansion and accelerates model convergence by preparing high-quality tokens.

NeMo Curator Overview

The NeMo Curator leverages GPU-accelerated data curation using Dask and RAPIDS, enabling users to mine high-quality text at scale from massive uncurated web corpora as well as custom datasets. For instance, a data curation pipeline can be constructed using the Thai Wikipedia dataset, a smaller subset of the Wikipedia dataset, which can be processed on a single GPU. Wikipedia is considered high-quality for LLM pretraining due to its accurate, well-structured content. NeMo Curator enhances this by detecting and filtering low-quality documents, ensuring only the best data is used for training.

Data Curation Pipeline Example

Using the Thai Wikipedia as an example, the data curation pipeline involves several steps:

  1. Download and extract the dataset to a JSONL file.
  2. Perform preliminary data cleaning, including language separation and Unicode text fixes.
  3. Advanced cleaning, such as GPU-accelerated exact and fuzzy deduplication, and heuristic filtering.

For the complete code sample for this tutorial, see the NVIDIA NeMo Curator GitHub repo.

Prerequisites and Setup

To use GPU-accelerated deduplication, the following hardware setup is recommended:

  • NVIDIA GPU: This tutorial uses the NVIDIA A10 24GB GPU.
  • CUDA and NVIDIA Drivers: CUDA 12.2 with Driver 535.154.05.
  • Ubuntu 22.04.
  • NVIDIA-container-toolkit version 1.14.6.

To install the NeMo Curator library, run the following commands:

git clone https://github.com/NVIDIA/NeMo-Curator.git
cd NeMo-Curator
pip install --extra-index-url https://pypi.nvidia.com "[cuda12x]"

Advanced Data Cleaning

Advanced data curation techniques such as deduplication and heuristic filtering are applied to yield better data quality. For example, the ExactDuplicates class removes identical documents using GPU-accelerated implementations from the RAPIDS cuDF library. Similarly, the FuzzyDuplicates class removes near-identical documents using the MinhashLSH algorithm, which is computationally efficient.

Heuristic Filtering

Heuristic filtering helps remove low-quality content from the dataset using simple, efficient-to-compute rules. At the time of publication, NeMo Curator provides 24 heuristics for natural languages and eight for coding languages. These filters can be applied using a YAML config file to define the filters for heuristic filtering.

Next Steps

The tutorial demonstrated how to construct a sample data curation pipeline for Thai Wikipedia data. For more information and examples, see the collection of data curation examples on GitHub. Enterprises can also request access to the NVIDIA NeMo Curator microservice, which provides streamlined performance and scalability.

Image source: Shutterstock
RECENT NEWS

Ether Surges 16% Amid Speculation Of US ETF Approval

New York, USA – Ether, the second-largest cryptocurrency by market capitalization, experienced a significant surge of ... Read more

BlackRock And The Institutional Embrace Of Bitcoin

BlackRock’s strategic shift towards becoming the world’s largest Bitcoin fund marks a pivotal moment in the financia... Read more

Robinhood Faces Regulatory Scrutiny: SEC Threatens Lawsuit Over Crypto Business

Robinhood, the prominent retail brokerage platform, finds itself in the regulatory spotlight as the Securities and Excha... Read more

Surprise Crypto Surge May Come This Week – Here Are The Top Coins To Keep An Eye On

This week’s crypto market shift has investors buzzing—find out which digital currencies could be poised for a breako... Read more

CFTC Wins $36m Victory In California Crypto Fraud Case

New York resident William Koo Ichioka agreed to pay $36 million in a CFTC case alleging cryptocurrency and forex fraud. ... Read more

Experts Predict 5000% Gains For This Solana Memecoin Set To Rival Dogecoins 2021 Surge

Discover a new memecoin on Solana, inspired by Dogecoin, with analysts predicting gains of up to 5,000%. #partnercontent Read more