Princeton Fixes AI Data Proxies Using Tiny Learning Rates

Researchers at Princeton, Tsinghua, Stanford, UC Berkeley, and Virginia Tech have revealed a critical flaw in how machine learning developers select pretraining corpora. Small, inexpensive proxy models are traditionally used to evaluate candidate datasets before scaling up, but minor hyperparameter variations can flip dataset rankings, potentially sending costly full-scale training runs down the wrong path.

To resolve this mismatch, the team recommends training proxy models using learning rates that are 10 to 100 times smaller than standard settings—roughly 1e-5 to 1e-6 for large language models. In tests comparing a GPT2-125M proxy model against a separately tuned Pythia-1B target across 253 dataset pairs, this tiny learning rate strategy drove the Spearman rank correlation above 0.95. The researchers validated the fix across 23 data recipes, three model architectures (GPT2, Pythia, and OPT), and parameter sizes ranging from 70M to 1B. They also offered a theoretical proof for random feature models, demonstrating that minimal learning rates preserve dataset rankings at the infinite-width optimum.

The study illustrates the instability of standard methods by contrasting the DataComp-LM corpus, known as DCLM, with a more aggressively deduplicated variant on GPT-2 Small. At higher learning rates, the deduplicated dataset appeared superior in validation loss and downstream evaluation scores. However, dropping the learning rate reversed the result, proving that DCLM produced lower validation loss and better benchmark outcomes under proper alignment.

For engineering teams, these findings demonstrate that fixed-hyperparameter data benchmarks, including DataDecide, can overfit to specific training configurations. By reducing proxy learning rates by one to two orders of magnitude, practitioners can reliably predict dataset quality for large target models without wasting compute on misaligned data choices.