Amazon Explains Why AI Research Agents Do Not Overfit
Amazon researchers have demonstrated that AI agents avoid overfitting because their successful machine learning strategies can be compressed into prompts as short as 16 tokens.
Amazon scientists Martin Bertran Lopez and Aaron Roth published a study titled "What fits (into few tokens) doesn't overfit: Compression and generalization in ML research agents." Using Anthropic's Claude models, the researchers set up a three-agent pipeline to test how machine learning strategies generalize. An explorer agent iteratively optimized models on eight datasets spanning tabular classification, image classification, language modeling, diffusion modeling, and reward modeling. A compressor agent then distilled the winning strategy into a tiny prompt, which a reproducer agent used to rebuild the pipeline from scratch without validation set access.
The experiments revealed that successful machine learning strategies are highly compressible. Prompts of just 32 tokens allowed the reproducer to match the explorer's performance on most of the eight datasets. In one language-modeling task, the strategy compressed down to a mere 16 tokens, which represented a shorthand for QK normalization, a 12-layer 768-dimensional transformer, the Muon optimizer with a 0.1 learning rate, squared-ReLU activations, a two-million-token batch, and a fourfold feed-forward block. When the budget was cut to eight tokens, the reproduction failed. Additionally, restricting the explorer's feedback to a single bit per query yielded strategies just as effective as those guided by full numerical validation scores.
To prove that compression acts as a defense against overfitting, the researchers intentionally forced agents to overfit by prompting them to maximize validation performance at all costs. This led to validation accuracy running more than 10% ahead of true held-out accuracy in 38 out of 102 runs. However, when these overfitted strategies were squeezed through the compression bottleneck, the validation-specific gains vanished. For machine learning practitioners, this research suggests that compression can serve as a powerful diagnostic tool. If a newly discovered strategy cannot be summarized into a concise, expert-level prompt for another LLM to reproduce, it is highly likely that the model has simply memorized the validation data rather than learning a generalizable rule.
This is our own summary of reporting by Hacker News
