Apple's DFN5B-CLIP Achieves 84.2% ImageNet Accuracy

Apple has introduced DFN5B-CLIP-ViT-H-14-378, a vision model that achieves 84.2% zero-shot ImageNet accuracy, outperforming common open baselines such as LAION-2B and OpenAI WIT. Built with a ViT-H/14 image encoder operating at a 378-pixel input size and paired with a CLIP text encoder, the model was converted from JAX Axlearn checkpoints to PyTorch. Distributed under the Apple Machine Learning Research license, it serves as a drop-in replacement within OpenCLIP for legacy ViT-L and ViT-H backbones. Despite launching without a broad product announcement, the model has accumulated nearly 800,000 Hugging Face downloads and powers dozens of community Spaces.

The key advance lies in the model's training data, known as DFN-5B. Researchers from Apple and the University of Washington selected five billion image-text pairs from a pool of roughly 43 billion candidate pairs, which included 12.8 billion from CommonPool-12.8B alongside 30 billion public pairs. To prune the pool, the team trained a small Data Filtering Network as a dedicated scorer. Training this filter network required less than one-fiftieth of the compute budget consumed by the subsequent CLIP training run, demonstrating that automated data curation can significantly boost performance without excessive training overhead.

The paper reveals that a network's downstream classification accuracy does not correlate with its capability as a data filter; instead, lower-accuracy models trained on small, refined datasets often proved to be superior filtering tools. For developers, the resulting checkpoint offers enhanced capabilities for natural-language image retrieval, zero-shot classification, visual collection clustering, and feature extraction. However, while the architecture excels at general classification and retrieval tasks, it remains weak at distance estimation, counting objects, and identifying long-tail wildlife. Fully reproducing the curation process still requires access to a candidate dataset and the computing capacity to score billions of image-text pairs.