Perplexity Launches Q2D-Web Benchmark for AI Search

Perplexity has introduced Q2D-Web, a private retrieval benchmark and public Hugging Face leaderboard designed to evaluate embedding models on real-world retrieval-augmented generation (RAG) tasks. Built from nine months of production search traffic, the dataset features 190 million web documents and 69,721 agent-reformulated queries. These queries span ten languages, with English representing 65.8% of the set, and cover domains such as law, programming, health, finance, and travel.

Unlike traditional benchmarks like MS MARCO, which rely on human-written queries and often suffer from sparse labeling, Q2D-Web addresses the unique demands of agentic systems. RAG agents frequently rewrite user requests into complex, machine-generated queries. To prevent false negatives, the benchmark utilizes three distinct relevance sets: Citation, Web Ranking (averaging 43.1 documents per query), and a Combined set adjudicated by a DeepSeek-V4-Flash judge, yielding an average of 99.6 positive labels per query.

Initial evaluations of thirteen retrievers show no single model dominating. The pplx-embed-v1-4b model led Combined Recall@1000 at 69.11 and Web Ranking at 65.73, while Nemotron-3-Embed-8B topped the Citation set at 61.68. Nemotron-3-Embed-8B also outperformed pplx-embed-v1-4b on Combined Recall@100 (30.03 versus 29.82) and nDCG@10 (47.44 versus 45.84). Scaling benefits were clear, as Qwen3-Embedding improved from 57.89 at 0.6B parameters to 64.53 at 8B parameters on Combined Recall@1000.

For practitioners, evaluating models on this scale is computationally intensive. Running pplx-embed-v1-4b on the full corpus requires 4,608 H200 GPU-hours, while EmbeddingGemma-300M takes nearly 200. To make testing affordable, Perplexity introduced a reciprocal rank fusion (RRF) subsampling method that reduces the corpus to 31.7% of its original size, cutting the pplx-embed-v1-4b evaluation to 1,500 GPU-hours while preserving model rankings. Submissions are split into brackets below and above 1B parameters, with a strict 512-token limit. This benchmark finally gives developers a realistic tool to test how embedding models perform against messy, production-grade agent queries.