LlamaIndex Benchmark Finds GPT-5 Fails on Wide Schemas

LlamaIndex and Kaggle have introduced ExtractBench, a public evaluation tool designed to test how artificial intelligence systems extract structured JSON data from complex business documents. In initial testing across 14 frontier systems, the benchmark exposed a severe vulnerability in top-tier models. When faced with a wide, 369-field financial reporting schema, every tested model—including OpenAI's GPT-5 and GPT-5.2, Google's Gemini-3 Flash and Pro, and Anthropic's Claude 4.5 Opus and Sonnet—collapsed to a zero percent success rate for valid outputs.

The open-source dataset contains 4,869 pages from 370 real and synthetic enterprise documents, representing 67 distinct document types across eight business domains. Unlike evaluations that rely on subjective LLM judges, ExtractBench uses deterministic scoring to measure completeness on repeated structures, null discipline for missing fields, and source grounding. Grounding is tracked via word-level and page-level F1 scores, verifying whether an extracted value's predicted bounding box aligns with the actual document page.

The benchmark revealed distinct trade-offs among different AI architectures. Standard vision-language models performed adequately on short files but frequently truncated long tables and record lists. Coding agents maintained higher accuracy on these complex tasks but operated at a significantly higher financial cost. Notably, LlamaIndex's own LlamaExtract Agentic Plus outperformed the competition across all three primary metrics, delivering accuracy comparable to expensive coding agents at a fraction of their operating cost.

For enterprise developers building automated workflows in finance, healthcare, or logistics, these findings show that standard demos do not reflect production realities. ExtractBench provides teams with a standardized command-line interface to test their own extractors, offering clear visibility into accuracy, grounding, and per-page costs. This allows practitioners to audit whether an agent can justify its decisions to human regulators before deploying it without a human in the loop.