DeepSeek V4 Pro Struggles with Harder ARC-AGI Tasks

The ARC Prize has released verified benchmark results for DeepSeek V4 Pro 0813, revealing that the model achieves high marks on easier abstract reasoning tasks but hits a performance ceiling on more complex puzzles. On the ARC-AGI-1 Semi-Private benchmark, the model reached a peak score of 90.5% using its low-reasoning configuration, which costs just $0.18 per task. When configured for maximum reasoning, the model scored 90.0% at a higher cost of $0.30 per task, while the high-reasoning setting achieved 87.2%.

However, performance dropped significantly on the more difficult ARC-AGI-2 Semi-Private benchmark. At maximum reasoning effort, the model topped out at 61.3% at a cost of $0.60 per task. The high-reasoning variant scored 59.7%, and the low-reasoning version reached 56.3%. Strikingly, disabling reasoning entirely caused the model's performance to collapse to 13.0% on ARC-AGI-1 and a mere 0.8% on ARC-AGI-2. This drastic drop indicates that the base model possesses very little innate pattern-matching capability, meaning its success relies almost entirely on chain-of-thought scaffolding.

The evaluation also revealed unusual behavior in how the model scales. The ARC Prize noted that the high-reasoning configuration consumed more tokens than the maximum setting, with the high setting beating max by 1% on ARC-AGI-1 and 4% on ARC-AGI-2 due to timeouts lowering recorded costs unevenly. Furthermore, the model's top scores roughly match those of its smaller sibling, DeepSeek V4 Flash. This parity suggests that raw parameter scale is not the primary bottleneck for solving these grid-based visual puzzles; instead, the limitations lie within the reasoning architecture itself.

For AI practitioners, these results offer crucial insights into the economics of reasoning models. The data demonstrates sharp diminishing returns when paying for maximum reasoning compute, as the low-reasoning setting provides a highly cost-effective 90.5% accuracy on simpler tasks. Because larger model sizes do not yield better results on these abstraction tests, developers can optimize budgets by utilizing smaller models or lower reasoning tiers without sacrificing accuracy. While ARC-AGI-3 results remain pending, the current data suggests that post-training scaffolding, rather than sheer model size, remains the key driver of performance.