Epoch AI Benchmark Shows Humans Outlearn Frontier Models

Epoch AI has released human baseline results for its EBR-bench, a benchmark based on the card game Earthborne Rangers. Designed to test if systems can learn unfamiliar tasks through trial and error, the evaluation gives agents 10 playthroughs. Only the final two runs are scored, allowing models to use a notes directory to carry knowledge. Epoch evaluated frontier models including GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro using a ReAct agent harness with tools like a card lookup, map images, and text pathfinding. Models were tested at compaction thresholds of 250,000 tokens and 90 percent of their maximum context window.

To establish a baseline, Epoch recruited 15 human players, paying them 30 dollars per hour up to a 60-hour cap, with 25 percent bonuses for scoring 10 or more and another 25 percent for scoring 15 or more. Thirteen humans completed the study. While several frontier models initially beat humans, about half of the human participants ultimately outscored every AI system. The top human achieved a perfect score of 21 out of 21 by their fifth playthrough, taking 32 hours overall, while another perfect scorer took 61 hours. Humans also explored more of the game's 32 deck archetypes for their 24-card decks, whereas models repeatedly chose a single strategy.

The tactical gap was evident in the game's fatigue mechanic, where random play causes 3.5 fatigue per round (5.3 rounds of play) and expert humans maintain 0.6 fatigue (15 rounds). GPT-5.5 and Claude Opus 4.8 languished near the random baseline at 2.1 fatigue per round, yielding only 7.7 rounds. Even when provided with an expert strategy guide, the models only improved their scores by 2 to 3.5 objectives out of 21. Ad hoc scaffolding experiments using code execution, Claude Code, and Codex also failed to yield significant improvements.

For AI practitioners, these results demonstrate that current frontier models remain heavily constrained by their training distributions and cannot yet improve their strategies through experience alone. Note-taking and context-based memory are poor substitutes for actual weight updates. Developers building agentic systems cannot rely on iterative attempts to improve performance on novel tasks; instead, they must still invest in targeted fine-tuning or highly structured scaffolding to achieve expert-level execution.