GPT-5.6 Sol Rewrites SWE-Bench Agent Harnesses

A recent experiment evaluated whether self-improvement techniques still work on highly engineered agent frameworks by using the GPT-5.6 Sol model to optimize two SWE-bench agent harnesses. The optimizer rewrote instruction-layer prompts by analyzing past worker failures while keeping all other variables—including the worker model, tools, runtime, and budget—completely fixed. The test compared a sparse harness against a heavily engineered one to see if automated prompt adjustments could yield genuine performance gains.

The results exposed a stark contrast between validation performance and actual generalization. For the sparse harness, the validation score doubled from 3/10 to 6/10 after a single accepted edit. However, when tested against a sealed set of 24 unseen tasks, this seemingly massive improvement resulted in zero net gain, remaining flat at 15/24 solves. The updated sparse harness merely swapped one previous win for a new loss, demonstrating how validation metrics can mislead developers.

Conversely, the heavily engineered harness showed a much more modest validation improvement, rising from 3/14 to 5/14. Yet, on the sealed test of 24 tasks, it actually gained two additional solves, climbing from 13/24 to 15/24. A patch-level analysis revealed that the instruction edit altered the agent's search strategy rather than simply increasing its search depth. In fact, one of the newly solved tasks required fewer API calls than the original failed attempt, though a behavioral-contract bias that helped win three tasks also caused a regression on a fourth.

For AI practitioners, these findings suggest that the selection gate, rather than the optimizer, is the primary bottleneck for self-improving agents. Strong validation signals can easily fail to predict held-out performance. To prevent shipping counterproductive updates, developers must implement larger validation gates, paired regression checks, and rigorous cost tracking alongside accuracy metrics before deploying any automated harness modifications.