A paper submitted to arXiv on 2026-07-14 rebuilds the evaluation protocol for automatic agent-harness evolution under matched budgets and held-out tasks, and reports that on Terminal-Bench 2.1 the evolved harnesses do not consistently beat plain test-time scaling and barely transfer.
Why it matters
The harness is the behavioral layer made concrete: the authors define it as “the prompts, tools, memory, verification routines, and control logic through which a model observes tasks and acts.” A field that reports gains from evolving that layer, while searching and scoring on the same public benchmark, cannot distinguish better behavioral design from more attempts at the same tasks. This paper is a measurement-integrity result about the exact artifact this site cares about.
Builder read
The setup separates two things existing protocols conflate: search budget and design quality. Harness evolution is compared against parallel sampling and sequential refinement “under matched feedback and inference budgets,” and the evolved harness is then tested on held-out tasks. Without unit-test feedback, on Terminal-Bench 2.1 across Claude Opus 4.6, GPT-5.4, and GPT-5.4 mini, average pass@1 was 68.2 for the initial harness, 72.3 for parallel sampling, 69.3 for sequential refinement, and 67.4 for harness evolution, which the paper notes “fails to outperform the direct sampling baseline on average,” dropping GPT-5.4 from 75.3 to 69.7. With unit tests available, every method improved, but harness evolution still trailed: 75.8 average pass@1 against parallel sampling’s 86.0, and 86.2 pass@5 against sequential refinement’s 91.8. The authors read that gap as diagnostic: “If harness revision genuinely produced better harnesses, we would expect the improvement to be reflected in pass@1.”
The generalization test is the sharpest number. Splitting the benchmark into 45 training, 10 validation, and 34 held-out test tasks, the best evolved harness gained 1.2 points on Claude Opus 4.6, 0.0 on GPT-5.4, and 0.6 on average, which the paper describes as “limited generalization and a strong tendency to overfit the search set.” One qualitative finding is worth its own note for anyone writing agent scaffolds: the meta agent starts at the prompt layer “adding behavioral rules to the system prompt and long-term memory,” and when “such advisory text plateaus, the meta agent escalates to runtime enforcement through middleware,” including turn-budget trackers and “finalization gates that block completion when deliverables are missing or unverified.” An automated designer, optimizing freely, converges on moving rules out of advisory prose and into enforcement.
Exec read
Two things to take from this. First, agent performance claims that come from tuning the scaffold rather than the model deserve the same scrutiny as any benchmark number: ask whether the tuning set and the scoring set were the same tasks, and whether the comparison spent equal compute on simply trying again. On this evidence, “we evolved our harness” and “we sampled more” can look identical from the outside, and the cheaper one often wins.
Second, the tempting shortcut is not free. The paper does not say harness design is worthless; it says these automated methods, on this benchmark, produce “task-specific shortcuts rather than genuinely better harness design principles.” That is a familiar failure shape: a system optimized against the measurement rather than the goal. If a vendor demo shows a self-improving agent harness, the question to ask is what it scores on tasks it never saw.
Caveats
This is a v1 arXiv preprint, not peer-reviewed. Findings rest on a single benchmark (Terminal-Bench 2.1) and three models, with the generalization test run on a 34-task held-out split, so absolute differences of a point or two carry real uncertainty. The harness-evolution methods are the authors’ controlled reimplementations of published approaches under their own budget protocol, and the original authors may contest the parameterization. The paper’s scope is automatic harness evolution: it does not test human harness engineering, which it cites as being able to “substantially affect agent performance even with a fixed model.”
Source
Primary: Rethinking the Evaluation of Harness Evolution for Agents, Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi, Teng Xiao (arXiv:2607.12227v1 [cs.AI], submitted 2026-07-14). Numbers are from Tables 1, 2, and 3; the harness-edit observations are from Section 5.1.