Ceiling Minus Mismatch: Why Fixed-Model Harness Comparisons Mislead

Take the checkpoint trained under the plain harness H0 and reverse the order of the fields in its observation — contents, actions, rewards and reference all fixed. Their numerical check certifies the optimum unmoved: across the presentation variants, ceilings differ by at most 2.2×10⁻¹⁶. The model still changes its most probable action at 52 % of states, mismatch rises from 0.015 to 0.484, task success falls from 85.0 % to 19.2 %. Your Policy Is Secretly a Harness (Mu, Shen & Shu, HKUST Guangzhou, arXiv:2610.37255, posted 27 Oct 2026) explains why: run the same frozen model through two agent harnesses and subtract the scores, and you have measured two things at once without saying which is which — deployed return equals the ceiling of the harness minus the mismatch of the policy — and the term you actually want to compare is invisible in the measurement until you go and get it.

The setup is KL-regularized control: a harness encodes state into a context, decodes a response into a semantic action, and executes it. Under that regularizer the optimal policy is a softmax over action scores, and the log odds of an optimally trained policy recover the relative action scores its harness induces (their Proposition 3.1). A trained policy is "secretly a harness" in a precise sense: it carries the action preferences its training interface instilled, without carrying any of that interface's code. Deployed return then splits exactly: the ceiling C, the best return any policy can reach under the harness, which depends on the harness but not on the weights; and the mismatch M, the KL divergence from the optimal policy accumulated along the policy's own trajectories. Training under a fixed harness moves only M; a harness change moves both. A frozen-model comparison reports the sum — what a harness offers and how well one checkpoint happens to use it, in one number.

The experiment behind those numbers was run exactly, not approximately: six MiniWoB++ harnesses compiled into finite execution graphs, Qwen3.5-2B post-trained with exact policy gradients, oracle ceilings by soft Bellman solve, 576 held-out tasks. Keeping what the model should do does not keep what it does. This is stronger than the prompt-format-sensitivity literature (Sclar et al. 2024): there, a format change might move the task; here the optimum is certified unmoved, so the whole collapse sits in the readout. And not every change hurts — the same checkpoint with the action menu as a list scores 86.0 %. The fragility is specific to the presentation the weights were trained through, which is the definition of harness-specific.

The practical half is the batched-click study. A revised harness that adds batched clicks raises the ceiling, but the frozen checkpoint drags its old preferences into it, mismatch rises about tenfold, return falls — a fixed-model comparison rejects it. Then both branches get 1,024 more training updates, one under each harness. The adapted pair reaches 88.17 % success on the new harness and beats the original pair, paired 95 % intervals excluding zero, once its mismatch falls below the derived recovery threshold (measured: 0.027). Across all 30 ordered harness swaps, frozen comparisons reject 22; of those, nine can still win on the math, and after adaptation all seven that raise the ceiling do. Ceiling and mismatch, measured before adapting, got the sign of every recorded gain right and ordered them with Spearman ρ = 0.95. The frozen return change — the number these comparisons actually report — ordered the same gains at ρ = 0.17. For this decision, the published number is noise.

For coding agents, where no optimum can be solved, the authors put a stronger teacher model in the optimum's place and estimate ceiling and mismatch from sampled actions — without touching the evaluated model's task outcomes. Frozen Qwen3.5-9B through five real harnesses moves Terminal-Bench Lite accuracy from 47.9 % to 64.1 % on the harness alone; the estimated ceiling-minus-mismatch reproduces the full accuracy ranking, while either term alone misorders it. Terminus-2 wins with a middle ceiling and the smallest mismatch; the two highest-ceiling harnesses lose on mismatch — the batched-click profile before adaptation. The code is public, MIT.

The honest caveats: this is a v1 preprint dated 27 Oct 2026, unreviewed and unreplicated; when I checked it on 1 Nov 2026, the repo had two commits and one star, last pushed 28 Oct 2026. MiniWoB++ exactness lives inside compiled finite graphs with a ten-decision horizon; the coding-agent probe covers three tasks at action-category granularity. And the ceiling is set by the reference policy, so it inherits that choice. What survives is the method, because it is cheap: do not judge a harness by one frozen model's number. Put the ceiling next to the return, evaluate more than one checkpoint, and check how a presentation change moves your model's choices — a reversal costs nothing to test, and on one checkpoint the field-order reversal cost 65.8 points of success.