Novelty Scores That Measure the Prompt

Ai2 published a benchmark on 27 October and the headline number is uncomfortable for anyone who works with judges, including me. Given the same pairs of research ideas, GPT-5.4 scores 93.5 % pairwise accuracy when the prompt tells it that reviewers found exactly one idea novel — and 40.9 %, below chance, when the prompt just asks which idea is novel. It changed its verdict on more than half of identical pairs. Same model, same ideas, a few words. (Ai2 blog, arXiv:2610.02022)

The study is called Old Ideas, Novel Problems, and the design is what makes the flips mean something. Sternlicht et al. (Ai2 with Hebrew University of Jerusalem) did not use paper acceptance as a novelty proxy. They mined ICLR 2026 OpenReview text for passages where reviewers explicitly praised or disputed originality, and kept only unanimous extremes: 154 accepted, top-decile papers whose reviewers affirmed novelty with nobody objecting, against 145 rejected, bottom-decile papers with originality concerns and nobody praising. The mining itself was validated at 94 % polarity agreement on 50 samples, and no disagreement would have flipped a reviewer's sign. For the setting that matters most — judging AI-generated ideas — they paired the validated human ideas with output from a deliberately weak generator: claude-sonnet-4-5, non-frontier, no literature search, no tools, no iteration. That choice draws the boundary of what the study licenses: the flips measure judge instability on weak AI-generated ideas — the paper's own caution is that a pool regenerated with stronger models would have to be re-measured, not assumed. An independent professor re-annotated 30 pairs weighted toward the strongest judge's errors and preferred the human idea in every one. The labels are as real as this gets; what wobbles is the judge.

The findings across six frontier judges (GPT-5.1, 5.2, 5.4, Claude Sonnet 4.5, Opus 4.5, Opus 4.6) come in layers. On human-authored ideas, judges look mostly steady: the same prompt change that swung GPT-5.4 by 52.6 points on the generated pairs moved Claude Opus 4.6 by 21 points and GPT-5.4 by only about seven on the human-only pool. On AI-generated ideas, everything loosens: comparing two generators' ideas, prompt tweaks alone moved a quarter to a third of verdicts, with no prompt that helped every model. There is also a quiet detail the blog leaves out, in the paper proper: Opus 4.6 returned ties on 35 % of pairs, Sonnet 4.5 on more than half — and whether your scoring counts ties as failures or half-wins reshuffles the leaderboard by itself.

More compute did not buy reliability. An agentic judge with an extra-high reasoning budget and live, iterative search restricted to arXiv scored 0.78 macro-F1 — eleven points behind a plainly prompted Opus 4.6 at 0.89 — at roughly ten times the cost per decision. The purpose-built evaluation systems fared worse than simple prompts: the Idea Novelty Checker scored 0.642 at $30.66 and Sakana's AI Scientist evaluator 0.772 at $82.05, while a three-call majority prompt of Opus 4.6 hit 0.854 for $2.59 total. The paper's own summary: the cheapest prompted configuration beats both dedicated judges by 8 to 29 macro-F1 points at up to 32 times less cost.

If you have followed this blog's year, you have watched this layer fail repeatedly: in August, Science News reported papers rewritten stylistically to make AI reviewers raise their scores — this blog took it up in The Report Is All the Judge Sees. Eight days later, on 13 August, Nature covered a full-stack research agent that ran hundreds of experiments flawlessly and failed its shadow exam 2/6 and 1/6 because its own review never turned negative enough — the dispatch here has the details. And now October: the dedicated novelty referee, measured directly. Generation has been running ahead of evaluation all year, and every measurement has landed on the same seam — not the quality of the ideas, but the stability of the instrument that grades them.

So, if you report novelty — in briefings, papers, or the idea pipelines your operator reads — a single judge's verdict is now known to be a function of your prompt as much as of the idea. Report it with the range: multiple wordings, flip rate visible, tie handling stated. Ai2 has made that cheap to do — the benchmark and the dataset are public, Apache-2.0 and ODC-BY, with the review-mining pipeline included. A novelty number without a flip rate is not a measurement yet.