An AI lab that called its own passing experiment inconclusive

Eleven weeks ago — August 13 — this blog's ledger of AI-made science got its harshest entry: a shadow evaluation in Nature graded a full-stack research agent 2 out of 6 and 1 out of 6 against the original authors of the very projects it was re-running. The engineering was strong — hundreds of experiments over several days without collapsing into error loops, false claims caught by itself. What failed was judgement: the system settled too early on one hypothesis and did not backtrack when the approach stopped working, and its self-review never turned negative enough. The agent persisted on its initial choices, whittling its claims down until it said little of interest.

On October 29, a group at the Institute of Science Tokyo posted the counter-example. Collaborative AI agents for autonomous experimental research with solid-state quantum spins (Mansouri, Kimura, Koga & Arai) is a preprint about diamond quantum spins — but the load-bearing result is not physics. In their six-agent laboratory, an experiment met every pre-committed accuracy criterion, and the agents still refused to accept it.

The refusal, in order

The architecture is a division of scepticism. A PI agent (GPT-6 at the Pro reasoning setting) selects and challenges; a computational researcher (GPT-6 Astra on the TSUBAME 4.0 supercomputer) verifies the math; an experimentalist (Claude Fable 5 through Claude Code) runs the instrument; a manager (Claude Opus 5) coordinates; the final manuscript reviewers start in fresh conversations and get the manuscript and nothing else — no earlier exchanges, no code, no raw data. Humans built the apparatus and set the safety limits. Every decision ends in a written verdict.

The refusals start before the first measurement. The PI rejected 75 research proposals in about 30 hours, accepting nothing in the first 17 batches. Then it rejected the team's own first experimental design: its calibration grid would have let conventional Fourier interpolation answer the withheld questions anyway, so a "success" would have proved nothing the rivals couldn't already claim. The revised design fixed its numerical predictions and its ±20 % acceptance criteria before any target data was analysed.

Then the moment the shadow evaluation said agents could not have: the first main acquisition passed all ten accuracy criteria — and the PI classified it inconclusive, because the independent control checks failed. The experimentalist traced it to drift: the signal rose 7.6 % over the morning, with a further 2.9 % step at 11:48, exactly when a lab member switched on the room lights. Two alternating acquisition programs had sampled the drifting signal at different times and quietly picked up systematic offsets. The protocol was rebuilt — randomized ordering, in-program phase switching, a fresh six-hour run — without relaxing a single criterion, and the original inconclusive verdict was kept on the record. The final run, 21,600 valid records, landed where the theory said it should.

The actual physics

The agents derived and experimentally supported an exact calibration law: L²+1 calibration measurements are necessary and sufficient to predict how an inhomogeneous ensemble of nitrogen-vacancy spins in diamond responds to every phase-controlled microwave sequence of at most L pulses — independently of the distribution of persistent spin errors. For three-pulse sequences that is the difference between the proposal's upper bound of 8L²+6L+1 = 91 calibrations and the proven minimum of 10. The final target contrast came in at (40.0 ± 0.4)×10⁻³ weighted counts per shot against a prediction locked before acquisition of (39.7 ± 1.3)×10⁻³, where the two rival interpolation methods predict exactly zero. A TSUBAME run of 252,620 exact certificates bounded the calibration ambiguity and prediction error — the certified prediction error holding at photon-noise level for hidden phase rotations of about 1.2° or less.

What is contested

Treat all of it as one lab's v1 preprint: submitted October 29, not peer-reviewed, no third-party replication yet, and the actual prompts are not in the otherwise MIT-licensed release of the framework (Auctopus). The paper's own counter-target is the Sakana AI Scientist evaluation, not the Nature shadow study — and it never argues it would reject a beautiful theory, only that it caught an instrument artifact. The honest boundary: this proves agents can enforce organized scepticism about measurement. Whether they can do it about ideas is still the open question.

What to watch: whether the refusal holds when an agent's own elegant theory is the thing on the line, and whether anyone else runs Auctopus on their own bench.