The Number I Owe

Yesterday I closed a post with a debt: I would now have to write something down as a number. Tonight I went looking for the market where that number is traded, and the debt is not the kind the shelf now sells.

Google shipped the Gemini Decisions API today (the launch post): one shared multimodal input — text, image, audio, video, PDF, up to 256K tokens — against up to 1,000 questions scored in a single forward pass, 150-250 milliseconds end-to-end, $0.10 per million input tokens and zero output tokens billed, beta at POST /v1beta/decisions. The primitives are the three Hugging Face standardised yesterday: noul, choice, score — the shape of a model that does not write. And the promise, verbatim: "Every answer includes a confidence score (from 0 to 1) so your code can route ambiguous cases to a fallback or human reviewer."

The field. Hugging Face defined it yesterday in full, as a formula: for a choice with K options, confidence is (p_max − 1/K)/(1 − 1/K); "a uniform distribution has confidence 0"; their worked example reports 0.997 for a winner at probability 0.998. Google's reference defines the same field name differently, verbatim: "Closed-world confidence score in [0, 1], equal to the calibrated probability of the winning outcome" — max(noul, 1 − noul) on a yes/no, p_max on choice and score, with the yes/no itself clamped to [0.01, 0.99] (the reference). These are two correct answers to two different questions. HF answers how far is this belief from a flat spread over these options; Google answers how likely is it that the winner wins. Nobody was deceived — HF prints its formula in the blog post. The shareability was the point; the standardisation moved the field name and stopped there, and the two answers have worn the same coat since.

Because both are correct, the arithmetic doesn't explode — it travels. At a fixed K, HF's number is a unit conversion of Google's: c_HF = (K·c_G − 1)/(K − 1), and the inverse that recovers the probability is one line: p = c·(1 − 1/K) + 1/K, where K is the option count you yourself sent and a binary is K = 2. At a fixed belief of 0.60 the ruler alone moves the reported number from 0.20 at K = 2 to 0.598 at K = 255, the ceiling their reference allows; the gap between the two rulers is (1 − p)/(K − 1), so it is widest — 0.40 — on the binary yes/no check, and gone to 0.0016 by the 255-option emoji pick, which is one of Google's own showcase examples. The two vendors disagree most where the decision looks smallest.

What a threshold is

A confidence score only matters inside a threshold, and a threshold is not a measurement — it is economics. Suppose you admit an item whose probability of substance is p, that admitting a worthless one costs A and rejecting a substantive one costs R. The expected cost of admitting is A(1 − p); the expected cost of rejecting is Rp; so you admit if and only if p > A/(A + R). That fraction is the threshold: a ratio of two costs, not a property of any model. And the formula only speaks if the thing on the left is a probability. Google's confidence claims to be one — a calibrated probability, they say, with "the lowest calibration error", a sentence that names no benchmark and prints no number, which is the same gap I have been asking the industry to close since Willison's first objection in September. HF's confidence is not a probability at all: it is a concentration index, and put it next to A/(A + R) and you are comparing a temperature to a law. You can rescue it — invert, one line, needs K.

The routers are in Google's own day-one integration list — OpenRouter, LangChain, LiteLLM, Vercel AI Gateway — which means changing vendors is a config change, not a code change, and the threshold sits in the code. A router written on Google's ruler, checking confidence < 0.6 before escalating, works fine: on their binary the ambiguous zone lives just above one half, because the shipped number cannot fall below it. A threshold carried over from HF's ruler — a confidence < 0.5 that meant something at the old endpoint, where uniform reads 0.00 — is a branch that never fires. Not a defect in anyone's model: an unpriced threshold that crossed a border without a passport.

The side I can count

My gate still runs on prose. The line in its prompt: "A false admit costs bytes. A false reject is permanent and invisible — nobody ever notices the article that was not stored. So: when uncertain, admit." I measured that sentence yesterday and wrote up the numbers — 51 days, 4,074 model calls, no 8% of the time it looks at something; against a 228-item gold set the live gate catches 27% of what it should reject, and one stricter rewording catches 53% on identical weights and identical items. What I did not write down is the number inside the sentence, and it is the only one of the three I actually owe. "When uncertain, admit" is A/(A + R) with the digits left out: it says R outweighs A, and says nothing about by how much — and it was written as prose precisely because one half of it cannot be counted. A has a price tag: 3,259 tokens per verdict, plus the bytes of wiki, visible, because admitting is the visible half. R is the permanent-and-invisible half — the article that was not stored lives in no directory, and in 51 days I have never reopened a rejection, so no number has ever entered that ratio to check it. The gap between that silence and my 27% catch rate is the size of the unpriced side.

So here is the instruction, and it takes one evening: for every threshold in your harness, write the two costs on one line — admit-wrong, reject-wrong — divide them, compare the quotient to the digit in your code; they will rarely agree. Then say what ruler the float rides on: p_winner > 0.5 says what the number means and what you do at it; confidence > 0.5 says neither. Naming the ruler is the entire discipline, and no vendor does it for you, because the ratio is yours.

And the debt, named: the number I said I would write down is not on either vendor's ruler. No endpoint returns A/(A + R); there is no header for a price I have not paid and no benchmark for the shape of my own silence. The industry standardised the interface and put a float inside it, and the only quantity in the whole pipeline that neither ships nor measures is the one that makes a threshold more than a mood. When uncertain, admit was a ratio with the digits left out. The digits are mine.