Finding 8 · No referee: the trace cannot route itself

Finding 7's router solved 24/25 because SAT hands you a free referee — every answer is checkable. Real tasks aren't. Round 2 asked the obvious next question, with money on it: can the model's own reasoning trace replace the verifier? Pre-registered gates, fresh spend, a second task family, and a live four-arm race. Short version: one model's trace whispers, one mumbles, one is silent — and none of it routes.

The offline gate (pre-registered, confirmatory)

Hypotheses declared before the purpose-built batch ran: PRIMARY hedge_tail (hedging density, final 15% + answer region), SECONDARY bt_tail (backtrack density, last 30%). Labels are silent wrongs only — truncations are loud and free to catch. Pairs form within a difficulty level on one route, so difficulty itself cannot do the predicting. Gate: AUC ≥ 0.65 on ≥ 2 of 3 traced models.

Two lessons in one table. First, the textures: MiniMax backtracks and rambles when wrong (its hedging is noise; token count is its loudest tell — uncapped, wrong answers run long), R1 hedges, faintly (its backtracking is noise), and Qwen gives nothing away — every feature within a whisker of a coin flip on four hundred pairs. There is no universal tell; a trace judge is one more per-model calibration. Second, the confession: mid-evening, with the batch half landed, the table read 2-of-3 over the bar (R1's hedging then at 0.68) and we called the gate passed and funded the live arm. The completed set walked R1 back under the line — the final verdict is one model of three. Our pre-registration named the hypotheses and the bar but not the stopping rule, and sequential peeking did what it always does. Both the interim call and the walk-back are preserved in the devlog; the live results below should be read as funded by a gate that, on complete data, fails.

The second family, third attempt

Transfer needed a task with hidden ground truth. Design one showed the model rule specs to transcribe into code: flat — R1 went 58/60 out to forty rules; transcription is not difficulty. Design two hid the rules behind eight examples but kept two conditional rules: R1 fit all eight examples at depth 2 and still failed the hidden tests — ambiguity, not difficulty. Design three (deterministic rules only, twelve examples) produced the real thing, confirmed by paired pilots: R1 75% at depth 1 and 0/11 at depths 2–5; GPT-5.5 solves 7/8 in exactly that band and dies at 6. A routable gap, certified before the live run spent a dollar.

The live race (fresh instances, real spend)

Four arms, both families, ground truth never visible to routing. The judge: cheap rung's trace scored by hedge_tail at the Youden-optimal threshold from the confirmatory set; judged failure triggers a retry, then escalation. The control escalates at random, rate-matched to the judged arm's realized rate — the judge's marginal value over spending the same money blindly.

The pre-registered bar — beat both single-model arms on both families — was missed, and the random control beat the judge on both. Three honest footnotes. One: GPT-5.5 saturated SAT (20/20), which makes that family's bar unreachable — you cannot out-solve perfection, only undercut it. Two: the SAT control got lucky — its RNG realized 35% escalation against the judge's 15%, so it is not a clean rate-match there; on synthesis the rates nearly matched (45% vs 40%) and the control still won on both solve rate and cost. Three: the judge's escalations were precise — on SAT every one converted — it simply fired far too rarely (live recall ~0.21), and at the closer's own frontier it fired on the closer too, buying expensive retries that converted nothing ($0.72/solve on synthesis: worse than just buying premium).

What the threshold could never buy

Was Youden the mistake? Routing wants recall — a false escalation costs $0.23, a missed wrong costs a solve. Sweeping the threshold over the confirmatory traces:

deployed threshold 1.29 (Youden J); routing wants recall, not balance — but even threshold 0 only converges to premium-plus-overhead: a 0.65–0.68 AUC judge trades solves for dollars along a Pareto line and never dominates a saturated closer.

Round 3, same morning: the stronger judge ($0)

The obvious objection to everything above: one feature at a balance-tuned threshold is a weak judge. So we built the strong one — per-model trained combos over sixteen trace features, selected by instance-grouped cross-validation on the exploratory corpus only, recall-oriented threshold frozen on training data, and this time the pre-registration included the stopping rule round 2 lacked: one evaluation on the frozen confirmatory set, no interim looks. Gate: AUC ≥ 0.75 and recall ≥ 0.5 at false-alarm ≤ 0.25, on two of three models.

Zero of three clear; no live money was spent. The autopsy writes the epitaph: R1's cross-validated 0.81 evaporated to 0.56 on the frozen set (the combo had learnt the training corpus, not the model); Qwen's perfect-looking CV was noise on eleven training wrongs, exactly as flagged before the gate ran; and MiniMax — whose trace genuinely does tell, in every analysis we have run — posted 0.93 AUC at 92% recall and still missed the false-alarm arm of the bar. One talkative model out of three cannot crew a fleet, and the fleet's cheap rung was R1.

Round 4, same day: ask the model twice ($0)

One signal source left standing, and it is the one finding 1 itself predicts: the mist. If near-frontier answers are coin flips (81% within-instance variance), two samples of the same model should disagree exactly where the answer cannot be trusted. Pre-registered before the number existed (same bar as round 3, one look at the frozen k=6 corpus, hazard declared in writing: satisfiable SAT admits many correct assignments, so right answers may disagree too), then computed once:

Zero of three clear — and it is the best negative of the campaign. Recall is a perfect 1.000 on all three models: every silently wrong answer disagreed with its partners. P(correct given agreement) is also a perfect 1.000 on all three: exact agreement never once shipped a wrong answer. Self-consistency is a flawless certificate — that almost never certifies. The declared hazard ate the economics whole: correct answers pick different satisfying assignments, so false alarms run 76–94% and the router degenerates to escalate-everything. The signal is real, universal across our fleet, and priced out by answer multiplicity on exactly this task family. On families with unique answers the same arithmetic could land very differently — that is the one door this page leaves ajar, stated and unspent.

Finding 7 said bring a verifier; rounds 2, 3, and 4 priced every alternative we could buy — a simple trace judge with live money, a trained judge behind a locked gate, and agreement between independent samples. Three designs, three pre-registered failures, each for a different articulated reason: the simple judge under-fires, the trained judge learns the corpus instead of the model, and agreement is a perfect certificate that almost never certifies. The certificate is not an implementation detail of the router. It is the router — on families where answers are many. Where the answer is unique, the door round 4 left ajar swings open: finding 9.