The journal: how it actually went

The brief lands: measure where reasoning models break, publish everything, one commit, $300 cap. Probes find the gateway speaks two APIs; a pilot roster of eight models is chosen. The plan prices the textbook protocol; the pilot will price reality.

Pilot runs into a declared router-core outage mid-flight; the resumable store shrugs. Gate 1 arithmetic: the textbook design would cost ~$530 — roster rescoped (Haiku promoted to core; Qwen and GLM to budget extras). MiniMax passes everything on the standard dial and is moved to an extended one.

The throughput wars: one provider serves ~3 calls an hour; pins are switched (novita → parasail/cerebras/baseten); 1,446 calls die at a mysterious 4,096-token wall on baseten and are purged. At $97.56 the gateway account runs dry mid-campaign. Everything pauses, resumably.

Credits topped up; sweeps resume; the session hosting the runs crashes twice and the jobs are re-launched detached, out of its reach. Haiku and o4-mini finish their full designs including fresh-instance retests.

The check family embarrasses everyone: 480/480 on arithmetic to 42 steps; the grid is extended twice before Haiku finally cracks past ~300 steps. The adaptive probe proves itself live: Haiku's frontier in 20 calls for $1.03.

Data cutoff. Full statistics (B=2000), four figures, paper compiled with the abstract reading its numbers from disk; site and repo published as one commit: split-half r = 0.99, 81% within-instance variance at the frontier, effort peak lagging it, 6.75× adaptive efficiency.

Late entries: GPT-5.5 runs off the end of both tracks (censored > 7.2). Fable 5 looks broken — until a differential probe through a second router convicts the gateway: it silently drops max_completion_tokens and replies default to 4,096 tokens. Baseten exonerated (same bug). Fable re-measured fairly: 6.34 ± 0.21. A validation fleet re-checks every row: four agree; gpt-oss-20b has drifted a full dial unit on the same endpoint in twelve hours.

The correction that earned its keep: Fable's 6.34 replicates perfectly on fresh puzzles (6.33) and is then falsified by direct test — 10/10 at α = 7.2; both runs had extrapolated past the track edge. Corrected verdict: censored, joint top with GPT-5.5. Precision is not truth.

The n=50 separator: GPT-5.5 13/21 with zero truncations; Fable 11/40, out-thinking even 64k budgets (but more accurate when it finishes); MiniMax 0/17. Fixing the parameter bug had silently re-run MiniMax's measurement at an enforced cap: frontier 6.58 uncapped → ≈3.6 at 32k. Budget elasticity becomes finding 3's second half: a frontier is a curve against the thinking budget.

The vendor credits $20 and reports the bug fixed; the same probe says otherwise: still 4,096, still empty. Raw data moves onto the page itself. The trace ledger is computed: billed thinking vs delivered text, by model and route — OpenAI withholds everywhere, open-weights gpt-oss stripped by every serving stack, Claude's trace depends entirely on the door. Where traces flow, loops mark the cliff better than length.

The practical turn: can the card drive a router? Offline replay over shared instances says yes, with two folklore-violating rules — retries beat escalation, and premium ≠ stronger. A LiteLLM integration ships; a live validation runs on fresh puzzles; MiniMax's n=50 failure hunt continues. The document you are reading reorganised itself twice in the meantime.

Both verdicts land. MiniMax's n=50 hunt dies of timeout — through a ten-minute read window nothing ships — then a thirty-minute window delivers the answer: 3/12 at the easy end, frontier at or below α 3. And the router meets 25 never-seen puzzles for real: cheap-always solves 12/25, premium-always 9/25 (at 19× the price per solve), the form-guide router 24/25 for $4.45 — escalating only where the card says the cheap horse falls. An accidental duplicate run re-measured the single-model legs at 10/25 and 6/25: the mist, live. Total cost of knowing all this: about $332 across two gateways — $306 on the recorded ledgers, the rest off-book: duplicate legs from a session collision and calls the meters never saw complete.

The vendor's second attempt: another $20 credit, another claimed fix — and this one is real. The differential probe that convicted the gateway twice now clears it: 7,217 and 10,855 billed tokens through the once-immovable 4,096 wall, natural stops, delivered answers, and two scored passes on fresh instances at α 5.0 and 5.6. Fable 5 is measurable through the front door at last. The moral survives the fix: don't trust a gateway's word about its own plumbing — the probe costs $0.94.

Round 2 opens on the question finding 7 left loaded: can the trace replace the verifier? Hypotheses pre-registered before the data existed (primary hedge_tail, gate AUC ≥ 0.65 on 2 of 3 models), a purpose-built batch of frontier calls launched — and the first save of the night lands within the hour: the second gateway delivers reasoning in a separate field the runner was silently discarding. $7 of gate calls had banked answers with no traces. One fix, every lane bounced, and from then on every call carries its evidence.

The gate wobbles — and the wobble is the story: R1's hedging crosses 0.65, dips to 0.648 at 440 pairs, recovers to 0.68; at the interim 2-of-3 read the gate is called passed and the live arm funded. (Hold that thought until dawn.) MiniMax's backtracking firms to 0.81 on a fresh-seed top-up. Two textures, not one signal — R1 hedges when wrong, MiniMax rambles. Meanwhile the transfer family earns its keep the hard way: design one (transcribe forty rules into code) is flat — R1 58/60; design two (induce hidden rules from eight examples) fails all its hidden tests while fitting every example — ambiguity, not difficulty; design three (deterministic rules, twelve examples) finally slopes, and paired pilots certify a routable band: the cheap rung 0/11 at depths 2–5, the closer 7/8 in exactly that band. GPT-5.5's real prices are caught wrong in the fleet file ($1.25 vs $5.50 per million in) before they could poison a single $/solve figure.

The live race answers everything, mostly with a no. GPT-5.5 saturates SAT 20/20 — the pre-registered bar (beat both single arms) is unreachable against perfection. On synthesis the closer is honestly fallible (13/20) and the bar is live — and the trace-judged router still misses it: 10/20, at a worse price per solve than always-buying-premium, because the judge fires precisely but rarely (live recall ≈ 0.21) and, at the closer's own frontier, fires on the closer too — expensive retries that convert nothing. The kicker, reported per the house rules: the signal-free random-escalation control beat the judge on both families. Then the last witness reports: Qwen's completed lane is flat — nothing in its trace separates wrong from right — and R1's hedging settles at 0.615, under the bar it danced around all night. The evening's 2-of-3 gate call, on complete data, walks back to 1-of-3: the pre-registration fixed the bar but not the stopping rule, and sequential peeking did what it always does. Offline whisper, live silence, and a methods bruise — all three published. The certificate is not an implementation detail of the router; it is the router. Round 2's whole interrogation: about $75.

Round 3, rescoped with the owner over coffee: the stronger judge. Per-model trained combos over sixteen trace features, selection and thresholds locked on the exploratory corpus only, and the pre-registration finally carries a stopping rule — one look at the frozen confirmatory set, ever. Training flatters as training does: R1 cross-validates at 0.81, MiniMax and Qwen post perfect 1.0s on eleven wrongs apiece (flagged as overfit-smell before the gate ran, in writing). The single permitted look: zero of three clear. R1's combo collapses to 0.56 — it had learnt the corpus, not the model; Qwen's perfection was noise; MiniMax posts a genuinely excellent 0.93 at 92% recall and still clips the false-alarm bar. No live dollar spent. Two rounds, two designs, one verdict: the trace cannot grade itself into a router. The door gets a trained-judge stamp on its way shut.

Round 4, the last signal standing: ask the model twice. Finding 1's mist says near-frontier answers are coin flips, and coin flips should disagree — so sample the cheap rung twice and escalate on disagreement. Pre-registered (hazard included, in writing: satisfiable SAT admits many right answers), one look at the frozen corpus: recall a perfect 1.000 on all three models — every silent wrong disagreed — and P(correct given agreement) also a perfect 1.000: exact agreement never shipped a wrong answer. A flawless certificate. That fires on 2–20% of calls, because right answers pick different satisfying assignments too: false alarms 76–94%, escalate-everything economics, gate failed 0-of-3 on the arm the hazard named. Three rounds, three pre-registered negatives, three different reasons. The campaign's verdict stands at a sentence: the certificate is not an implementation detail of the router — it is the router. Spend for rounds 3 and 4 combined: $0.

Round 5 walks through round 4's ajar door: on the synthesis family answers are unique, so agreement means two programs computing the same function on probe inputs — multiplicity can't tax it. A fresh 120-call gate corpus (pre-registered, stopping rule included, hazard named: your correct answer gets flagged when your partner misses) delivers the campaign's first gate PASS on its one look: recall 0.92, false alarm 0.163, AUC 0.822. The live race runs into the small hours: the router ships 7 answers on agreement — all 7 correct, the certificate still has never lied — and solves 17/25, six clear of a control that escalated the exact same number of instances by coin flip (round 2's control flaw, fixed and avenged). The spin-free ledger: it ties always-premium's solve rate but pays $0.69 a solve to premium's $0.50, because this mix was deliberately frontier-heavy and the double-probe is a toll on every instance. Four rounds of no, one conditioned yes: where answers are unique, two cheap samples grade each other. Finding 9 goes up before dawn.

The confidence programme becomes a control programme. The question: instead of waiting for a trace score, use persistent backtracking, hedging, give-up, and repetition to launch backup models while the primary is still thinking. The distinction from a model council is fixed in writing: a council waits to combine completed answers; the speculative controller races candidates, accepts only an externally verified winner, cancels losers, and invokes a council judge only as an exception. The implementation expands to all four registered arms, deterministic ordering, durable result validation, isolated Git worktrees, and explicit unknown usage for cancelled calls. Then the zero-spend gate answers with another useful no: 84 of 89 visible traces trigger, including every failure but also 34 of 39 correct calls. At 87.2% false hedges the trace controller is an always-on council wearing a sensor; a matched-rate timer also catches every failure. Old traces have no chunk timestamps, so warning time is censored. The false-hedge bar already fails; the paid branch stops at $0.