⚠ working notes — what we've learnt so far · updated 12 July 2026 difficulty–response curves

Finding 9 · Ask the model twice

Round 4 ended on a door left ajar: agreement between two samples never once vouched for a wrong answer, but on SAT — where many assignments are correct — right answers disagree too, and the economics die. The synthesis family has unique answers: two programs either compute the same function on probe inputs or they don't. Round 5 walked through the door with the same rules as every round before it — pre-registration first, stopping rule included, live money only behind a gate.

The gate that finally passed

Fresh corpus (120 calls, D 1–3, committed hypotheses before it existed): flag a call when its partner sample's program disagrees on eight seeded probe inputs. Bar: recall ≥ 0.9 and false alarm ≤ 0.25, one look.

recallfalse alarmAUCescalation sharecorpus
0.920.16 0.8264% 43 right / 75 wrong

First gate of the campaign to survive its one look. The declared hazard — a correct answer flagged because its partner missed — stayed contained (16% false alarms) because the cheap rung's home turf passes at 96%.

The live race (fresh instances, real spend)

policysolved$/solvedescalated
always cheap (R1@deepinfra)6/25$0.18530
always premium (GPT-5.5)17/25$0.49520
consistency router (ask twice)17/25$0.68818
random escalation (same quota)11/25$0.676418

Three facts, in order of importance. One: the certificate never lied live — all 7 answers shipped on agreement were correct, replicating the perfect precision of round 4 and the gate. Two: the signal's marginal value is now causally clean — the control escalated exactly the same number of instances (18, quota-matched by draw, the fix for round 2's flawed control) and solved 11/25 to the router's 17/25: six extra solves are attributable to where the disagreement signal pointed, not to how much was spent. Three: the economics are mix-dependent, reported without spin — the router tied always-premium's solve rate (17/25) but paid \$0.688 per solve against premium's \$0.4952, because this instance mix was deliberately frontier-heavy: 20 of 25 puzzles sat beyond the cheap rung, so the double-probe was overhead on most of them. The pre-registered stretch bar (beat premium outright) was therefore missed; the primary bar (beat the matched control) was cleared by six solves. On a workload where the cheap rung's home turf is the majority — most real workloads — the skip-rate arithmetic flips.

Honest edges

One cheap rung (R1), one family (synthesis), one mix. The closer's own frontier caps every arm near 0.68 here — ten of eighteen escalations converted, and no router fixes the closer. The live run overshot its $25 cap to $28.67 on in-flight calls; recorded as spent. And the signal costs a second cheap call on every instance — it is a toll, not a trick.

Four rounds of no bought one conditioned yes: where answers are unique, two cheap samples grade each other — perfectly, in every dataset this campaign produced — and route six solves past a coin at equal spend. The certificate is still the router; it turns out you can sometimes mint one from agreement.