Everything on this site condenses to a practical question: given a request, which model should answer it? We turned the card into a router and replayed it against our own recorded data — hundreds of real attempts on shared puzzles, with real per-call costs.
First, the idea that died (reported per the house rules): using a single call's token count to judge whether that answer is wrong. Measured AUC ≈ 0.5 — a coin flip. The invoice locates frontiers in aggregate; it does not grade individual answers. Bring a verifier.
Then, the replay that pays — policies over the same instances, real costs (cheap rung: gpt-oss-20b; "premium": o4-mini):
| policy | solve rate | $/solved |
|---|---|---|
| always cheap (gpt-oss-20b) | 0.65 | $0.00888 |
| cheap, retry ×3 on verified failure | 0.776 | $0.0135 |
| cascade: cheap → premium on failure | 0.692 | $0.0395 |
| always premium (o4-mini) | 0.363 | $0.15475 |
Two rules fall out, both violations of routing folklore. Retries beat escalation: near-frontier failures are coin-flips (finding 1), so re-rolling the cheap model solves more, cheaper, than calling in the big one. Premium ≠ stronger: on this task family the $4.84/M model solved half as much as the $0.22/M one — the only ladder that means anything is measured frontier on your task, which is what the card is.
25 fresh instances, levels [3.0, 3.6, 4.2, 4.8, 5.3], via OpenRouter; total spend $5.963.
| policy | solved | $/solved |
|---|---|---|
| always_gpt_oss_20b | 12/25 | $0.0074 |
| always_o4_mini | 9/25 | $0.1434 |
| formguide_router | 24/25 | $0.1854 |
The router ships in the repo under integrations/litellm: a LiteLLM-compatible FormGuideRouter (cheapest capable rung → mist retries → frontier escalation, per-rung token budgets, both max-token spellings), a machine-readable formguide.json generated from the measured card, and recalibrate.py for the drift problem (finding 2): frontiers moved a full dial unit in twelve hours in our data, so calibrations carry dates and a re-probe costs cents.
The card is a snapshot; the router is the reason to keep the snapshot fresh. One load-bearing caveat: this router had a free referee — SAT answers are checkable. Whether the trace can stand in for the verifier got its own round of experiments: finding 8. (Spoiler: bring the referee.)