⚠ working notes — what we've learnt so far · updated 12 July 2026 difficulty–response curves

Finding 5 · Traces: the thinking you pay for but rarely see

Traces: the thinking you pay for but rarely see

A reasoning model doesn't just answer. First it fills a private scratchpad — often ten to forty thousand words of trying things, checking clauses, muttering "wait" — and then writes the short answer you see. That scratchpad is called a trace. Every word of it lands on your bill. Whether the words themselves are handed back to you is, it turns out, a lottery decided by the pipe.

Fig. T5a — what a reasoning model actually produces. You are billed by the pile, not the sticky note.

Here is a real slice of one, from DeepSeek-R1 solving one of our puzzles (we hold ~130 of these in full):

…so we need x8 to be T for the clause to be true. So x8 must be T. Similarly, clause 47: (NOT x8 OR NOT x5 OR x4), x5 T, so NOT x8 OR x4 must be true? Wait. (NOT x8 OR NOT x5 OR x4), x5 T, NOT x5 F, so NOT x8 OR x4 must be true…

Now the lottery. Ask three different companies for the scratchpad and you get three very different experiences at the counter:

Fig. T5b — trace delivery in practice. The open models hand it all over; OpenAI shreds it and gives you the count (on every route, including their own); Claude's depends entirely on which door you knock at.

The receipts — billed thinking vs delivered text, measured from our own calls
model	route	billed tok/call	delivered	verdict
minimax-m2.5	TrustedRouter → parasail	29,234	77%	full trace delivered
deepseek-r1-0528	TrustedRouter → novita	13,799	63%	full trace delivered
qwen3-235b-a22b-thinking-2507	TrustedRouter → novita	19,521	61%	full trace delivered
claude-haiku-4.5	TrustedRouter → Anthropic Messages (native)	6,817	6%	mostly withheld
claude-fable-5	TrustedRouter → Anthropic (chat translation)	11,169	3%	mostly withheld
claude-fable-5	OpenRouter → Anthropic	35,908	1%	count only — text withheld
gpt-5.5	TrustedRouter → OpenAI	9,330	0%	count only — text withheld
o4-mini	TrustedRouter → OpenAI	9,966	0%	count only — text withheld
gpt-5.5	OpenRouter → OpenAI	22,833	0%	count only — text withheld
gpt-oss-120b	TrustedRouter → cerebras	13,838	0%	count only — text withheld
gpt-oss-120b	TrustedRouter → novita	12,932	0%	count only — text withheld
gpt-oss-20b	TrustedRouter → novita	19,481	0%	count only — text withheld
minimax-m2.5	TrustedRouter → novita	25,538	0%	count only — text withheld
gpt-oss-20b	TrustedRouter → parasail	28,395	0%	count only — text withheld

The strangest row is gpt-oss: those models are open weights and emit reasoning by design, yet every serving stack we used strips it. Open weights ≠ open traces.

And the payoff question: where you do get the scratchpad, is reading it better than counting it? Yes — because of what stress sounds like:

Fig. T5c — what we measured in the traces. Repetition ("wait, try x3…" on loop) spikes exactly at each model's breaking point (within 0.5 dial units for R1, dead-on for Qwen); sheer length peaks well past it (+1.4 to +1.8). MiniMax is the exception that proves it: its giant scratchpads barely repeat — that's genuine search, not circling.

Working conclusion: count tokens to find the cliff anywhere; listen for the stutter to find it precisely — on the rare pipes that let you listen at all.

"The rare pipes that let you" is doing a lot of work in that sentence. Time to talk about the pipes.