Agents are good at tools. They fall apart on control planes.
The strong tool-use results come from synchronous, single-turn settings. Real work persists, holds state, fires events, and asks you to delegate. Three numbers say it isn't solved.
We watch and grade the failure directly (missing events, 21 models × two protocols, on real tmux state) — and stay honest about cause: scaffold, model capability, and training data are all candidates we don't claim to have separated.
The signal is right there. The agent isn't looking.
A control plane fires asynchronous events — a sub-agent pane dies, a long job finishes, a bell rings, an interrupt arrives. Agents miss them: they poll at the wrong moment, never arm monitoring, or are busy in another pane. tmux makes every event queryable, so a missed-event rate is gradable.
Polling loses the event; arming a wait-for channel (or a hook / monitor-activity) catches it. The benchmark grades not just the end state but whether the agent perceived and acted on each event — per type: pane death, activity/silence, bell, job completion, interrupt.
Orchestrating sub-agents makes both worse.
When one agent drives sub-agents across panes, the trajectory gets longer, more asynchronous, and more out-of-distribution — turning up both dials at once. Here it fans work across panes, waits on a channel, and aggregates.
Each command runs against an isolated server (tmux -L <uuid> -f /dev/null); a PATH shim scopes the socket and logs every call — an exact, replayable trace of what the agent did.
tmuxbench — measure both, on real state.
- A · solo (sync) — sessions, panes, formats; a tmux-fluency baseline we subtract out.
- B · events (async) — the event taxonomy, graded for a missed-event rate: pane death, activity/silence, bell, completion, interrupt.
- C · orchestration — an agent spawns sub-agents in panes; fan-out, producer/consumer, failure-injection, handoff.
The metric
- Missed-event rate — per event type, did the agent arm monitoring, read the signal via a valid path, and react before proceeding? Graded on real state, not prose.
- Pass / score / efficiency — weighted assertion score and turns, with the pillar-A fluency baseline subtracted so event effects aren't raw tmux skill.
A sample event task, exactly as graded the load-bearing bit
id: B-DIE-002 tier: T4 pillar: events prompt: A sub-agent in pane 1 will exit unexpectedly. Detect it and respawn it, then finish the task. fires: pane death (remain-on-exit on) at a nondeterministic-looking time checks: - detected query: did the agent read #{pane_dead} / exit sink op: eq - respawned query: pane alive again + correct cmd op: eq - end_state query: final aggregate sink (sha256) op: eq metric: missed-event rate (death) · EHC · trajectory length // hooks may not fire their action in a scripted server, so we grade via // a file sink + #{pane_dead} polling + registration check.
Two hypotheses, designed to be wrong.
wait-for) beats polling; the gap widens with trajectory length.21 models. Sync fine. Events fail.
5,313 tool-calling runs across 21 models ($0.02–$3/M, via OpenRouter, graded on real tmux state, 11 tasks). The result: 20/21 score lower on events than on synchronous tasks — mean 64% sync → 28% events (a 36-pt gap). It holds under a second, independent agent protocol too: the text RUN:/DONE loop (9,334 runs) gives 21/21, mean 71%→21%, missed-event 0.88. The gap survives the protocol change — it's not a harness artifact. Reproduce: harness/run.py · tasks/.
Tool-calling loop, pass% with bootstrap 95% CI:
| Model | Pass (95% CI) | A·sync | B·events | C·orch | missed-evt |
|---|---|---|---|---|---|
| deepseek/deepseek-chat-v3.1 | 85% [80–89] | 97% | 75% | 78% | 0.25 |
| anthropic/claude-sonnet-4 | 78% [73–83] | 100% | 51% | 100% | 0.60 |
| openai/gpt-4o | 77% [71–82] | 92% | 57% | 96% | 0.43 |
| google/gemini-2.5-pro | 67% [61–73] | 90% | 41% | 87% | 0.51 |
| deepseek/deepseek-chat | 65% [59–71] | 80% | 44% | 96% | 0.63 |
| openai/gpt-4.1 | 62% [57–68] | 76% | 44% | 87% | 0.54 |
| openai/gpt-5-mini | 60% [54–66] | 77% | 55% | 4% | 0.32 |
| qwen/qwen-2.5-72b | 60% [55–66] | 76% | 44% | 61% | 0.67 |
| mistralai/mistral-small-3.2 | 50% [43–56] | 73% | 25% | 57% | 0.78 |
| google/gemini-2.5-flash | 48% [42–54] | 64% | 22% | 96% | 0.86 |
| openai/gpt-5 | 46% [40–53] | 52% | 50% | 0% | 0.38 |
| openai/gpt-4o-mini | 42% [36–49] | 76% | 10% | 35% | 0.89 |
| openai/o4-mini | 42% [36–47] | 72% | 19% | 0% | 0.76 |
| meta-llama/llama-3.3-70b | 39% [33–45] | 63% | 22% | 0% | 0.97 |
| deepseek/deepseek-r1 | 36% [30–42] | 62% | 17% | 4% | 0.80 |
| qwen/qwen3-32b | 27% [22–33] | 57% | 2% | 4% | 0.98 |
| google/gemini-2.5-flash-lite | 26% [21–31] | 49% | 7% | 4% | 0.92 |
| mistralai/mistral-nemo | 23% [18–28] | 46% | 3% | 0% | 0.99 |
| meta-llama/llama-3.1-8b | 16% [12–21] | 25% | 11% | 0% | 1.00 |
| qwen/qwen-2.5-7b | 8% [5–12] | 18% | 0% | 0% | 1.00 |
| meta-llama/llama-3.2-3b | 0% [0–0] | 0% | 0% | 0% | 1.00 |
Real mistakes, not grader artifacts (reference solutions pass): respawn-pane without reading #{pane_dead}, wait-for with no -S/-L, never reading #{window_activity_flag}/#{window_bell_flag}, finishing while a sub-agent pane sits blocked on input. Protocol note: the loop you use shifts individual sync scores (gpt-5 sync 41%→52%, gemini-2.5-pro 60%→90% from text→tool) — so cross-model ranking is harness-sensitive, but the within-model sync→events drop holds under both loops. That's the load-bearing result.
Does multi-model fusion escape it? No.
We ran TrustedRouter Synth — a fusion system (a panel of models → a judge → a synthesizer) — through the text loop on the 9-task suite (180 runs, both live presets). It scores among the strongest configs (82–86%) — yet shows the same sync→events drop, and dies on the same event: pane death (B-DIE) passes just 5/20, missed-event 0.75. A committee of models still doesn't watch #{pane_dead}.
| Fusion preset | runs | Pass | A·sync | B·events | C·orch | missed-evt |
|---|---|---|---|---|---|---|
| trustedrouter/synth · budget | 90 | 86% | 96% | 73% | 70% | 0.40 |
| trustedrouter/synth · quality | 90 | 82% | 100% | 63% | 50% | 0.55 |
Panel → judge (Kimi K2.6) → synthesizer (GLM 5.2), via the OpenAI-compatible /v1/chat/completions + a trustedrouter:synth tool; ~55s/call, ~$0.03/task. The blog's named presets (iris/prometheus/zeus) aren't callable on a standard key yet — only budget/quality are live. Smaller sample (10 seeds) than the 21-model grid, but the within-system sync→events drop (98%→68%) matches the headline. Code: harness/agent.py.
What closes the gap: a mature scaffold + frontier models.
The deficit above is not immutable. Run the same tasks and verifier through a mature agent scaffold (mini-swe-agent) with the newest frontier models — same shell-command interaction, just a stronger stack — and it nearly vanishes: a native sweep of 36 trials scores 34/36 (94%), events handled almost perfectly.
| Model (native harness) | Passed | Cost |
|---|---|---|
| anthropic/claude-sonnet-4.5 | 9/9 | $0.33 |
| openai/gpt-5.2 | 9/9 | $0.23 |
| z-ai/glm-5.2 | 9/9 | $0.00 |
| qwen/qwen3-coder | 7/9 | $0.02 |
Only 2 failures, both qwen3-coder: a format-string slip (A-FMT) and — tellingly — the same pane-death miss (B-DIE, perceived_death = 0) that sinks weaker setups. Takeaway: the missing-events deficit is a function of both capability and scaffolding — near-universal for a minimal scaffold across 21 models, but largely solved by the strongest current models given a mature scaffold. The hard residue is still pane death. Caveat: this sweep changes both scaffold and models vs. the pilot (and n=36), so it shows the gap is closable, not which factor closes it — a clean cross (frontier models in the minimal loop; mini-swe-agent on weaker models) is future work. Run via the Oddish/Harbor orchestrator (Modal); dashboard f547229c.
Small open models on the real suite (H100 size ladder).
The leaderboard above reaches open models only down to ~7B (via OpenRouter). To see the small end clearly we ran the full 11-task suite over a Qwen2.5 size ladder served on an H100 (text loop, 5 seeds, 314 runs):
| Qwen2.5 | A·sync | B·events | C·orch | missed-evt |
|---|---|---|---|---|
| 0.5B | 16% | 0% | 0% | 1.00 |
| 1.5B* | 48% | 0% | — | 1.00 |
| 3B | 20% | 0% | 0% | 1.00 |
| 7B | 60% | 0% | 0% | 1.00 |
| 14B | 80% | 0% | 60% | 1.00 |
| 32B | 80% | 28% | 60% | 0.70 |
Sync competence climbs cleanly with size (16%→80%), but events stay a flat 0% all the way to 14B — only 32B cracks them at all (28%, missed-event 0.70). For open models the missing-events wall is near-total below ~32B; the within-model sync→events gap is total (14B: 80%→0%). (*1.5B under-sampled, 39 runs.) Data: data/suite_h100.jsonl.
The generalization cliff: where does tmux get in the way?
To isolate tmux overhead from task difficulty, we hold the goal trivial and constant (retrieve N random tokens → a file) and scale two axes: indirection depth (D0 plain files, no tmux → D1 read a tmux option → D2 capture a pane → D3 target the right pane among decoys → D4 respawn a dead pane) and op-count N. Then we sweep a small-open-model size ladder (Qwen2.5 0.5B→14B, served on an H100) plus frontier models, 600+ runs. The cliff is sharp and marches right with capability:
| model | D0 files | D1 option | D2 capture | D3 targeted | D4 event |
|---|---|---|---|---|---|
| Qwen2.5-0.5B | 7% | 0% | 0% | 0% | 0% |
| Qwen2.5-1.5B | 20% | 0% | 0% | 0% | 0% |
| Qwen2.5-3B | 40% | 0% | 0% | 0% | 0% |
| Qwen2.5-7B | 93% | 33% | 0% | 0% | 0% |
| Qwen2.5-14B | 87% | 40% | 0% | 0% | 0% |
| gpt-4o-mini | 100% | 93% | 7% | 0% | 0% |
| gpt-4o | 100% | 67% | 47% | 20% | 0% |
| gpt-4.1 | 100% | 67% | 40% | 47% | 0% |
Pass %, averaged over op-counts 1–16. Three clean reads: (1) the "tmux tax" — a single indirection (D0→D1) collapses every model ≤3B and halves 7B/14B; (2) capture-pane (D2) is a wall for every open model (0% across the whole Qwen ladder) — reading pane output is the universal break point; (3) respawn/events (D4) wall everyone, including gpt-4.1. Op-count compounds independently (Qwen2.5-3B clears D0 at n≤4, not n≥8). Data + generator: data/ · cliff.py.
Can you train past it? RL moves the cliff exactly one rung.
A naive first try (RL on all depths) just memorized the generator. So we ran the controlled version: SFT + GRPO on Qwen2.5-3B over D0–D2 only, with a KL penalty to the base policy, holding D3 and D4 entirely out of training. Then we evaluate every depth:
| depth | base | RL'd (trained D0–D2) | |
|---|---|---|---|
| D0–D2 | 0/45 | 45/45 | trained |
| D3 · targeted pane | 0/15 | 15/15 (100%) | HELD OUT |
| D4 · respawn dead pane | 0/15 | 0/15 (0%) | HELD OUT |
| real 11-task suite | 3/33 (9%) | 5/33 (15%) | capability |
The cliff moves exactly one rung. D3 — a recombination of trained skills (capture-pane + a targeting flag) — generalizes completely (0→100%, never trained on it; Fisher p≈3×10⁻⁶). D4 — a genuinely novel operation (respawn, in no trained task) — transfers zero (0→0). So RL extends the agent to compositions of what it learned, not to operations it never saw. The real suite didn't regress (9%→15%, n.s.) — the KL penalty avoided forgetting, but brought no significant real-task gain. Scope: one model, one seed, single-turn; D3/D4 are harder same-family variants, not a new control plane — in-family compositional generalization, not general tmux competence. Code+data: rl/ · data/rl/.
The reading.
tmuxbench · missing events on interactive control planes · graded on #{real_state} · tmux 3.6a
code: github.com/posix4e/tmuxbench ·
harness/ ·
tasks/ ·
spec BENCHMARK_DESIGN.md ·
write-up paper/PAPER.md