Rounds
Every batch of testing gets a writeup: what ran, what broke, and what the scores mean. The leaderboard is the summary — this is the evidence trail.
One Model Claimed 63 Escalations and Delivered 2: First Results from the Production-Replay Suite
We replayed a real production agent's nightly documentation work through four local models and scored them against what actually shipped — including a truthfulness check on their own work reports. One model fabricated its numbers. Only one answered all twelve tasks.
August 15, 2026 · suite 1H79/80 on Paper, Cloth Won't Pull in the Browser: Why We Now Run Every Front-End Artifact
A new browser-artifact suite produced a near-perfect 79/80 static score — then a human loaded the artifacts. The cloth sim doesn't respond to the mouse, the car is drawn backwards, and all four models in the field failed the same interaction the same way.
August 15, 2026 · suite 1A/1BMore Bits Isn't More Brain: A 5-Quant Ladder Where Q6_K Beat Q8_0
We ran the same 27B model at five quantization levels through the same 21-test gauntlet at temp 0. Quality was not monotonic with bits: Q6_K took the all-time roster record while Q8_0 — 12 GB heavier than Q4 — bought exactly zero extra points.
August 15, 2026 · suite 1A/1BThe Knob That Wasn't Attached: Sideloaded MLX Models Silently Lose Their Effort Controls
Two MLX builds of the same model kept stalling at settings that worked fine on their GGUF siblings. The reason: LM Studio accepts the reasoning-effort parameter for sideloaded MLX models at the API — then silently discards it.
August 15, 2026 · suite 1AThe Shipped Default Cost 39 Points: When "Think Harder" Makes a Model Worse
Qwen3.8-27B ships with reasoning effort set to xhigh. Head-to-head at a 49K token budget, xhigh scored 218/260 against medium's 257/260 — 5.2x the reasoning tokens, 3.5x the wall time, for a net minus-39.