The GAUNTLET

The baseline gauntlet. A cross-section of the workloads a local daily driver actually gets handed: dev/ops scripting, document processing, professional content production. If a model can’t hold up here, the rest is academic.

Can the model behave inside an agent harness? Tool-call formatting, protocol adherence, and robustness when state gets messy. A model that free-styles its tool calls fails in production no matter how smart the prose is.

Vision-grounded document understanding: transcription accuracy, field and checkbox extraction, table reasoning — including a deliberately nasty real-degraded tier of scans and handwriting that separates demo-grade OCR from usable OCR.

Long-context retrieval under adversarial conditions: needles, distractors, ops-log reasoning, faithful summarization, and multi-needle MRCR. Scores here reflect what the model does with a genuinely full context window, not a marketing number.

The Thinking axis measures reasoning-budget behavior: how many reasoning tokens a model burns to get to an answer, and whether it stalls in its own chain of thought and never emits one. This matters enormously for agentic use — a model that thinks for sixteen thousand tokens and returns an empty answer is a dead turn to any harness, whatever its accuracy elsewhere.

Scoring (v1, stall census): we scan every archived run — 1,000+ per-test results across all suites — and count stalls (reasoning emitted, final answer empty) and runaways (token cap hit with no stop). The score is 100 × (1 − (stalls + runaways) / tests observed). A model needs at least 15 observed tests to be scored; below that the axis shows n/d rather than a number built on thin data. Repeat runs count as repeat observations — this is a behavior census, not a leaderboard average. Each model’s page shows its raw census: stalls, observed tests, and reasoning-token appetite (mean and max). Non-reasoning models can score here too: a stall-free record is a stall-free record. Effort-setting sensitivity is collected and will fold into a v2 of this metric. ) : key === 'tp' ? (

Throughput is the one axis that isn’t judged — it’s measured. We record mean generation speed (tokens/second) across every suite a model runs, on this host, and normalize against the fastest model tested so the axis lands on the same 0–100 scale as the rest of the radar.

Speed is reported per suite on each model page too, because it varies: long-context work is dramatically slower than short-form generation, and that difference is part of the evidence.

One-shot builds that must actually run. Output is executed in a real browser and verified at runtime — not just read by a judge.

Why runtime verification exists: static judge scores overstate reality. A judge reading source code will happily award points to a build that throws on load. The Live axis closes that gap — each artifact is opened in a browser and verified to actually run before it scores. What survives contact with a runtime is the score.

Coding depth beyond scripting: concurrency, algorithms, security review, performance work, refactoring under constraint, and test writing. The suite probes whether a model can engineer, not just autocomplete.