Instrumentation & controls
Lab
The bench under the bench: instrumentation, hardware, and runtime findings that shape how we measure — published because a benchmark that doesn’t control its own confounds is just vibes.
Scores are only as trustworthy as the machinery that produced them. This section is the running record of what we’ve learned about that machinery — the silicon, the inference engines, the metrics themselves — and the controls we’ve put in place when a finding exposed a gap. Investigations stay open; entries get dated updates as the picture changes.
The M5 Neural Accelerator
Apple put a matrix engine in every GPU core. The speedup is real — and lopsided. Where it actually shows up in inference numbers, why decode barely moves, and how we confirmed our own MLX runs are served by the NA-aware engine.
TransparencyRuntime Version Tracking
A silent engine update between rounds is an invisible confound on every round-over-round comparison. Our policy: auto-update off, engine versions captured per run, deliberate review before any bump. Current engines and the changelog live here.
More investigations will land here as they mature: memory-pressure telemetry during long runs, prompt-length-controlled TTFT baselines, and effort-setting sensitivity in reasoning models are all on the workbench.