Local-LLM proving ground

Proving local agents before they act.

GAUNTLET Bench runs local models through real workloads — agentic, live, verified — and publishes the evidence.

All scores measured on Apple M5 Max, 128 GB unified memory, LM Studio

Current podium

Qwen3.8-27B Q6_K GGUF

27B dense · Q6_K GGUF

8/8 13.2 tok/s

Gemma-4 26B-A4B 8bit MLX

26B total / 4B active per token · 8bit MLX

8/8 66.2 tok/s

Qwen3.6-27B Dense 8bit MLX

27B dense · 8bit MLX

8/8 13.4 tok/s

Leaderboard — 35 models · click a column to sort

#▲	Model▼	Params▼	Quant / Format▼	G▼	A▼	U▼	N▼	T▼	L▼	E▼	T▼	Progress▼	tok/s▼
1	Qwen3.8-27B Q6_K GGUF	27B dense	Q6_K GGUF	99	100	90.1	89.3	96.5	75.7	98.5	5.5	8/8	13.2
2	Gemma-4 26B-A4B 8bit MLX	26B total / 4B active per token	8bit MLX	95.5	98	79.7	77.8	88	55.5	91	27.7	8/8	66.2
3	Qwen3.6-27B Dense 8bit MLX	27B dense	8bit MLX	91	100	93.2	99	97.2	68.3	91.5	5.6	8/8	13.4
4	Qwen3.6-27B Fable-Fusion-711 Uncensored Heretic NEO-MAX MTPLX 8bit MLX	27B	8bit MLX	95.5	97.5	88	—	91.5	45.5	100	6.4	7/8	15.3
5	Mistral Small 3.2 24B 8bit MLX	24B	8bit MLX	79	92.5	78.7	80	100	—	70	5.4	7/8	12.9
6	Qwen3.6-35B-A3B 4bit MLX	35B total / 3B active per token	4bit MLX	93	87.5	—	91.2	97.1	93.8	—	37.4	6/8	89.4
7	Qwen3.6-27B Fable-Fusion-711 Uncensored Heretic NEO-MAX Q8_0 GGUF	27B	Q8_0 GGUF	89	99.5	—	90.5	97.7	—	89	4.6	6/8	11
8	Qwen3.6-35B-A3B Unsloth Dynamic UD-Q8_K_XL MLX (Brooooooklyn)	35B total / 3B active per token	UD-Q8_K_XL mixed-precision MLX	93.5	95	—	91.2	100	—	—	28.6	5/8	68.4
9	Qwen3.5-122B-A10B 4bit MLX	122B total / 10B active per token	4bit MLX	92.5	95	—	90.2	100	—	—	16.5	5/8	39.4
10	Qwen3-Coder-Next 80B 4bit MLX	80B	4bit MLX	89	95	—	—	100	—	71.5	28.2	5/8	67.3
11	Laguna S 2.1 (Q4_K_M GGUF)	118B total / 8B active (MoE, 10-of-256 experts + 1 shared)	Q4_K_M GGUF	86	96	—	—	100	—	83.5	22.3	5/8	53.3
12	Bonsai 27B Ternary 2bit MLX	27B	2bit (ternary, PrismML extreme compression) MLX	82.5	97.5	—	—	93.1	—	99	17.2	5/8	41.1
13	Nemotron 3 Nano 30B-A3B 4bit MLX	30B total / ~3B active per token	4bit MLX	79.5	92	—	72.8	96.7	—	—	36.2	5/8	86.7
14	Qwen3.8-27B Q4_K_M GGUF	27B dense	Q4_K_M GGUF	94.5	99.5	—	—	95.7	—	—	9.3	4/8	22.2
15	Qwen3.8-27B Q8_0 GGUF	27B dense	Q8_0 GGUF	94.5	99.5	—	—	100	—	—	9.6	4/8	22.9
16	Gemma-4 E4B 4B 8bit MLX	4B	8bit MLX	82.5	93	—	—	100	—	—	31.8	4/8	76
17	Devstral Small 2 24B 6bit MLX	24B	6bit MLX	78.5	94.5	—	—	100	—	—	8.5	4/8	20.3
18	Muse-Glimmer 30B (GGUF, kquant 17GB)	30B (unconfirmed -- filename-derived, see note)	kquant (unconfirmed exact scheme -- filename says "kquant", not a standard llama.cpp quant label) GGUF	75	77	—	—	100	—	—	7.6	4/8	18.2
19	GPT-OSS 120B	120B	mxfp4 MLX	65.8	—	—	—	100	—	—	19.2	3/8	45.9
20	Qwen3.8-27B MLX 6bit (lmstudio-community)	27B dense	6bit MLX	51.9	75	—	—	n/d	—	—	6.7	3/8	15.9
21	Gemma 3 1B QAT 4bit MLX	1B	QAT 4bit MLX	27.5	39	—	—	n/d	—	—	100	3/8	239.2
22	Qwen3-VL 30B-A3B Instruct 4bit MLX	30B total / 3B active per token	4bit MLX	—	—	81.7	—	100	—	—	20.6	3/8	49.2
23	Qwen3.6-35B-A3B MLX 8bit Uniform (lmstudio-community)	35B total / 3B active per token	8bit uniform MLX	85.5	—	—	—	n/d	—	—	36.6	2/8	87.5
24	Gemma-4 E4B 7.5B Q4_K_M GGUF	7.5B	Q4_K_M GGUF	78	—	—	—	n/d	—	—	37.4	2/8	89.4
25	Devstral Small 2 24B 4bit MLX	24B	4bit MLX	74.5	—	—	—	n/d	—	—	14.5	2/8	34.7
26	Hermes-4 70B 4bit MLX	70B	4bit MLX	73	—	—	—	n/d	—	—	4.3	2/8	10.4
27	Qwen3-4B-2507 (non-thinking) 4bit MLX	4B	4bit MLX	68.5	—	—	—	n/d	—	—	57.5	2/8	137.6
28	GPT-OSS 120B Fable-5 Distilled	120B	mxfp4 MLX	62.7	—	—	—	n/d	—	—	14.6	2/8	34.8
29	GPT-OSS 20B (OpenAI mxfp4)	20B	mxfp4 MLX	62.3	—	—	—	n/d	—	—	36.4	2/8	87.1
30	Gemma 3 4B QAT 4bit MLX	4B	QAT 4bit MLX	61.5	—	—	—	n/d	—	—	68.3	2/8	163.3
31	DeepSeek-R1-Distill-Qwen-32B 8bit MLX	32B	8bit MLX	60.5	—	—	—	n/d	—	—	5.5	2/8	13.1
32	DeepSeek-R1-Distill-Llama 70B 8bit MLX	70B	8bit MLX	58	—	—	—	n/d	—	—	2.4	2/8	5.7
33	GPT-OSS 20B MLX	20B	MLX	52.3	—	—	—	n/d	—	—	14.1	2/8	33.8
34	GPT-OSS Safeguard 20B MLX	20B	MLX	50.9	—	—	—	n/d	—	—	24.3	2/8	58
35	Phi-3.5-mini-instruct 4bit MLX	3.8B	4bit MLX	31.5	—	—	—	n/d	—	—	46.1	2/8	110.2

Axes are 0–100. — = suite not yet run for that model. n/d = Thinking axis needs ≥15 observed tests to score (see methodology). Default order: GAUNTLET progress, then Generalist score.