Q8 Stopped Costing Speed: How Speculative Decoding Rewrites Quantization Math on Apple Silicon

The problem that started this

Last time I benchmarked Gemma-4-31B it was on an RTX 5090, where the whole game was fitting things into 32 GB of VRAM. KV cache quantization wasn’t optional there — it was the price of admission for long context.

This time the same model runs on the opposite kind of machine: a 2021 Mac Studio, M1 Ultra, 128 GB of unified memory, iogpu.wired_limit_mb raised so the GPU can address 112 GiB of it. Nothing here is squeezed. The Q8_0 weights are 32.6 GB and they rattle around in that budget. So the 5090 questions — what fits? — are boring on this machine.

The interesting question is different: which weights should you run when memory is free but bandwidth is not?

The M1 Ultra moves about 800 GB/s. A dense 31B model at Q8_0 streams ~32.6 GB through the GPU for every generated token. At Q4 it streams ~18.8 GB. Divide and you get the entire conventional wisdom of local inference:

 THE NAPKIN MATH (decode is bandwidth-bound)

 ceiling = memory bandwidth / bytes per token

 UD-Q4_K_XL:   800 GB/s ÷ 18.8 GB  ≈  42.5 t/s ceiling
 Q8_0:         800 GB/s ÷ 32.6 GB  ≈  24.5 t/s ceiling
                                       ─────────────────
                                       Q4 should win by 1.73x

Q4 should win by 1.73x. Everyone knows this. It’s why the default advice for Apple Silicon is “run the biggest Q4 you can fit.”

Measured, on llama.cpp build 9886, flash attention on, f16 KV cache:

weights	in-memory size	pp2048	tg128
UD-Q4_K_XL	17.5 GiB	136.2 t/s	15.8 t/s
Q8_0	30.4 GiB	138.9 t/s	13.0 t/s

Q4 wins by 1.22x, not 1.73x. More than half the theoretical advantage is missing. That gap is the first clue that “decode is bandwidth-bound” is a simplification with an expiry date — and the rest of this post is about what happens when the expiry date arrives.

Under the hood: work out the effective bandwidth each config achieves and the story tells itself. Q8_0: 13.0 t/s × 32.6 GB = ~424 GB/s. Q4_K_XL: 15.8 t/s × 18.8 GB = ~297 GB/s. The 8-bit quant streams weights much closer to the machine’s limit. Q4_K’s super-block format — scales of scales, packed mins, bit-shuffled nibbles — costs real compute to unpack, and on decode that unpacking sits directly in the critical path. Q8_0 is just a block of int8s and one scale: the GPU barely notices the dequant. Fewer bytes, but more work per byte. The napkin only counts the bytes.

---

Surprise one: KV cache quantization is a pure loss here

On the 5090, quantizing the KV cache was how long context fit at all. So the first thing I did on the Mac — muscle memory — was set -ctk q8_0 -ctv q8_0.

KV cache	tg128, fresh context	tg128 @ 8k tokens deep
f16	16.4 t/s	15.2 t/s
q8_0	13.7 t/s	5.6 t/s

Fresh context: an 8% loss. Deep context: a 3x collapse. The Metal flash-attention kernel pays a dequantization toll on every KV read, and at depth the KV reads dominate the whole token. On M1-generation GPUs, that toll is brutal.

The reason you’d accept that toll is memory pressure — and on this machine there is none. Gemma 4 uses interleaved sliding-window attention, so most layers keep only a small rolling KV window regardless of context length; even a 262k-token f16 cache fits without drama in 112 GiB.

Worth being precise: this is the mirror image of the 5090 result, not a contradiction of it. On a 32 GB card, KV quantization buys context that otherwise does not exist — slower long context beats no long context. On 128 GB of unified memory the trade buys nothing and costs 3x. Same knob, opposite verdict, because the constraint moved. Benchmark advice that doesn’t name its hardware is astrology.

So: f16 KV cache, always, on this machine. That’s constant number one.

---

The 515 MB sidecar nobody told me about

Unsloth’s GGUF repo for this model has a file that’s easy to scroll past: mtp-gemma-4-31B-it.gguf, 515 MB. It is a trained multi-token prediction head — a one-layer draft model that reads the base model’s hidden state and predicts the next-next token. Google shipped it with Gemma 4; llama.cpp grew support for it this spring (the gemma4-assistant arch, wired into the speculative-decoding framework).

Speculative decoding, one paragraph, no math: the expensive thing about generation is that every token requires streaming all 31B weights. But checking K proposed tokens costs barely more than generating one, because verification runs them through the model together, as a batch — like prefill does. So if something cheap can guess the next few tokens and the big model merely verifies, you get several tokens for the bandwidth price of one. Wrong guesses are discarded; correctness is untouched. The output is identical to non-speculative generation — you only pay extra when drafts miss.

The MTP head is that cheap guesser, fed by the model’s own internal state. Turn it on with two flags:

--spec-draft-model mtp-gemma-4-31B-it.gguf --spec-type draft-mtp

Measured on the Q4 weights, greedy decoding, 256–640 token generations:

task	baseline	+ MTP	acceptance
prose explanation (TCP congestion control)	13.6–14.3 t/s	15.3–15.5 t/s	160/283 (57%)
code edit (add type hints to a file)	12.7–14.2 t/s	21–23 t/s	470/480 (98%)

Two things in that table matter. First, the gain tracks predictability: +10% on free-form prose, +60% on a code edit. Second, that 98% acceptance number — when the model is reproducing code with a mechanical change, its own draft head almost never misses.

The ceiling is structural: Gemma 4’s MTP file contains one trained head, so it drafts one token per step. You can’t ask it for deeper drafts. To go further you need a second drafter with a different personality.

---

The free lunch: n-gram self-speculation

llama.cpp has a family of speculative modes that need no draft model at all: ngram-simple and friends. The drafter is a lookup table over the current context. If the model starts emitting a token sequence it has seen earlier — in the prompt, or in its own output — the n-gram drafter proposes the continuation of that earlier occurrence, many tokens at a time.

For an “apply this edit to this file” workload, think about what the output is: overwhelmingly a copy of the input with deltas. The n-gram drafter is a photocopier with a nervous twitch at the changed lines. It drafts long verbatim runs, the model verifies them in batches, and:

config	code edit tg	drafted / accepted
baseline	12.7–14.2 t/s	—
ngram-simple alone	33.5–36 t/s	594 / 443 (75%)

2.6x. From a flag. No download, no memory cost, no quality change.

And because MTP and n-gram fail in different places — n-gram is helpless on novel text, MTP only ever sees one token ahead — llama.cpp lets you stack them:

--spec-type draft-mtp,ngram-simple

 THE LADDER (code-edit workload, UD-Q4_K_XL)

 baseline            █████████████                          ~13.5 t/s
 + MTP               █████████████████████                  21–23 t/s
 + ngram (no MTP)    ██████████████████████████████████     33.5–36 t/s
 + MTP AND ngram     ██████████████████████████████████████████  42.5–43 t/s

                     737 drafted, 567 accepted (77%) → 3.2x

That’s the second constant: --spec-type draft-mtp,ngram-simple stays on.

---

The reveal: Q8 stopped costing speed

Now re-run the quantization comparison — the one Q4 won by 1.22x — with the speculation stack on:

weights	code edit, spec stack	acceptance
UD-Q4_K_XL (17.5 GiB)	42.5–43.0 t/s	567/734
Q8_0 (30.4 GiB)	44.7–48.5 t/s	560/734
UD-Q8_K_XL (32.6 GiB)	43.2–47.9 t/s	560/734

The 8-bit quants are now at least as fast as the 4-bit one. Nearly twice the bytes; same or better speed.

This is not a fluke; it’s the regime change the first table hinted at. Sequential decode is bandwidth-bound: one token per pass, all weights streamed per token, bytes are destiny. But speculative verification processes drafted tokens in batches — it’s prefill-shaped work, and prefill is compute-bound. Look back at the very first table: Q8_0’s pp2048 (138.9 t/s) was already marginally faster than Q4’s (136.2), because what matters under compute pressure is dequant cost per byte, where Q8_0’s flat int8 blocks beat Q4_K’s super-block gymnastics.

Speculation moves generation from the bandwidth column into the compute column. The quant tax was a bandwidth tax. Different bottleneck, different winner.

 WHICH LIMIT ARE YOU SITTING ON?

                sequential decode        speculative verify
                (one token / pass)       (batch of drafts / pass)
                ──────────────────       ────────────────────────
 bound by       memory bandwidth         compute
 bytes matter   directly (Q4 wins)       barely
 dequant cost   hidden by mem stalls     on the critical path
                                         (Q8_0 wins)

The practical consequence is genuinely new advice for this class of machine: with a working speculation stack, pick your quant on quality alone. The speed argument for Q4 — the entire reason it’s the default recommendation — evaporates on workloads where drafting works.

---

So: Q8_0 or Q8_K_XL?

Unsloth ships both. I read the tensor maps: UD-Q8_K_XL is Q8_0 with exactly 30 tensors promoted to F16 (embeddings and output-adjacent layers) — that’s the whole 2.2 GiB difference. Speed under the spec stack is identical within noise (raw sequential decode is ~11% slower: 11.6 vs 13.0 t/s).

So the tiebreaker is quality. And measuring it took a detour worth telling.

The model that forgot how to be a language model

The standard move is wikitext-2 perplexity. I ran it. Gemma-4-31B-it scored a perplexity of ~7,600. Not 7.6 — seven thousand six hundred. For calibration, a healthy model this size lands in single digits; 7,600 is “the model finds ordinary encyclopedia text almost unpredictable.”

My first assumption was a broken eval — wrong flags, wrong mask, a branch bug. I chased all of it: flash attention on and off (identical), full-size SWA cache (identical), BOS metadata (correct). Then the actual answer walked in the door. I asked the model for a raw completion — no chat template, just “The history of computing begins with” — and it produced:

the need to perform calculations calculations calculations calculations
calculations calculations calculations calculations calculations ...

Greedy decoding, straight into a repetition attractor. This instruct model is no longer a general language model. Its distribution has collapsed so far toward chat-formatted input that raw text — the thing perplexity benchmarks feed it — is out-of-distribution to the point of degeneracy. The wikitext number isn’t a bug; it’s true, and it’s useless. (Every coherent benchmark result earlier in this post came through instruction-shaped prompts, which stay in-distribution even without the template. That’s why generation looked fine while perplexity screamed.)

Worth being precise: if you see a wikitext perplexity table for a heavily post-trained instruct model — any vendor, any quant — treat it with suspicion. The measurement can be dominated by the model’s template dependence rather than by anything the quantization did.

Measuring what quantization actually changes

The metric that survives this mess is KL divergence between quants: run the same text through the F16-promoted UD-Q8_K_XL as reference, capture its full output distribution at every position, then measure how far each smaller quant’s distribution diverges from it. Corpus calibration cancels out — both models see the same tokens; we’re only measuring what the quantization perturbed.

I ran it twice: once on wikitext anyway (knowing it’s OOD), and once on an in-distribution corpus — eight full conversations (technical explanations, code, a refactor, a short story, math, systems advice) generated by the model itself and rendered through its own chat template. On that corpus the reference model’s perplexity is 1.97. The model is perfectly healthy in its element; it just isn’t a wikitext model anymore.

vs UD-Q8_K_XL reference	mean KLD (wikitext, OOD)	mean KLD (chat corpus, in-dist)	in-dist 99% Δp
Q8_0	0.0249	0.000198 (median 0.000000)	1.8%
UD-Q4_K_XL	0.625	0.026 (median 0.000116)	20.6%

Read the in-distribution row for Q8_0 again: median KL divergence of zero. At 99% of positions its top-token probability moves less than 1.8% relative to the F16-promoted reference. The 30 promoted tensors in UD-Q8_K_XL buy nothing this measurement can see.

Q4 is a different story. Its mean divergence is ~130x higher on both corpora — so no, this is not a wikitext artifact — and at 1 in 100 positions it shifts the top token’s probability by more than 20%. That’s not “broken,” and it’s consistent with the community lore that Q4_K quants are fine. But “fine” was always a trade: you accepted a measurably perturbed model because it decoded 1.7x — sorry, 1.22x — faster. The whole point of this post is that the speculation stack repealed the speed side of that trade.

Verdict: plain Q8_0. It’s KL-indistinguishable from the bigger UD-Q8_K_XL, decodes faster than it sequentially (13.0 vs 11.6 t/s), matches it under the spec stack, and is ~130x more faithful to the reference distribution than Q4 — at zero speed cost once speculation is on. The Q8_K_XL file is 2.4 GB of measurable nothing; the Q4 file is speed you no longer receive, paid for with fidelity you don’t have to give up.

---

The recipe

Everything above, as one command — this is what actually runs on my machine now:

llama-server \
  -m gemma-4-31B-it-Q8_0.gguf \
  --mmproj mmproj-F16.gguf \
  --spec-draft-model mtp-gemma-4-31B-it.gguf \
  --spec-type draft-mtp,ngram-simple \
  -c 262144 --mlock --cache-reuse 256 \
  --temp 1.0 --top-p 0.95 --top-k 64

What’s deliberately absent matters as much as what’s present:

• No -ctk/-ctv — f16 KV cache; quantizing it costs 3x at depth on Metal and buys nothing at 128 GB.
• No -ngl — llama.cpp offloads all layers automatically, and the new --fit machinery trims to device memory on its own.
• No -fa — flash attention defaults to auto/on now.
• No -ub — I swept microbatch 512/1024/2048; the 512 default won.

Two toggles I tested so you don’t have to: Metal op fusion (default on) is worth ~11% on decode — leave it alone. And forcing the new Metal 4 tensor API on M1 (GGML_METAL_TENSOR_ENABLE=1) works — the driver even reports the Metal4 GPU family — but delivers exactly nothing on this generation, which is why upstream gates it to M5-class chips.

---

The gotcha that wrote a 211 GB file

Confession, in the spirit of keeping negative results. Mid-benchmarking, I scripted a quick A/B through llama-cli with -no-cnv, the old “just complete, don’t chat” flag. This llama.cpp has split that job out into a separate llama-completion binary — and llama-cli warns you about the retired flag, then carries on into interactive chat mode anyway. Scripted, with stdin at EOF, it sat in a loop printing an empty > prompt. For two days. Into a log file that reached 211 GB before anyone noticed.

The benchmark data was fine — rerun properly through llama-server, which also reports draft-acceptance counts, which is how you got every acceptance number in this post. But let the record show the machine spent a weekend being extremely fast at generating nothing.

---

What I’d trust and what I wouldn’t

What I’d trust: every number above is from this specific machine (M1 Ultra, 128 GB, macOS 26), this specific build (llama.cpp 9886/c3fb97241), measured 2–3 runs per config, and internally consistent across llama-bench and llama-server harnesses.

What I wouldn’t generalize without re-measuring:

• The 3.2x is a code-edit number. Prose gets +10–15%. Your mix of novel-vs-derivative output decides your real-world gain; agentic loops, code review, and refactoring sit at the happy end.
• The KV-quant collapse is an M1-generation Metal result. Newer Apple GPUs (and CUDA, per the 5090 post) have different dequant economics.
• Greedy decoding flatters acceptance slightly. At the recommended temp 1.0 the draft-accept rates will dip; the ladder’s shape survives, the exact rungs move.
• The KL corpora are small — ~41k tokens of wikitext, ~7k tokens of chat-formatted text — and the in-distribution corpus was generated by the reference model itself, which is standard practice for this measurement but worth knowing. The 130x Q4/Q8 divergence ratio replicated across both corpora; the absolute values carry the usual small-sample salt.
• n=2–3 runs per cell. This is a field report, not a paper.

The one-line version: on a big-memory Mac, stop asking “which quant is fastest” — turn on the speculation stack, keep the KV cache in f16, and let quality pick your weights. The bottleneck you learned to optimize for may not be the one you’re sitting on anymore.