uncategorized

On AI

6 posts

Latest

Q8 Stopped Costing Speed: How Speculative Decoding Rewrites Quantization Math on Apple Silicon

The bandwidth math says a 4-bit quant should decode 1.7x faster than 8-bit. On an M1 Ultra it's 1.22x — and once you stack llama.cpp's MTP head with n-gram self-speculation, the gap disappears entirely. A benchmark story about which hardware limit you're actually sitting on, with one 211 GB mistake included.

llm · benchmarks · apple-silicon · 19 Jul 2026 · 19 min read

  1. Case File DS4 — A 284B Model on a 2021 Mac: The Local Frontier Verdict

    antirez's ds4 squeezes DeepSeek V4 Flash — 284B parameters — into 80.8 GiB that runs fully resident on a 2021 Mac Studio. We benchmarked it to 262k context, ran an independent HumanEval, AIME 2025, and GPQA Diamond suite it was never tuned for (92.7%, 10/13, and ~92% of the official GPQA score at 2 bits), A/B'd the 90.9 GiB hybrid quant, and kept the one negative result: speculative decoding makes this machine slower.

    llm · local-inference · apple-silicon · 6 Jul 2026 · 14 min read

  2. Case File VibeThinker-3B — A 3B vs. the Giants: The Reasoning Verdict

    VibeThinker-3B is Qwen2.5-Coder-3B with a reasoning post-train bolted on. We re-ran it under one harness — caught two benchmark bugs — and it ties a 27B reasoner on AIME and HMMT, beats a small frontier model, and leaps +90 over its own base. It's also the weakest model here on broad science. Both are true.

    llm · benchmarks · distillation · 20 Jun 2026 · 12 min read

  3. Case File Gemma-4 — Distilled vs. Stock: The Reasoning Verdict

    A 12B distilled from frontier teachers (Composer 2.5 + Fable 5) for Python coding, benchmarked head-to-head against stock gemma-4-12B across 249 items. It lost off-domain reasoning — and then lost its own home turf too.

    llm · benchmarks · distillation · 17 Jun 2026 · 8 min read

  4. The Chaos Engine: How Chunked Prefill Unmasked the Non-Linear Nature of Text Diffusion

    A 26 GB text-diffusion model demanded 523 GB of RAM. The fix was one parameter — and the parameter changed what the model said. Running diffusiongemma at its full 262K context on a Mac.

    llm · benchmarks · diffusion · 12 Jun 2026 · 24 min read

  5. Turbo3 vs q8_0: What KV Cache Quantization Really Costs on an RTX 5090

    A benchmark of two KV cache formats on Gemma-4-31B, and what the numbers actually mean for people who use long context.

    llm · benchmarks · 14 Apr 2026 · 26 min read