MTP speculative decoding vs context depth (qwen3.6-27b / 35b-a3b)

Date: 2026-07-10 · Box: single RTX 3090 · Server: llama.cpp via llama-swap (llama_cpp_on_my_desktop).

Measures how much MTP (--spec-type draft-mtp --spec-draft-n-max 2, self-speculative multi-token prediction using each model's built-in MTP draft head) speeds up decode, and how that benefit changes from a near-empty context to a nearly-full one. Companion to the KV-quant change committed the same day (35b → f16, 27b stays q4_0 — see the llama-swap config).

TL;DR

• MTP is a decode win at every depth for both models, and its relative benefit GROWS as the context fills — it does not fade at long context.
• 35b-a3b (MoE): +19% (empty) → +30% (94% full).
• 27b (dense): +25% (empty) → +48% (94% full). The dense model gains more from MTP despite a lower accept rate.
• Absolute decode roughly halves from empty → near-full for both (KV attention cost), independent of MTP.
• MTP also costs VRAM: ~0.96 GB (35b), ~1.55 GB (27b) for the draft buffers — relevant to the 35b's tight f16 margin (see caveats).

Method

For each model, KV quant, and MTP state: prefill the KV cache to a target depth (a token-dense digit filler; increasing depths reuse the cached prefix), then generate 160 tokens and read llama.cpp's .timings from /v1/chat/completions via llama-swap. Thinking off, temperature 0, a fixed essay-writing instruction after the filler. Depth = prompt_n + cache_n (true context length). MTP accept% = draft_n_accepted / draft_n. Each point is a single 160-token burst → treat as ±a few %. Tool: /tmp/mtp_depth_sweep.py (reproduce by toggling the --spec-type line in each qwen base macro and restarting llama-swap). "94% full" = near each model's context ceiling (262144 for 35b, 240000 for 27b).

qwen3.6-35b-a3b — MoE, f16 KV, 262k window

Context depth	MTP on (tok/s)	MTP off (tok/s)	MTP speedup	accept%
~2k (empty)	166.8	140.0	+19%	64.0
~63k	139.4	114.5	+22%	61.8
~126k	121.5	96.9	+25%	62.5
~189k	106.2	84.1	+26%	61.3
~247k (94%)	97.0	74.9	+30%	62.0

qwen3.6-27b — dense, q4_0 KV, 240k window

Context depth	MTP on (tok/s)	MTP off (tok/s)	MTP speedup	accept%
~2k (empty)	47.0	37.7	+25%	57.8
~58k	34.2	26.1	+31%	52.3
~115k	28.1	20.1	+40%	52.3
~173k	23.1	16.3	+42%	50.3
~226k (94%)	20.4	13.8	+48%	51.3

Why the benefit grows with depth

Accept rate is roughly flat with depth (35b ~62%, 27b ~50–58%), so the growing speedup is not from better speculation deep in the context. It's the forward-pass economics: as the KV cache grows, each decode forward pass costs more (attention over more tokens), so each draft token MTP accepts — i.e. each forward pass it skips — is worth proportionally more. The dense 27b's forward pass is heavier than the ~3B-active MoE's, which is why the 27b gains more from MTP at every depth even with a lower accept rate.

Caveats

• Content-dependent. MTP gain tracks how predictable the output is. This used a middling essay task. Highly templated output accepts far more — the check-gpu-offload.sh "count from 1 to 40" probe hit 185 tok/s on the 35b (vs ~167 here); near-random output would gain less. Real agent/code workloads land around or above these numbers.
• KV quant differs between the two models (35b f16 vs 27b q4_0), so the absolute tok/s are not cross-model comparable — only the MTP on/off ratio within each model is.
• VRAM. MTP's draft buffers cost ~0.96 GB (35b) / ~1.55 GB (27b), measured near-empty. On the 35b at f16 this leaves only ~0.5 GB free at full context; if that margin ever OOMs, dropping to q8 KV (~2.9 GB free) or disabling MTP (~1.4 GB free, but −15–20% speed) are the levers.
• Single-burst measurements, -fit off (a fit is real), one warm llama-swap slot, no other GPU load.