Tiel-Coder 35B-A3B on RX 6800M: MTP vs non-MTP at 262K – VelsTech Lab

🧪 VelsTech Lab Speculative decoding via Multi-Token Prediction (MTP) on a 35B-A3B MoE model – does the draft head actually speed things up on a 12 GB mobile GPU with 28+ CPU expert fallback?

Two builds tested

• Machine: R9 5900HX (8C/16T) · AMD RX 6800M 12GB (RDNA2, mobile) · 32 GB RAM · Ubuntu 26.04 · ROCm 10.0 · llama.cpp (current build)
• Model (no MTP): Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf – 35B total, 3B active MoE, Q4_K_XL quant
• Model (MTP): Tiel-Coder-35B-A3B-MTP-UD-Q4_K_XL.gguf – same model with a multi-token prediction draft head
• Context: -c 262144 with --cache-type-k q8_0 --cache-type-v q8_0 -fa on
• Offload: -ngl 999 (forced – see warning below)
• CPU experts: --n-cpu-moe 28 (non-MTP) vs --n-cpu-moe 32 (MTP) – note the slight difference
• Prompt: ~608-617 tokens (same prompt both runs), n_threads=8, kv_unified=false, n_slots=1, n_ctx_slot=262144

How we ran it

Run 1 – no MTP

export LD_LIBRARY_PATH=/opt/rocm/lib:$LD_LIBRARY_PATH
llama-server \
 -m /home/user-name/models/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf \
 -ngl 999 \
 --n-cpu-moe 28 \
 --cache-type-k q8_0 \
 --cache-type-v q8_0 \
 -c 262144 -fa on \
 --jinja --parallel 1

Run 2 – MTP speculative decoding

export LD_LIBRARY_PATH=/opt/rocm/lib:$LD_LIBRARY_PATH
llama-server \
 -m /home/user-name/models/Tiel-Coder-35B-A3B-MTP-UD-Q4_K_XL.gguf \
 --spec-type draft-mtp \
 -ngl 999 \
 --n-cpu-moe 32 \
 --cache-type-k q8_0 \
 --cache-type-v q8_0 \
 -c 262144 -fa on \
 --jinja --reasoning-preserve --parallel 1

What we measured

Run 1 – no MTP (--n-cpu-moe 28)

prompt eval:   5220.06 ms / 621 tokens (  8.41 ms/tok,  118.96 tok/s)
eval (decode): 5155.86 ms / 132 tokens ( 39.36 ms/tok,   25.41 tok/s)
generate:      100 tokens @ 25.39 tok/s (tg), 25.65 tok/s (tg_3s)
total:         10375.93 ms / 753 tokens
graphs reused: 131

Run 2 – MTP (--spec-type draft-mtp, --n-cpu-moe 32)

prompt eval:   5574.58 ms / 621 tokens (  8.98 ms/tok,  111.40 tok/s)
eval (decode): 5105.25 ms / 131 tokens ( 39.27 ms/tok,   25.46 tok/s)
generate:      100 tokens @ 29.09 tok/s (tg), 29.38 tok/s (tg_3s)
draft:         acceptance = 0.54667 (82 accepted / 150 generated), mean len = 2.64
total:         10679.83 ms / 752 tokens
graphs reused: 50

Head-to-head

Metric               No MTP              MTP           Delta
─────────────────────────────────────────────────────────────
Prompt eval          118.96 tok/s       111.40 tok/s    -6.4%
Decode (slot)        25.41 tok/s         25.46 tok/s    +0.2%
Generate (tg)        25.39 tok/s         29.09 tok/s   +14.6%
Generate (tg_3s)     25.65 tok/s         29.38 tok/s   +14.5%
Total time           10.38 s             10.68 s        +3.0%
Graphs reused         131                  50            -62%
Draft acceptance      –                    54.7%          –
Mean len              –                     2.64          –

akeaway: MTP speculative decoding delivers a +14.6% decode speedup (25.39 → 29.09 tok/s) on a 35B-A3B MoE model. The draft head accepts 55% of its predictions with a mean run of 2.64 tokens. Graphs reused drop 62% (131 → 50) because the draft context reuses fewer cached graph executions. Prompt eval takes a minor hit (-6.4%) due to the extra MTP context. Note that --n-cpu-moe also differs (28 vs 32) – the extra 4 CPU expert threads in the MTP run may contribute a small part of the gain.

What the logs said – and what broke

1. Both runs: W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 999, abort – You forced -ngl 999 on a 12 GB card, so partial CPU offload is inevitable. The model is 35B at Q4 (~17.5 GB weights) + q8 KV at 262K (~12 GB+) – nowhere near fitting in 12 GB. Most MoE expert computations fall back to CPU via --n-cpu-moe.
2. Both runs: W tensor overrides to CPU are used with mmap enabled – consider using --load-mode none for better performance – Some tensor overrides force CPU fallback. --load-mode none may improve performance by decoupling mmap.
3. MTP run only: W device 'ROCm0' does not have support for op TOP_K needed for sampler 'top-k' – Relevant for the MTP draft sampler; may affect draft quality. The 54.7% acceptance rate might improve with a backend that supports top-k natively.
4. MTP run only: I common_speculative_init_result: creating MTP draft context against the target model '...MTP-UD-Q4_K_XL.gguf' – MTP creates its own draft context which adds ~200ms to init time (visible in the slightly higher prompt eval time).
5. gent difference: --n-cpu-moe 28 (non-MTP) vs --n-cpu-moe 32 (MTP) – this 4-thread difference is a confound. Re-running both with the same value would isolate the MTP-only gain.

Bottom line

On an RX 6800M 12GB at 262K q8 KV, the MTP draft head provides a meaningful +14.6% decode speedup on this 35B-A3B MoE model, raising decode from 25.4 to 29.1 tok/s. The overhead is minimal (3% more total time, 6% slower prompt eval). Given that --n-cpu-moe also differed, the actual MTP-only gain is likely slightly lower – call it ~10-12% net. If your backend supports top-k for the MTP sampler, acceptance rates may improve further. For chat workloads (decode-bound), MTP is worth enabling on any MoE model that ships a draft head.