Ornith 35B MoE at 262K on RX 6800M: ROCm vs Vulkan – VelsTech Lab

🧪 VelsTech Lab Second measured Lab: same 12GB RX 6800M, same q8 KV, but a 35B MoE with 28 CPU experts and a 262K context window. Tiny prompt, huge KV reservation.

What we tested

• GPU: RX 6800M 12GB (RDNA2, ~384 GB/s) – same card as Qwen 27B 16K test, so MoE vs dense is apples-to-apples on hardware
• Model: Ornith-1.5-35B-A3B-AD-Q5_K-Q4_K.gguf – 35B total, 3B active (A3B MoE), Q5_K + Q4_K mixed quant
• MoE CPU offload: --n-cpu-moe 28 – 28 experts forced to CPU (hence tensor overrides to CPU with mmap warning)
• Context: -c 262144 -fa on --cache-type-k q8_0 --cache-type-v q8_0, n_slots=1, kv_unified=false – one slot, 262K KV pool reserved
• Other: -ngl 999 --jinja --reasoning-preserve --parallel 1 --temp 0.6 --top-p 0.95 --top-k 20, n_threads=8
• Builds: Run 1 – llama.cpp/build/bin/llama-server (ROCm); Run 2 – llama.cpp_vulkan/build-vulkan/bin/llama-server (Vulkan)

How we ran it

llama-server \
 -m Ornith-1.5-35B-A3B-AD-Q5_K-Q4_K.gguf \
 -ngl 999 --n-cpu-moe 28 \
 --cache-type-k q8_0 --cache-type-v q8_0 \
 -c 262144 -fa on --jinja --reasoning-preserve \
 --parallel 1 --temp 0.6 --top-p 0.95 --top-k 20

Same 335-token prompt both runs (progress 322→331), generate ~274 vs 307 tokens. Server: listening on 127.0.0.1:8080, llama threadpool n_threads=8.

What we measured

Run            Prompt eval                 Decode (tg)                 Total          Graphs
─────────────────────────────────────────────────────────────────────────────────────────────
ROCm           3995 ms /335 tok = 11.93 ms/tok = 83.84 tok/s   100:25.16 179:25.52 257:25.60 → 25.62 tok/s   14.65s /609 tok  272
Vulkan         4029 ms /335 tok = 12.03 ms/tok = 83.13 tok/s   100:19.65 160:19.66 219:19.65 → 19.66 tok/s   19.59s /642 tok  305

Takeaway: At 262K + MoE, ROCm is ~30% faster at decode (25.62 vs 19.66). Prompt eval is tied ~83 tok/s – 335 tokens is trivial vs 262K capacity, so prefill doesn’t stress context. Total wall time reflects decode: ROCm 14.65s vs Vulkan 19.59s for ~600 tok.

What the logs actually said – and what broke

1. W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 999, abort (ROCm run) – same as Qwen: you forced -ngl 999 so --fit is ignored. Fix: drop -ngl 999 and log fitted layers, or explicitly set -ngl 60 and --n-cpu-moe together.
2. W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance – with --n-cpu-moe 28 you’re mmap’ing tensors that immediately get copied to CPU RAM. Add --load-mode none for MoE CPU offload.
3. kv_unified=false, n_slots=1, n_ctx_slot=262144 – one slot, 262K KV pool reserved even though prompt is 335 tokens. That reservation is why you need q8 KV – f16 at 262K would be ~2× larger and likely OOM. Decode speed here is with the pool reserved, not filled.

How this maps to the calculators

Weights ~35B mixed Q5/Q4 ≈ 13–14 GB (Q5 0.66 + Q4 0.61 averaged). KV q8 at 262K for a MoE is not the simple 2×layers×kv_heads×head_dim×2×context – MoE shares KV per token, so our VRAM Calculator overestimates if you enter 35B; enter 3B active + 28 CPU experts separately or use custom model with reduced KV. Even at 3B active, 262K q8 is ~262K × ~0.5KB → ~130 GB if fully resident – you’re not resident, you’re MoE-sparse + q8 + CPU offload, hence 19–25 tok/s is plausible, not 60 tok/s bound from GPU AI Performance Calculator which assumes full VRAM.

weights = params × bytes_per_weight / GiB
kv_per_token = 2 × layers × kv_heads × head_dim × 2 bytes (f16) or ×1 (q8)
kv_cache = kv_per_token × context × batch / GiB

decode_tok/s ≈ (memory_BW × 0.8) / bytes_per_token

Bottom line

Ornith 35B MoE at 262K does run on 12GB with q8 KV + 28 CPU experts, but you pay in decode: ROCm 25.6 tok/s is usable for short answers, Vulkan 19.6 is slower here (inverse of Qwen 16K). Both need --load-mode none and no -ngl 999 for optimal. Next Lab will be MoE vs dense on the same RX 6800M: this Ornith 35B-A3B (3B active) vs Qwen 27B dense 16K – active params vs context trade, same card, both engines.