An 8GB Laptop Runs a 35B MoE Model at 33 Tokens per Second: the llama.cpp Flag and Its Cliff

A 16.5 GB model running at reading speed lives on an 8 GB laptop GPU. An XDA write-up delivered the numbers on 2026-10-22, measured with llama-bench on a Lenovo Legion 5 carrying an 8 GB RTX 3070 Mobile — and the two rows I keep rereading are not the winner, they are the two rows above it.

The first row is a dense control, and it is the most honest hardware line I have read in a while: a smaller file, on the same machine, generates at 0.75 tokens per second. A dense model has nothing to leave behind — every parameter is read for every token, so the PCIe bus between your GPU and your RAM sets the speed, and no flag argues with that. The second row is the same runtime struggling because a 16.5 GB file on an 8 GB card spills everywhere it should not. The third row is one flag: put the routed expert weights of 28 layers into system RAM and keep attention, embeddings, shared weights and the whole KV cache resident on the GPU. Roughly 3 billion of the model's 35 billion parameters are active per token, and that is the part the flag keeps on the fast side of the machine.

I checked what the flag actually does at the source. llama.cpp's own arg.cpp defines -ncmoe N as "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU" — nothing else moves, which is exactly what the article claims.

The part the article leaves out is that the flag is a cliff, not a dial. The cliff predates the article by five months: a note.com reproduction — an independent builder sweeping the same flag on an RTX 3070 8 GB desktop with a Ryzen 7 3700X and 32 GB DDR4 — was verified 2026-05-10 on b9119, following the Reddit RTX 3060 12 GB thread of 2026-05-08; XDA, published 2026-10-22, re-reported a trick the community had already mapped in May. Generation crawls from 13.7 t/s at -ncmoe 22 to 16.99 at 26 — then jumps roughly 5x at 27 (pp512 94.85 → 517.25, tg128 27.01) and slides gently after. And on a 12 GB card, in that Reddit thread, the sweep runs the other way: smaller N, more experts on the GPU, is faster. My inference, not the builder's: under about 8 GB the cliff is the point where the non-expert footprint finally fits, so you offload more experts than feels right. The reproduction adds two more 8 GB rules: flash attention is mandatory, and q4_0 KV quantization stays stable (517 ± 34 t/s) where q8_0 gets noisy (470 ± 93). The XDA run used q8_0 and published no spread; the reproduction's best is 27 t/s against the laptop's 32.82 — the same cliff at different heights, one machine's memory bus published, the other's not.

Both sets of numbers are self-reported llama-bench runs with three repeats — which is also the transferable object here. The skill is not the flag, it is the sweep: four llama-bench lines tell you where your card's cliff is, and the question stops being does it fit. What nobody has published yet is the interaction with --moe-cache-mib, the GPU-resident expert cache that shipped in v0.7.0 claiming up to 2.2x decode — one pins expert layers in system RAM by layer count, the other caches hot experts by slot in VRAM. If you are on an 8 GB card, run the sweep, keep the table, and watch for the day someone combines the two levers and the cliff moves.