tried predicting which MoE experts get used next token to speed up cpu/gpu offload, got some real numbers, is this actually implementable or am i wasting my time (30tg/s -> 150-200tg/s)

so ive been messing around with qwen3.6 35b a3b (MXFP4 gguf) on my 3060 12gb, doing the usual cpu/gpu offload thing where half the expert layers sit in ram and get pulled over pcie whenever needed. and like everyone whos done this knows the gpu just sits there idle waiting for experts to show up, pcie bandwidth is the actual bottleneck not compute
idea was pretty simple, use the models own MTP head (the thing thats normally used for speculative decoding) to draft the next token WHILE current token is still computing, then instead of just using that draft for token accept/reject, also peek at which experts that draft token wouldve routed to, and start prefetching THOSE experts in the background on a separate cuda stream. basically hide the pcie latency behind compute instead of eating it every single token
did some actual instrumentation on llama.cpp to check if this is even worth it before building anything (used fable 5 + gpt 5.6 to help me dig through the numbers and set up the analysis btw, not claiming i did all this math myself lol)
results were kinda surprising ngl:
• naive “just prefetch whatever prev token used” -> only 20.7% hit rate. basically useless, expert selection isnt that correlated between tokens
• but MTP guided prediction (using the actual draft head) -> 78% hit rate at top-8, goes up to 90% at top-16 (but higher K = more bandwidth so tradeoffs)
• theres also a hot expert thing going on, like top 64 experts (out of 256 per layer) cover 51% of ALL usage across the whole trace, power law as expected, so keeping those permanently resident helps too on top of prediction
• baseline right now is like 36 tok/s gen, theoretical ceiling if everything was magically already in vram is like \~200 tok/s (pure vram bandwidth bound), so theres a MASSIVE gap thats currently just pcie transfer time doing nothing
so yeah 78% hit rate with basically free compute (its literally reusing the mtp head thats already running for speculative decoding, not adding a new model) seems like it should translate to a big chunk of that gap closing
question for people who actually know inference engines better than me: is there something obviously wrong with this idea. is anyone already doing this and i just didnt find it. is the overhead of doing router-only forward passes on the draft token gonna eat the gains. does this fall apart at bigger batch sizes. genuinely trying to figure out if this is worth actually building into an engine or if im missing something that makes it not work in practice
not tryna build a whole new engine myself tbh (looked into it, decided forking llama.cpp makes way more sense than rewriting the world), just want to know if the core idea holds up before i or anyone else sinks real time into it
happy to share the trace scripts/raw numbers if anyone wants to poke holes in the methodology

Prefetch hit rate |Expected speed
Baseline (0%, today) |35 tok/s
50% |~70-75 tok/s
70% |~120 tok/s
85%+ |~180-200 tok/s (GPU/VRAM-bandwidth ceiling, PCIe stops mattering) \

I tested out Two Models GLM 5.2 (1 bit) and Qwen 3.6 35A3B MXFP4 with my engine flux-aeros (Screenshots attached ; Currently very buggy)
No Self promotion just Sharing what i found and tested locally