Fork of FreeToken with DeepSeek-V4.1, vision and speculative decoding (2x3090 numbers inside)

I've been running FreeToken on my 2x3090 box for a while and ended up maintaining a fork of it. Posting it in case it's useful to anyone else here.

Quick context if you haven't used it: FreeToken is an edge-native MoE serving engine. It offloads experts to host RAM/NVMe and co-executes on CPU+GPU so you can run big MoE models on consumer hardware. Upstream is here: https://github.com/FlashML-org/FreeToken

What I added on top of upstream:

- DeepSeek-V4.1-Flash (mHC, CSA2 sparse attention, lightning indexer, Engram n-gram memory, DSpark draft)

- Vision for Qwen3.8-Flash-Next, incl. OpenAI image_url input end to end

- Speculative decoding: MTP draft head (embedded or external artifact), rejection sampling, commit-the-accepted-prefix instead of re-extending, optional round chaining

- FTW (the fast weight format) now carries side tables (PLE/Engram) and the MTP head, plus TP slicing

- Tensor-parallel loading for a few families, optional fp8 all-reduce

- Reworked the quant config layer so each module's scheme comes from one place

- Various fixes (FTW TP band offsets, O_DIRECT reads, scheduler spec budget, Engram hashing)

My box, for reference:

- 2x RTX 3090 (24GB each, PCIe x16), tensor parallel = 2

- AMD EPYC 7203P, 8 cores / 16 threads, 8x DDR4-2400

- Model: nvidia/Qwen3.8-Flash-Next-NVFP4 converted as FTW (512 experts, offload)

- Serve flags: --moe-strategy offload --tensor-parallel-size 2 --moe-cache-auto --ple-backend pinned --expert-load parallel --max-prefill-length 8192 --disable-moe-prefill-overlap

Bench (256 output tokens, 5 runs, fixed prompt, tok/s avg / median):

- greedy, MTP off: 48.0 / 52.4

- sampled (temp 0.8), MTP off: 46.2 / 50.9

- greedy, MTP on: 21.5 / 22.4 (accept rate ~56%)

- sampled, MTP on: 34.0 / 43.8 (MTP is gated off for sampled by default)

So with CUDA graphs on the plain path, MTP is currently a net loss on this config. Under matched conditions (eager, no overlap) it was ~1.5x plain (17.2 vs 11.4), and the per-batch-size verify graph moved the MTP path from 16 to 28 tok/s in isolated runs. That verify graph is the lever I'm on right now, not claiming a win yet.

Still text-only serving like upstream; vision is the one thing I wired up. DeepSeek-V4.1 keeps its Engram table on disk by default, so budget the space.

Repo (anonymous fork, Apache-2.0, upstream credit kept): https://github.com/anon761/freetoken

It's a fork, not upstream, so expect rough edges. If you try it and something breaks, reply here or open an issue and I'll take a look. Also curious which models/quant formats people would want next.