I run 35B–480B coding models on my 36 GB MacBook by streaming MoE experts from SSD — self-contained app, and I publish the benchmarks that *failed* too
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
I got tired of "your Mac can't run that" so I forked llama.cpp to stream a MoE model's expert weights from SSD instead of forcing the whole thing into RAM. A MoE only fires a few experts per token, so most weights sit idle — Slipstream keeps the always-needed weights resident and streams the routed experts on demand, into a bounded RAM cache that refuses to load anything that would swap your Mac. It ships as a native macOS app with the engine bundled inside — download the .dmg, drag to Applications, open. Nothing to compile. Point Kilo/Cline/Cursor/OpenCode at localhost:8080. (Not notarized yet, so the first launch is right-click → Open.) The evidence-first part, because this sub deserves real numbers: - Qwen3.6-35B-A3B (Q4), streamed from internal NVMe: ~13 tok/s at a 10 GiB cache (78% hit), ~19 tok/s at 14 GiB. The 35B is the interactive workhorse. - Laguna 118B-A8B, which does NOT fit in 36 GB: ~2.8 tok/s. Usable for batch coding, not chat. - The single biggest lever wasn't a clever kernel — I moved ONE file (the streamed experts) onto the faster disk and got 2.7×. Storage placement > everything. - A zero-copy path I'd wrongly written off at a tiny cache turned out to be +13–24% at real cache sizes — now on by default. And the ones that DIDN'T work, because that's the honest part: - Dual-SSD striping: negative on internal-NVMe + slow-USB (shared bus). - Speculative prefetch predictors (static + online): −8%. Prediction can't beat the read cost. - HOT-expert reservation: −1 to −6% (it shrinks the general cache). Full write-up incl. the negatives: BENCHMARKS.md. Repo + self-contained .dmg in the link. It's MIT, runs 100% on-device. Inspired by JustVugg/colibri (which does this for CPU/CUDA); I adapted it for Apple Silicon + Metal. Happy to answer benchmark questions — and if someone with two fast NVMe drives wants to test the dual-SSD path (it should win there), I'd love the numbers. [link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.