If your t/s is low enough, you can see speculative decoding with your own eyes
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
The other day I was trying out a distillation of DS4 Pro, and it came with MTP. It was slow as hell on my hardware, barely 2-3 t/s, BUT the speed got bumps every once in a while, and I noticed it was in moments like:
- United States of America
- First law of thermodynamics
- The enshittification of the internet
Basically, every time a very predictable phrase came up, it was instantly written. A fun thing to see.
But it also has me wondering - would MTP work together with n-grams? Since n-grams are Markov chains, the same engine behind autosuggest, how much sense would it make to combine them with speculative decoding?
[link] [comments]
More from r/LocalLLaMA
-
Voice conversations between Gemma4 12B and E2B on GPU and Jetson Orin
Sep 8
-
For Strix Halo - Official llama.cpp isn't ideal and how to highest possible throughput
Sep 8
-
WSJ: Unregulated Open-Weight AI Is an Invitation to Disaster
Sep 8
-
I made Warrior Quest, a local LLM-powered dark-fantasy RPG where the model only plays NPCs and the actual game state stays deterministic
Sep 7
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.