Apparently you can skip entire transformer blocks at load time with minimal performance impact

The benefit is another trick to allow fitting a model that wouldn’t fit in your hardware otherwise. People currently rely on quantization, and this is just another tool that can be used for that purpose (and they can be used together as well)

Following recent (very cool) papers, I implemented this as a --skip-layers flag to a llama.cpp fork, so it just never instantiates the blocks you tell it to skip. Bake-time pruning already exists (--prune-layers, mergekit passthrough etc.); this is just the runtime version of the same idea.

One important note here is that which blocks you skip matters by orders of magnitude, so i had to ship it with a selector mechanism.

Results, figures, and the failure cases are in the writeup.
- Writeup: https://open.substack.com/pub/itayinbarr/p/you-can-skip-llm-layers-at-runtime
- Fork: https://github.com/itayinbarr/llama.cpp

Happy to hear what you think in general!