Session-Adaptive Orthogonal Distillation (SAOD)? Technology compresses 744B (1.5TB) to under 100GB?
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
| | Tweet : https://xcancel.com/jun_song/status/2079914426334167258#m Looks like 8GB VRAM could do more like even run 70-100B MOE models possibly. Sorry about the clickbait title, I want more eyes on this..... zzz [link] [comments] |
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.