What models you guys running on 8GB? 16GB VRAM? 24GB? 32GB? 48GB?
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
And what are you using for kv cache and context? What kind of performance are you getting?
What is your hardware? And what are you using your models for?
I figure with how fast everything moves, its worth asking once in a while to congeal our experiences.
[link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.