Self-hosting LLMs on budget hardware: general principles, hardware, benchmarks and frontends
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
Hello,
I've been self-hosting LLMs on various budget hardware for a while (6x RTX 3060 12 GB, Intel Arc Pro B60 24 GB, RX 9070 XT, etc). Over the last few months, I wrote about it in 4 articles:
- General principles
- Hardware and inference optimization
- CPU+RAM offloading, MoE, prefill speed and benchmarks (or "Why some influencers sell unrealistic use cases")
- Frontends and example of complete configuration
I hope it may be useful to some people :-)
[link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.