r/LocalLLaMA · · 1 min read

How to run LLMs as regular guy with low resources?

Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.

Hello everyone. I hope is all well with you all. I been around this sub for a few months and been quietly reading and I have seen how many of you are spending $10k on the Mac M5 and I am assuming you are using it for work or just for fun but have the resources to spend like that. I am a regular guy with a 9-5 job and I would like to experiment with local LLMs but I dont have much disposable income to spend big amounts on GPUs or Macs. I currently have an I7 - 8700 and Nvidia 2060 6Gb VRAM - 32GB RAM

Now my question is, does anybody have an idea on what models or hacky stuff I can do to try the latest AI models on my hardware?

I have checked hugging face but I see so many models with the same name but letter and numbers at the end. I understand some of them mean Quantization which is dumber a model but I also found out tiny models like Linq which can be run even on the CPU. So I would like to connect with someone with more experienced to have fun with AI.

Thank you everyone .

submitted by /u/pet3121
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/LocalLLaMA