Hi all, I’m building a budget inference machine primarily for personal use (chat/assistant tasks, possibly some RAG). I'm torn between two hardware paths and would love input from anyone who has actually benchmarked these setups.

The Dilemma:

• Option A (GPU for dense models): Buy GPU(s) with 24GB VRAM and run the dense 27B model entirely on the GPU. For example, a RTX 3090 or 2 RTX 3060.

• Option B (RAM for MoE models): Buy a CPU build with 4 channels, perhaps 64GB of DDR4 RAM. The idea is to run the MoE 35B model entirely on CPU RAM using llama.cpp/GGUF.

• Option C (CPU for dense models): Most budget friendly, but how would the inference speed be? I assume it'll be too slow.

My core questions to the community:

1. Specific hardware advice: If I go CPU-only for the MoE, what is the minimum memory bandwidth (GB/s) and RAM channels I should target to make this viable?

2. Is a budget GPU necessary for CPU build? I saw discussions around that having a GPU will help with prompt processing, is this a necessary purchase?

I’m prioritizing a smooth chat experience over batch throughput. Any firsthand experience, llama.cpp benchmarks, or warnings about hidden bottlenecks would be hugely appreciated.

For context, I am UK-based, only considering used hardware. Budget: £500-600. Thanks in advance!