For Strix Halo - Official llama.cpp isn't ideal and how to highest possible throughput
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
I've been making a lot of comments about optimal setup for Strix Halo (gfx1150) and from my observation, 90% of our community is using offcial llama.cpp for it, which is NOT optimized for Strix Halo at all, official llama.cpp is having extremely hard time to reach 50% hardware theory, wasting the silicon of this device.
Here's alternatives that can bring the speed of Strix Halo to a totally different world, I will link to user's sastifaction comment to prove that the result is real:
- https://github.com/peonist-ai/halogen-flash-server - ~50t/s decode and 1200t/s prefill - 90% theory - Optimized for Strix Halo and Qwen 3.8 Flash Next (Q38FN - The best goto model for Strix Halo) only, if you've heard about Ninfer, this is exactly Ninfer for Strix Halo.
- https://github.com/myhacsint/llama.cpp/tree/production/strix-halo-qwen4exp-b10685 - Almost 60t/s decode and 600t/s prefill - 80% theory - This is an experiment fork for Q38FN, having very high decode and pretty good prefill
- https://github.com/halo-box/strix-llama.cpp - Almost 30t/s decode and 800t/s prefill - 75% theory - The latest commit pushed prefill through the roof, this is r/StrixHalo very first fork that has been updating nonstop with a Discord server with a lot of users.
Note: Official llama.cpp running Qwen38FN at 2xt/s and 2xxt/s prefill - 50% theory.
Hopefully this will be helpful to the Strix Halo users.
[link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.