According to their GitHub, the file needed is the -Q2_0.gguf version, which requires compiling their version of llama.cpp.

Easiest way to download is to use huggingface-cli if you have it available:

hf download prism-ml/Ternary-Bonsai-27B-gguf --include "*-Q2_0.gguf*"

If not, just download the file from https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf/tree/main and save it somewhere.

Now, we have to compile llama.cpp:

``` git clone https://github.com/PrismML-Eng/llama.cpp prism-llama.cpp cd prism-llama.cpp

For Mac:

cmake -B build -DGGML_METAL=ON cmake --build build --config Release -j$(sysctl -n hw.logicalcpu) ```

I'm using -DGGML_METAL=ON since this is a mac. If you are on another platform and need CUDA you can use something like this:

``` cmake -B build -DGGML_CUDA=ON cmake --build build --config Release -j$(nproc)

```

Note also the varying format of -j that's just the number of cores you want to use for compilation.

Now, you can run llama-server via:

build/bin/llama-server -hf prism-ml/Ternary-Bonsai-27B-gguf:Ternary-Bonsai-27B-Q2_0 --port 8080 --host 0.0.0.0 --temp 0.7 --top-p 0.95 --top-k 20

If you downloaded the file by hand, pass -m with the path to the gguf instead of the -hf switch.

Add any extra parameters as needed. --temp 0.7 --top-p 0.95 --top-k 20 are the recommended defaults according to their page.

Finally, you can visit localhost:8080 in your browser to interact with it. Note that --host 0.0.0.0 will bind with all interfaces, so it will listen internally and externally. If you don't need that, remove that switch.