Ternary-Bonsai-2-27B-PQ2_0 is not completely lobotomized

I decided to run prism-ml/Ternary-Bonsai-2-27B-PQ2_0 through my own set of UNSCIENTIFIC benchmarks.

I needed something to compare it to, so I decided I would compare with another 27B model by filesize: unsloth/Qwen3.8-27B-UD-IQ2_XXS.

Since anyone considering running a 27B model with whatever VRAM budget these 2 models demand will end up choosing between these 2.

My table will be a bit bear with just these 2 models, so I threw in a few others too that are larger. I know it's a mix of MoE and dense models, but I have the benchmarks on hand so why not include them. Qwen3.8 q3/q4 quants also give you an idea what we are striving to match and there is 3.6 A3B and the newer Ornith 1.5 and Tiel Coder too.

About the benchmarks and what they test.

Most of them only test memory of context and retrieval, phrase reconstruction and understanding of context in different ways.

Standard Needle: This is the kind of needle test everyone runs and most people score > 90%. It hides passkeys in 21 locations of the context and asks the model to retrieve them. Any score below 100% is questionable.

Hard Passkey Needle with decoys: I was not happy with the standard needle test because most modern models pass 100% making it difficult to compare models. So, I developed a hard mode needle test. This, test hides 21 Passkeys in the context but has many decoy Passkeys. The end of the context I list the CONFIRMED passkeys (GUID's) but do not say which Passkey number they are. Asking for Passkey[01] means it has to go through all the Passkey[01] decoys in the context and compare them to the confirmed Passkeys. So multiple hops around the context are required to retrieve a passkey. When a model does badly at this even upping KV cache to F16 does not help save it.

Phrase reconstruction: I got this one somewhere on reddit and it still trips up some models. It breaks up phrases into multiple parts (8 in my testing) and asks the model to reconstruct the phrase from its parts. Models might leave out a word if the phrase still makes sense.

500 Multiple Choice Science Questions: Exactly that. Just tests science knowledge with 4 options A-D and the model chooses the correct answer. This test mostly shows how much knowledge was lost through quantisation when compared with other quants. Originally the test was designed to count the number of answer flips between 2 given KV quants. But I did not use it that way here. I got this test from a youtuber so the answers are public and possibly trained but as explained you can still see if damage was done to the model's general knowledge in the quantisation process.

Prose Challenge: I wanted to test a models understanding of a document or a prose and its ability to recall facts from that prose as well as test its ability to recall during a long conversation. So, I created a 1000 paragraph prose. I also have 2 questions about every paragraph. I feed it 1 paragraph from the prose, then ask it Question 1 related to that paragraph. Feed it the next paragraph and Q1 for that paragraph, until the context is mostly full. In my testing below, that was 250 paragraphs. Then I ask all the question 2's in a randomised order. How much can it truly remember? A Q1 score below 100% is not very good and means the model lacks attention of even recent tokens a few sentences back. For Q2 the score varies and higher is better. But the larger I make the context the worse the models perform. This has led me to the conclusion to not always chase higher contexts. It's pointless if it suddenly starts to forget most of what was said anyway and a compaction summary in a smaller context will retain more than a larger context. I also use the test to test at various KV quants and it can improve things a little but not as much as you think. But that's not being tested here today.

JS Coding: My latest test I developed. 100 Javascript challenges each requiring it to pass multiple test cases. It's kind of still under development and I have not run this yet for every model as it does take time. The challenges range from Easy to Very hard. Failing 1 test case fails the whole test. Tests are run with reasoning disabled. However, on failure it can retry with up to 16,384 token reasoning budged, then it must answer again. I realise models today are designed to perform best with reasoning but in order to speed up the tests I see if it can pass the test without reasoning first. I also keep track of the number of thinking tokens used, answer tokens, how many tests it had to reason but I won't be showing those here.

Toolery

You can download this bench for yourself. It's not mine but it tests tool use. So much good stats in the app but I will just list the Overall % score and I have not yet tested all models on this one since I just discovered it today.

How I tested

All tests done at 89,088 context, KV q4_0/q4_0. Seem a bit odd? I optimise for 16GB VRAM, so all my tests are done at these settings initially and I test higher KV quants if VRAM is available. And as I said higher KV in many cases makes little difference and, in some cases, perform worse. All tests are seeded at start and most repeated 5 times so the results are deterministic. All tests are also done with MTP disabled, so I do not test the draft models KV cache which might be F16. Yes, MTP can change the results in some cases but in my finding it's tiny and mostly does not happen.

Here are the results:

https://preview.redd.it/cmdnqqmxcjqh1.png?width=1143&format=png&auto=webp&s=3a3cbd90fd8bc9d6b74c7d6d6c8cdd69d488153c

Findings

Bonsai did not do all that badly compared to Qwen3.8-27B-UD-IQ2_XXS.

Standard Needle

Bonsai scored close to 100% and UD-IQ2_XXS did poorly, worse than Q1. Generally, I expect 100% in this test. But notice that Ornith 1.5 and Tiel Coder score low 90's which is a red flag.

Hard Passkey

Not many smaller models can 100% this but a few come close Qwen3.8 Q4 obviously did the best. And ISTA at Q3 does excellent. Bonsai does a bit better than Qwen3.8 Q2 of similar filesize and it's not far from our former favourite model Qwen3.6 35B A3B. But the real shocker here is Ornith and Tiel Coder's scores. These models are supposed to be upgraded 35B A3B models. As you will see this trend continues and these 2 models have serious memory retention issues.

Phrase reconstruction

Bonsai aces this test with almost a perfect score compared to UD-IQ2_XXS at only 69%. Tiel coder performs worst even worse than a Q1 model.

500 Multiple Choice Science Questions

Bonsai shows almost no knowledge loss compared to even Q4 models. UD-IQ2_XXS on the other hand does start showing a loss and Q1 even more so.

Prose Challenge

Question 1 I expect 100% and most including Bonsai achieved that. Concerning again that Tiel Coder and Ornith could not even recall from the last paragraph.

Question 2 Bonsai and UD-IQ2_XXS are close maybe margin of error. Q3/Q4 models outperform it but a large margin. Except Swift, which is a model with significantly less reasoning. Here we can see some of the damage that was done to the model to achieve that. Tiel Close and Ornith again clock in with shocking results. Tiel Coder's 6% is probably as good as just guessing. I would say maybe 3B active parameters are just not enough. But Qwen3.6 A3B scores 39.2% significantly better. I tried upping Ornith's KV to F16 and it improved to 24.4%. I also tried a Q6 quant of the model at q8_0 which scored 25.5%. End of the day I think Ornith and Tiel Coder have an issue with recall regardless of Quant and KV Quant.

JS Coding

Bonsai was actually able to hold it's own against ISTA Q3. It did burn significantly more thinking tokens and had to reason on many more challenges. UD-IQ2_XXS on the other hand shows significant loss of coding ability. 10% below Bonsai.

Toolery

Bonsai did better than UD-IQ2_XXS. I am still learning to interpret the numbers, but the app has options to select your use case and it's applies weights to calculate a score. It also tells you the strength and weaknesses of each model you test. I also found that upping KV quant improves this score but a KV F16 Tail using beellama makes the biggest difference since tool calls are happening in the tail.

Conclusion

If you are VRAM constrained <= 12GB Bonsai might be a model to consider. But it will depend on how you plan to use it. Since I have 16GB I will stick with ISTA Q3 and I can run it with kvarn5/kvarn5 and MTP (kvarn2/kvarn2) and a 1024 token F16 tail.

Disclaimer

These tests do not test intelligence or real-world performance. They are purely synthetic.