A better coder for the small-GPU/small-RAM crowd!

I’ve been working on making small models more capable at agentic coding and work, because most people in the world don’t have the sort of hardware needed to run 3.8-27B, or even 35B-A3B or 9B dense, and I want to extend local agentic coding capability to less privileged users. This quant can be run on a smart phone or older gaming laptop, and can solve real coding problems autonomously in a way I have never seen or measured for this model class. Spark-X2.5-4B is already around best-in-class for its size, and I think these improvements bring out the best in it. I hope this little step up in small-model capability and speed in real-world coding might give new life to older hardware that would otherwise be forgotten in the AI frontier race.

The changes SharpSpark makes to Spark-4B are in three parts: First of all it fixes issues with the chat template, and replaces the system prompt with one that improves agentic coding behaviour, token use, and correctness. Then a custom importance matrix is calibrated for the model, which relocates bit precision within tensors to the parts that are more important to agentic coding work. The imatrix corpus is heavily weighted against both agentic coding and cybersecurity, which together protect the cognitive core used to find and solve hard bugs.

Then Spark is quantized with an optimized non-standard quantization strategy, that allocates bits differently per-tensor than standard llama.cpp GGUF quantization. I built a tool that explores and tests different per-tensor allocations to optimize KL-divergence and long-context retrieval for this model, but ended up making some manual changes that ended up favouring SWE-bench-Live performance over traditional fidelity measures like KL-divergence, which published science indicates is actually a poor proxy for real-world performance on complex tasks below a certain point.

If you have a small GPU and/or <= 16GB RAM and can’t run a 35B-a3b MoE-based model with partial GPU offloading, this is likely your best option for long-context agentic software development right now. SWE-bench-Live is chosen as the metric for its genuinely difficult real-codebase problem set.

I’m just a volunteer doing this as a non-profit side project, so please be kind about the fact that my benchmarks are not extensive. They are what I could afford the time and effort to run, with all my other projects, and I see them as just good enough to prove the improvements on the specific kind of work this quant was designed towards.

https://huggingface.co/peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF