Hello fellow local AI people!

I took "you must create your own benchmarks" literally, and built a website for this.

How does the end result look like

Let's say I want to know which model has most common sense in its responses, I did everything including evaluating responses (see below for what it is), and now I can ask:

What model did respond best (on average)?

1. I look into the results table for models' pipelines, best are at the top (first post picture)

Ok, where do the models fail?

1. I look at the second post picture which shows average score of all models over a query (second post picture)

Let's see that infamous car wash question runs

1. Runs table (third picture)

Okay, if they all more or less fail, let's see Gemma-26B-A4B for example

1. Gemma fails (fourth picture)

For a quick look, the last picture gives a more lucky example with more evaluation criteria. For the entire benchmarking workflow see https://beta.locallm.top and click "Manual" in the sidebar/mobile menu .

So at the current stage website lets us 1) create benchmarks, with sets of queries 2) create model/system prompt combinations ("Pipelines") to test 3) evaluate incoming answers and 4) see the comparison table.

There is like

• 10+ narrow-domain benchmarks with 5-18 questions (not all public) and 3-8 models fully or almost fully evaluated, (mostly created by domain experts in such fields as food safety, laser physics, wireless networking, English literature, psychology) and
• 25+ small 2-4 queries benchmarks, some incompletely evaluated, of various quality (created just for curiosity and/or by non-technical people).

By domain expert I don't necessarily mean a PhD, but at least a bachelor degree student or a person who has significant domain work experience.

Questions, questions...

What can I do so the website is more convenient to use / gives more insights? Are there any people who can contribute benchmarks in their own language - widespread, like Hindi/Arabic/French, or less so? Or some more or less known domains?.

There is so much mess everywhere in code (in UI, you should have noticed it in the screenshots already; in the parts of backend code which I didn't care to review and/or rewrite well). Only some things I implemented I am not fond:

• UI/UX is junky on some pages
• I called LLM + system prompt (+ rest which is not yet implemented) thing a "Pipeline" (used to call it "Template", now using that name for other translation languages only). What should it be called?
• Only simple Chat "Pipeline" is implemented (there are plans and architectural foundation for more, but that's all so far)
• How should I evaluate multi-turn conversations? (It is possible to chat with models with given system prompts and evaluate individual responses outside of the benchmark, they are saved in DB, but not the benchmarks currently). But how to weigh them? E.g. if a model asks for details, how to evaluate if it was valid (the question ambiguous) or stupid (every detail was already given)?

But the most mess is in my head about what to do next. Some things am not sure what to do with:

• If benchmark's responses are evaluated according to multiple criteria, should I add a formula to calculate e.g. weighted average score?
• Which next type of a "pipeline" should I focus on? RAG? Structured responses? Data extraction? Agentic?
• Should I add Audio transcription / image understanding benchmarks?
• I don't want to push image/audio generation, as this clearly leads to AI slop takeover, but maybe I am wrong?

There is much more I can write, but I don't want to overwhelm this already inflated post. Overall, I look at the thing and I am happy, but then - what next? Anyway, I plan to develop this website for years to come.

How does it work so far

Thanks to llama-server routing implementation, it is comparatively easy to write an *.ini preset file with list of HuggingFace IDs of the models, and when you request that model's response llama-server downloads and loads the model before responding.

Backend API gets models' answers come from a dumb relay which polls the website's backend (so that llama-server instance isn't facing internet directly), and passes the requests to llama-server instance it sees has access to (I called this thing "Model service"), then passes responses to backend. Job distribution between multiple services each having access to its own llama-server instances is possible. See Appendix for a table.

Acknowledgements

I am grateful to my wife, who supported me in my decision to spend a few months on this project at expense of other things.
Thanks to Rodrigo for the inspiration to work on this project and valuable discussions about LLMs.
Ačiū jūms, Monika, už kruopštų testų kūrimą ir klaidų taisymą.
Dėkoju Konstantinui už vertingas diskusijas.
Спасибо Вике, Никите и Анне :)
Dėkoju Kristinai ir visiem, kas padėjo kurtį kalbos modelių testus.
Thanks to the AI researchers and open-weight model companies and developers.
Thanks to llama.cpp and HuggingFace teams!

Appendix

Some philosophy and details

Public/Private benchmark split

I think private part of any benchmark is very important for benchmaxxing/contamination mitigation. Another reason to have private questions is for experimentation (e.g. you're not sure a question is simple/difficult enough to differentiate models). It is also possible to create different "Pipelines" (model + system prompt) and set them private; possible to run them over public queries or your own private queries in a public benchmark, and runs will only be accessible for you. Only one thing will be visible - average evaluation scores for the pipeline.

Multi-cultural

Questions are usually in English, Lithuanian and Russian, some few quesions in different languages too.

It would be nice to see language/domain coverage in LLMs.

Even frontier closed-source models sometimes lack nuance in understanding non-English/non-Chinese or a small country language which has little digital footprint (like Lithuanian), and this is even more a problem for small open-weight models. Even DeepSeek V4 Pro/Flash doesn't generate naturally sounding Lithuanian. The website is almost fully translated to Lithuanian (my country of origin) and Russian (my mother tongue). With help of my Mexican friend it is almost entirely translated to Mexican Spanish (not completely because it takes time for him to sync with my chaotic development).

Data export

All the data that is accessible to you can be exported (e.g. your own private benchmarks, private questions and private pipelines runs over public questions, which are only visible to you, will be in the data), also.

How is the llama-server instance working with the backend

llama-server	"Model service"	backend API	User on UI
	pokes backend	0 jobs
	pokes backend	0 jobs
			asks model something
		saving job entry	UI: is there an answer?
	pokes backend
		1 job	UI: is there an answer?
	asks llama-server
Loads model if not loaded			UI: is there an answer?
<LLM processing>
<LLM processing>			UI: is there an answer?
responds
	sends the answer		UI: is there an answer?
		saves the answer to DB
			UI: is there an answer?
		sends the answer to UI
			response visible in UI

Why this website

Generic part

Local LLMs usage is not limited to coding, and model fitness evaluation for a specific task is tedious sometimes. Of course, we see benchmarks and community impressions, but often this is not enough, and we don't have time to test all the new models well. I strongly dislike 1) choosing something with "gut feeling" 2) LLM-as-judge and I almost hate 3) benchmaxxing (which spoils everything related to benchmarks). If you're going to use a model even an hour a day for the next few months, it is worth spending an hour testing and more rigorously comparing it to the alternatives. Maybe I am just old-school, so be it.

Frustration part

I was participating in local AI project for a small business. So many times we changed a system prompt or some setting and got unexpected (in a bad way) responses to some questions. Running on a Mac and on RTX GPUs worked differently. Also, the company we tried to serve had inflated expectations of LLMs capabilities. The project reached some milestones since I abandoned it, but since then I wanted to have a convenient tool to A/B test models pipelines in a more relaxed setting while having more precise results.

Educational part

To date, even technical people and (non-technical folk even more) are often ignorant about open-weight models' capabilities. And to be honest, some local AI enthusiasts are ignorant in a way that local models aren't as capable as we imagine. Still, as an AI enthusiast (and similar to a way I am a Linux enthusiast) I believe we need to spread the word that closed-source 0-privacy is not everything that exists - that for a lot of tasks local AI is better given privacy and some other constraints. I had already made some impact in my small circle with the help of this site, I hope.

Learning part

Self-explanatory, at least for this sub's people ;)

Dreams

I hope it is only a beginning of the adventures of (not Narnia) LM Bench :)

P.S. No AI used for writing this post, even for spell-checking (Firefox extension works well enough for this). I hate AI slop. As I am a fan of time-tracking also, I know exactly writing this took 3 hours and 54 minutes.

P.P.S. I liked writing essays and learned markdown before lazy people with LLMs spoiled everything.