DIY Jev

So this jev thingy is getting kind of big... tbh it seems overhyped by a large margin, but here we are. Not that its bad, just feels like we usually ignored larger things...

Anyway to the point of this post:

I’ve been experimenting with a simple Jev-like inference setup using ordinary open weight LLMs.

Ive done jev-like thing before with llms and i never felt the need that we have to have a separate "system one models" for that and that llms do fine.

So i played around a bit.

The main difference from OpenJev is that there’s no NLI fine-tuning or classifier head.

For each candidate answer I turn the problem into a boolean verification:

<state>...</state>

<question>...</question>

<options>...</options>

<candidate>B</candidate>

Then instead of generating anything, I read true/false logits for every candidate separately, subtract for every candiddate and softmax those scores.

So 3 candidates is 3 diffs that you softmax over.

The expensive state/question/options prefix you evaluate once, then candidate branches (only diff is last few tokens) are batched through llama.cpp.

On a 32,235-example benchmark:

Qwen3-4B 65.0% accuracy ~27 req/s

Qwen3 27B 75.3% ~2.9 req/s

Qwen3.6 35B-A3B 75.5% ~5.3 req/s

This req/s is measured on a laptop 5090 24gb.

The interesting part is that the approach works surprisingly well with completely unmodified models. Turns out same model can out perform the openjev fine tune.

Not claiming this reproduces Jev or that the benchmark is perfectly apples-to-apples, mostly interested in how far you can get without training anything and just playing with prompt effectively.

Repo: DIY-Jev GH

Check it out, give feecback and build cool things :)

Edit: I forgot to say hah The repo is a rust web server with jev compatible API that you can run local ggufs from HF in the style of jev. benchmarks included for a few models.