I built a GBNF grammar compiler that makes 8B models reliably call tools - here's how it works (deep dive)

I've been building a local agent in Rust (Eris) that runs on llama.cpp and uses an Obsidian-compatible vault as memory. ~50 tools (vault read/write, memory, reminders, web fetch, email, calendar, vision).

The biggest pain was getting small models to emit valid tool-calling JSON. You all know the drill: the model wraps JSON in code fences, invents tool names, forgets closing braces, adds trailing prose after the object.

My solution: I compile each tool's JSON Schema into GBNF rules at session start, so the sampler enforces not just 'valid JSON' but 'valid JSON with exactly the right keys, types, and enum values for this specific tool'. Then before each LLM call, I narrow the grammar to only the tools that the semantic router matched for this turn. 8B model choosing between 3 tools instead of 50 = way more reliable.

Wrote a detailed technical post about how it all works with real code from the project: https://eris-system.dev/blog/gbnf-grammars

Running Gemma 4 12B on a 4080 (16GB VRAM). Works great for chat + ~32k context + vision.

Repo: https://github.com/janpauldahlke/eris (Apache 2.0)

Happy to answer questions about the grammar compiler, the recovery loop, or the architecture.