LabyrinthBench: a local-focused, judge-free LLM benchmark that measures context recall under interference for multi-step agentic tasks.

LabyrinthBench measures the thing that actually kills long agent runs — whether a model can still use what it learned twenty turns ago — deterministically, with no LLM judge, on your own hardware, with a swappable harness for testing whatever context-management strategy you think will fix it. The first registered experiment surprised me in both directions: the same context trick that lifted seven models broke two. Two commands and a browser tab get you a live run on your own rig. And there's a leaderboard — the strategy lane is open, and I'd love to see someone beat mine.

I think just about everyone who's used AI for any meaningful amount of time has run into this scenario: you make a great plan with the AI and get most of the way through it — then you hit the context limit. You reluctantly hit "compact" or equivalent and keep going, and everything eventually comes off the rails because the compaction removed something weight-bearing. "It's time for me to come clean. You expressly told me not to do that and I did it anyway." This is an unfortunately common user experience, and personally makes me want to see how far I can throw my keyboard in the aftermath.

I wanted to measure what's actually happening there — objectively, deterministically. And I absolutely hate that benchmarking an AI currently requires some other reasoning entity to score, grade, and judge it. We've spent literal decades (if not centuries) building standardized, objective scoring methods in both the classroom and the laboratory. Ask the right questions in the right way, expect probabilistic output, and you don't need a judge at all. If there is a way to measure something objectively and deterministically, why would we ever settle for measuring it subjectively and probabilistically?

So I put the model in a maze and told it to find the way out. A maze on its own isn't novel; couple the navigation with simple, deterministically scored questions and different context-management strategies, and some really interesting things start to shake out.

The maze came first — dead ends, a loop trap, one exit — with simple standalone gates like Calculate: 38 + 17 and Evaluate: NOT(FALSE OR FALSE). It worked, but navigation was crushing the smaller models before memory ever got tested, and rather than build an even bigger harness to help them navigate, I cut navigation out entirely for now. The second map family is a straight 20-gate corridor where the questions chain instead: gate 1 is Calculate 3 + 4, and by the back half you're getting gates like Subtract your c1b answer from the previous gate's answer — every later gate reaches back to a specific earlier answer, so the model has to recall anything it's done, on demand. And because clean recall is still the easy case, the third family changes the values behind your back: Variable A is initialized to 3. What is the value of A? … then, gates later, E changes: E is now 4 … then the same question you already answered, with a change in between — recompute, don't reuse. Navigation → retention → currency. Each map exists because the previous one answered its question. (Sample questions from every family: https://labyrinthbench.ai/data/questions — the launch maps are the open practice set; competition instances are minted fresh per season and sealed.)

The first experiment: 13 local models on the 20-gate corridor, twice over — once keeping the full chat history, once with context wiped every turn and only the model's own recorded gate answers re-injected. Six runs per model per condition; the pass bar (median depth up 5 or more gates) was locked before any wiped run existed.

https://preview.redd.it/gdyyt3h33yhh1.png?width=1280&format=png&auto=webp&s=8d50e9d21f518bae1669a8aac353c77cfc644f67

Wiping won in 7 of the 9 models that had room to show a gain — and backfired in the other 2. (The remaining four of the 13 already ran at the map's ceiling, where a gain can't show — they got their own registered follow-up; that brief is in the data annex.) deepseek-r1:70b went from a median of 1 to 20: five of its six control runs cleared exactly one gate; wiped, it exited all six. glm-4.7-flash and qwen3:14b both gained 15.5 gates of median, and four of the seven winners went from exiting 0–33% of their runs to 83–100%. Wiping lifted qwen3:14b to a 20/20 median — the same ceiling the 120B-class models occupy unaided on this map.

The two reversals: llama3.3:70b fell from a median of 15 to 9; llama4:scout from 10 to 7. Parameter count doesn't predict the direction — the biggest gainer and the biggest loser are both 70B models. The traces say why for one of them: the re-injected answers carry no record of what already failed, and llama3.3:70b burns all four lives re-submitting the identical wrong answer in every wiped run. With history intact it never does that.

The costs are measured too — winners cut turns-per-gate to 0.11–0.42× control, though one model paid 21× the output tokens for its depth. And one more thing ships as a flag rather than a finding: --look-gate, because telling a model to observe before answering did almost nothing (4.5 → 5.5 median, inside noise) while forcing it took the same model to 21+ with zero guesses — full brief in the data annex. This one still blows my mind a bit.

Cloud models will require an api key and, for now, a proxy in front (see the FAQ). Registering a run on the leaderboard will require git. Non-standard installs might need an .env tweak. But if you run models locally and just want to see how your local model fares? You don't need to make an account. You don't need api keys, git, or .env edits. Only some hardware capable of running an AI model and the LB docker compose file.

The wiping policy I ship demonstrably doesn't win everywhere. I'm looking forward to someone beating my attempt with their own harness — I have my own ideas for a few improvements here and there. A single lucky run can't top the board, either: rank is a conservative statistical bound (the one-sided 95% bootstrap lower confidence bound on median depth), so evidence moves you up, not variance.

Everything is public: repo, full briefs, pre-registrations with lock dates visible, raw run logs, and the board.

Repo: https://github.com/owl-fleet/labyrinth-bench · Board: https://labyrinthbench.ai · Data: https://labyrinthbench.ai/data