Hellooo I'm just a guy that learned about ML the past year pursuing an obsession: catch a model making things up the instant it commits to its first token. One forward pass, frozen model, before you see any text.

I fit four families of internal signals (attention shape, residual motion, readout geometry, confidence — 29 total), let an honest selector pick one per model, and pre-registered everything before the data existed. Twice.

Run 1 — 2 tasks, 10 models (20 deployments):

- Geometry-only detector clears its pre-registered bar: 18/20 (needed ≥17). Bolt on the model's own confidence and you get the same 18/20 — same two misses, zero rescued — so the stricter "confidence covers more" claim (bar ≥19) is falsified. The model's confidence is redundant with the geometry.

- No universal best signal — 12 different signals win across the 18 working cases, so the top pick is per model+task. But a universal floor: one fixed combo, calibrated on nine models and tested on the tenth — each pass excludes one model and scores that excluded model, rotating through all ten. Still beats chance on 9/10 (ANLI) and 10/10 (TriviaQA).

Run 2 — a separate 6-task extension, incl. HaluEval-QA (spotting hallucinated answers):

- Calibrate per model on the new task: 10/10 beat chance.

- The one drop-in detector I pre-registered — fixed signal, fixed sign, applied blind — 6/10. The four misses don't lose the signal; they read it backwards (AUROC as low as 0.17). Same geometry, inverted sign.

It's a universal fitting procedure and an above-chance floor — not one shippable detector. Even the sign has to be set per model.

Every registered score matrix is public; both pre-registered verdicts re-derive from them with zero model inference (python stage_b/verify_endpoints.py, no GPU).

Please find the hole.

Paper: furnace.baby/cc-paper · Code: github.com/flowstyleliving/commit-confluence

---

FAQ

"Isn't hallucination just a quantization artifact? You're running 4-bit models."

Fair worry, so I pre-registered a falsifier for it: run the same models at nf4 → int8 → bf16 → fp32 and check whether the signal degrades as precision drops. It didn't. The detector's numbers are precision-invariant on the models where it works — so it's measuring real computation, not rounding noise. (int8 wobble on the small models washes out by 32B.)

"How do I know your research itself isn't a hallucination? AI-assisted, big claims."

Everything's checkable without trusting me. Every registered score matrix is public, and both pre-registered verdicts re-derive from them with zero model inference — python stage_b/verify_endpoints.py, no GPU, reproduces the summary byte-identically. The pre-registration was frozen (hashed) before the data existed, on a fresh disjoint seed, with shuffled-label controls, and went through four adversarial review passes. If I'm wrong, the matrices will show it.

"You report a 7/20 FAIL but say the signal was fine. Which is it? And did you just re-run a tiny sample?"

Neither cheap out. Two of Run 2's six tasks were straight replications of the sealed pair (ANLI and TriviaQA), but on fresh, disjoint data at five times the sample — n=200 → n=1000 — so this isn't a small-n fluke. The raw geometry held: 18 of 18 deployable. The 7/20 comes from a pre-registered zero-error-budget rule that voids an entire task's cells if a few rows ever fail to commit to a clean YES/NO first token. Here's the honest catch: the detector reads the commit-moment, so it assumes answer-first output — if a model emits a newline or starts chain-of-thought before answering, that row counts as a non-commit even if its eventual answer is correct. A literal handful of such rows out of 1,000 (one model occasionally reasoned instead of judging) is what forced the FAIL. That rule was frozen before I saw the data, so I can't soften it now — the honest headline is the number the rule produces, 7/20, with the raw geometry shown beside it. A future pre-registered amendment will separate a one-off blip from genuine off-task reasoning; I'd genuinely like eyes on whether that answer-first constraint is the right call or a hidden thumb on the scale.

"Isn't the 6/10 transfer failure just noise?"

No — that's the interesting part. The four misses don't lose the signal, they read it backwards: AUROC as low as 0.17, i.e. confidently wrong. Each model, calibrated on its own data, picks the opposite sign. So a single frozen detector can't be dropped across models blind. Per-model, it's 10/10. The failure is specifically about fixed-sign transfer, not signal.

"What about the tasks I don't see bars for?"

Three of Run 2's six tasks were exploratory — ANLI Round 2 (a deliberately harder NLI round), plus HaluEval's dialogue and summarization splits (hallucination detection in two formats beyond the QA one). By registration these carry no pass/fail bar and can't move a headline claim. But for what it's worth, under per-model calibration the signal stayed above chance on every one — 25 of 25 model-task cells, all confidence intervals clearing 0.5. I'm not counting them toward any verdict; I'm just not hiding them.

"Why no single accuracy number across the whole thing?"

Because there isn't an honest one. It's three experiments with different constructs and different pre-registered bars (18/20 needing ≥17; 10/10 vs 6/10 needing ≥8; a 7/20 gate rule). Averaging them would be a category error — a single number would look cleaner and mean less. The through-line isn't a score, it's a claim: no universal detector, but a universal floor that partially transfers.

edit: took isparavanje 's feedback and changed intro/lack of pedigree :)