Validated a trust-propagation rule against 577k real transmission chains - κ 0.871 vs 0.331 between the human experts themselves [P]

Short version: I built a claim-level provenance framework for multi-agent systems that caps a chain’s trust at its weakest link. The obvious objection is that weakest-link is arbitrary — so I tried to falsify it against a dataset where humans have already done the labelling, at scale, for centuries.

Classical hadith scholarship graded chains of narrators and recorded verdicts. That’s a labelled corpus of transmission-chain trust judgments, 577,024 chains deep, produced independently of anything I built.

Results: Cohen’s κ 0.871 strict, 0.761 lenient, against the scholars’ own verdicts. For context, inter-scholar agreement on the same chains is κ 0.331 — the rule agrees with the tradition more than the tradition agrees with itself.

The honest read on that: it’s not evidence the rule is smart, it’s evidence the rule is consistent where humans were noisy. That’s still useful for a machine system, but it’s a weaker claim than the number looks. The strict/lenient gap is also where the interesting failure modes live, and I report the inconclusive parts of the evaluation in §8 at the same weight as the wins.

Paper: arXiv 2607.24117. Code Apache-2.0, pip install isnad. Would genuinely like to be told where the methodology is wrong.