Operadic consistency: a label-free signal for compositional reasoning failures in LLMs
Nathaniel Bottman, Yinhong Liu, Kyle Richardson
Operadic consistency (OC) is a label-free confidence signal that checks whether an LLM's direct answer to a multi-hop question agrees with the answer it would produce by explicitly composing the sub-questions in a stated decomposition of that query. Across 12 models and 4 multi-hop QA benchmarks, OC correlates with accuracy at Pearson r ∈ [0.86, 0.94] uniformly — outperforming chain-of-thought self-consistency, which collapses to r ≈ 0.45 on MuSiQue and StrategyQA. At the per-question level, OC adds statistically significant information beyond both CoT-SC and semantic entropy, and delivers AUARC lifts of +0.086–0.096 and AUROC lifts of +0.092–0.164 over a tuned CoT-SC baseline at equal sampling cost (K=3).
The repos linking to this paper are all paper-tracking aggregators with no implementation work — no one has built OC into an inference pipeline, eval harness, or selective-prediction system yet. Zero citations as of publication. The signal is computationally cheap to implement (requires only decomposition + two forward passes) and the claims are strong enough to be worth testing in any system that runs multi-hop reasoning at scale, but production traction is zero right now.
Detecting LLM reasoning failures at inference time without ground-truth labels has motivated a wide range of confidence baselines, including self-consistency, semantic entropy, and P(True), built on within-question sampling and self-evaluation. Operad theory, the formalism for systems built by iterated substitution, suggests a complementary diagnostic: a model's direct answer to a compositional query should agree with the answer it produces by composing a stated decomposition of the same query. We instantiate this idea as operadic consistency (OC), a per-question signal. Across twelve instruction-tuned LLMs (4B to 671B parameters, open-weights and closed-source) on four multi-hop QA datasets, OC is strongly correlated with accuracy on every dataset (Pearson $r \in [0.86, 0.94]$, all $p \leq 0.0004$), and is the only signal we evaluate with $r \geq 0.85$ uniformly across all four datasets. Chain-of-thought self-consistency (CoT-SC; Wang et al., 2023) matches OC on HotpotQA and DROP ($r = 0.93, 0.87$) but drops to $r \approx 0.45$ on MuSiQue and StrategyQA. At the per-question level, OC contributes information beyond CoT-SC and semantic entropy on every dataset (cluster-robust $p \leq 10^{-16}$ for the OC coefficient), and the conclusion is robust to additionally controlling for constructed decomposition-aware baselines ($p \leq 10^{-13}$). The same signal yields selective-prediction improvements (accuracy at fixed coverage) over a tuned CoT-SC baseline at the equal-cost $K = 3$ budget (AUARC lifts of +0.086 to +0.096 and AUROC lifts of +0.092 to +0.164; 95% CIs exclude zero on every cell). On five frontier thinking models, where the decomposition is extracted from the model's own chain of thought, the same equal-cost comparison gives positive selective-prediction point-estimate lift on all 16 (dataset, budget, metric) cells tested, with 95% CIs excluding zero on 12 of the 16.