While recent work often uses LLMs to extract causal graphs of the external world, we flip the approach: we use causal graphs to model LLM inference itself. This provides a transparent view of exactly how models perceive, organize, and connect high-level concepts to make a prediction.</p>\n<p>Our approach in brief:</p>\n<ul>\n<li>Concept Mapping: Discovers human-interpretable concepts and maps inputs to LLM-perceived concept states.</li>\n<li>MCMC-Inspired Augmentation: Generates chains of counterfactuals to expand sparse observational data.</li>\n<li>Causal Discovery: Runs σ-CG on this enriched data to yield stable, informative causal graphs.</li>\n</ul>\n<p>Does it work?<br>We evaluated this across 3 LLMs on three diverse tasks: disease diagnosis, sentiment analysis, and LLM-as-a-judge.</p>\n<ul>\n<li>The learned graphs showed high predictive fidelity and structural stability.</li>\n<li>They successfully capture meaningful dependencies that are actually consistent with the LLMs' internal reasoning.<br><a href=\"https://cdn-uploads.huggingface.co/production/uploads/62d6a0c18faee0ac953c51fa/WrG1AkTEmPinNAH270ZHz.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/62d6a0c18faee0ac953c51fa/WrG1AkTEmPinNAH270ZHz.png\" alt=\"IMG_0038\"></a></li>\n</ul>\n","updatedAt":"2026-06-08T06:41:35.200Z","author":{"_id":"62d6a0c18faee0ac953c51fa","avatarUrl":"/avatars/ca818cebdb089a8d853c5bc4d5e0987b.svg","fullname":"Nitay Calderon","name":"nitay","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.9012912511825562},"editors":["nitay"],"editorAvatarUrls":["/avatars/ca818cebdb089a8d853c5bc4d5e0987b.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.05972","authors":[{"_id":"6a25b8b3e4c258a029491ed0","name":"Nirit Nussbaum-Hoffer","hidden":false},{"_id":"6a25b8b3e4c258a029491ed1","name":"Nitay Calderon","hidden":false},{"_id":"6a25b8b3e4c258a029491ed2","name":"Liat Ein-Dor","hidden":false},{"_id":"6a25b8b3e4c258a029491ed3","name":"Roi Reichart","hidden":false}],"publishedAt":"2026-06-04T00:00:00.000Z","submittedOnDailyAt":"2026-06-08T00:00:00.000Z","title":"LLM Explainability with Counterfactual Chains and Causal Graphs","submittedOnDailyBy":{"_id":"62d6a0c18faee0ac953c51fa","avatarUrl":"/avatars/ca818cebdb089a8d853c5bc4d5e0987b.svg","isPro":false,"fullname":"Nitay Calderon","user":"nitay","type":"user","name":"nitay"},"summary":"Causal graphs provide a high-level language for making mechanisms transparent. Recent work uses Large Language Models (LLMs) to recover causal graphs of external-world processes. Instead, in this paper, we use causal graphs to model LLM inference itself, providing stakeholders with a transparent view of how the model perceives and organizes high-level concepts to produce a prediction. We propose a four-phase method for constructing such graphs. Given a target LLM and a set of textual examples, our method discovers class-discriminative, human-interpretable concepts and maps each input to LLM-perceived concept states. We then introduce an MCMC-inspired counterfactual augmentation procedure that expands the sparse observational data through chains of counterfactuals. This enables stable causal discovery with σ-CG, yielding informative, interpretable graphs. We apply our method to three LLMs across disease diagnosis, sentiment analysis, and LLM-as-a-judge classification tasks. We evaluate the learned graphs for predictive fidelity and structural stability, and the MCMC-inspired augmentation for convergence and downstream utility. Our results show that the discovered causal graphs capture meaningful dependencies consistent with LLMs' reasoning. Together, this paper provides a foundation for concept-level explainability of LLMs.","upvotes":8,"discussionId":"6a25b8b3e4c258a029491ed4","ai_summary":"Causal graphs are used to model large language model inference processes, enabling transparent visualization of how models perceive and organize high-level concepts for predictions through a four-phase method involving concept discovery, mapping, and MCMC-inspired counterfactual augmentation.","ai_keywords":["causal graphs","Large Language Models","causal discovery","counterfactual augmentation","MCMC-inspired","concept-level explainability","predictive fidelity","structural stability"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","organization":{"_id":"6393322be2364bc1eea56e45","name":"Technion","fullname":"Technion Israel institute of technology","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1670591001944-63926124526c29d5b5011374.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"657af98df81f6b44b87ef5f8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/yzM-IeHPSgHb2QFofq788.jpeg","isPro":false,"fullname":"lotem peled-cohen","user":"lotem-peledcohen","type":"user"},{"_id":"60d84af7eac5e05d4594f010","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60d84af7eac5e05d4594f010/KnGxUR7OUOAGg0S67tRaY.png","isPro":false,"fullname":"Alan Arazi","user":"alana89","type":"user"},{"_id":"625bc8dd41b529656cdb9055","avatarUrl":"/avatars/ce687dfb0404b6794adacb956207d843.svg","isPro":false,"fullname":"Dvir Lafer","user":"DvirL","type":"user"},{"_id":"63ff8fbfe7767a89533854c6","avatarUrl":"/avatars/e2094d362a7892bf26b991cab3bd711f.svg","isPro":false,"fullname":"G","user":"KerenGK","type":"user"},{"_id":"60ef001bed64a34082bfa0dd","avatarUrl":"/avatars/78e4daeac169edbf4dc42fbed9b50d59.svg","isPro":false,"fullname":"Omer Shubi","user":"scaperex","type":"user"},{"_id":"62d6a0c18faee0ac953c51fa","avatarUrl":"/avatars/ca818cebdb089a8d853c5bc4d5e0987b.svg","isPro":false,"fullname":"Nitay Calderon","user":"nitay","type":"user"},{"_id":"65bb578a81fe76980ea770c6","avatarUrl":"/avatars/16cfdbe893f068bcc76f0312d1a576ce.svg","isPro":false,"fullname":"Nirit Nussbaum Hoffer","user":"Nirit","type":"user"},{"_id":"6a267235a63bb2b72653a866","avatarUrl":"/avatars/4a67949bb9443076b5ac5d1cc89f97a0.svg","isPro":false,"fullname":"Gil davidovic","user":"gilnd99","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6393322be2364bc1eea56e45","name":"Technion","fullname":"Technion Israel institute of technology","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1670591001944-63926124526c29d5b5011374.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.05972.md"}">
LLM Explainability with Counterfactual Chains and Causal Graphs
Abstract
Causal graphs are used to model large language model inference processes, enabling transparent visualization of how models perceive and organize high-level concepts for predictions through a four-phase method involving concept discovery, mapping, and MCMC-inspired counterfactual augmentation.
Causal graphs provide a high-level language for making mechanisms transparent. Recent work uses Large Language Models (LLMs) to recover causal graphs of external-world processes. Instead, in this paper, we use causal graphs to model LLM inference itself, providing stakeholders with a transparent view of how the model perceives and organizes high-level concepts to produce a prediction. We propose a four-phase method for constructing such graphs. Given a target LLM and a set of textual examples, our method discovers class-discriminative, human-interpretable concepts and maps each input to LLM-perceived concept states. We then introduce an MCMC-inspired counterfactual augmentation procedure that expands the sparse observational data through chains of counterfactuals. This enables stable causal discovery with σ-CG, yielding informative, interpretable graphs. We apply our method to three LLMs across disease diagnosis, sentiment analysis, and LLM-as-a-judge classification tasks. We evaluate the learned graphs for predictive fidelity and structural stability, and the MCMC-inspired augmentation for convergence and downstream utility. Our results show that the discovered causal graphs capture meaningful dependencies consistent with LLMs' reasoning. Together, this paper provides a foundation for concept-level explainability of LLMs.
Community
While recent work often uses LLMs to extract causal graphs of the external world, we flip the approach: we use causal graphs to model LLM inference itself. This provides a transparent view of exactly how models perceive, organize, and connect high-level concepts to make a prediction.
Our approach in brief:
- Concept Mapping: Discovers human-interpretable concepts and maps inputs to LLM-perceived concept states.
- MCMC-Inspired Augmentation: Generates chains of counterfactuals to expand sparse observational data.
- Causal Discovery: Runs σ-CG on this enriched data to yield stable, informative causal graphs.
Does it work?
We evaluated this across 3 LLMs on three diverse tasks: disease diagnosis, sentiment analysis, and LLM-as-a-judge.
- The learned graphs showed high predictive fidelity and structural stability.
- They successfully capture meaningful dependencies that are actually consistent with the LLMs' internal reasoning.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.05972 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.05972 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.05972 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.