Shortcuts and Identifiability in Concept-based Models from a Neuro-Symbolic Lens

We thank the reviewer for finding the addressed problem important and for appreciating our theoretical contribution. Below, we reply to the points of their review.

Clarity of presentation

Thank you for the corrections and suggestions, we will implement them. We will be more explicit in Sections 1 and 2 about the problem we aim to solve, what we learn, and how we use supervision. We have also made use of the extra page to include a table of notation to help navigate the formalization. Below, we address different points separately. See also the reply to reviewer D2Sw.

Validity of assumptions

Assumptions 4.1 and 4.2 are mild and in fact they hold in all the tasks available in the largest benchmark of NeSy reasoning shortcuts [Bortolotti et al., 2024], including the ones we experiment with. They hold because, for each input image $\mathbf x$, it is possible to recover the ground-truth concepts in the image $\mathbf g$ and the corresponding labels $\mathbf y$.

We agree that it would be useful to lift these assumptions assumptions, but doing so is technically challenging. It is possible to relax Assumption 4.2 (i.e., labels cannot be entirely determined by the concepts), following [Marconato et al. 2023]; however, this can lead to a scenario where Theorem 4.9 no longer holds. Precisely, the absence of deterministic RSs does not inform us about whether non-deterministic RSs would appear or not [Marconato et al. 2023, Proposition 3].

We view extending the current theoretical framework to the case where the assumption can be violated as an exciting research direction.

Question: Intended semantics with fixed inference layer. Is it then implied that no permutations are allowed?

Yes, you are correct. When learned in an unsupervised fashion, concepts are “anonymous”: their meaning can only be figured out in a post-hoc manner.

If the inference layer is fixed, e.g., in the MNIST-Addition task, concepts are no longer anonymous, and permutations of the concepts (as in Eq. (5, left)) result in reasoning shortcuts, as in [Marconato et al. 2023]. These permutations can yield wrong results on other tasks. E.g., permuting the two values of the MNIST digits, while giving correct predictions in MNIST-Addition, would give incorrect results in “MNIST-Subtraction”.

Excluding the permutations when the inference layer is fixed is accounted for in the deterministic RSs count of Corollary 3.7, where only one solution (precisely the identity) has the intended semantics. We will make this more explicit in the text.

Question: How CBNMs are trained and the use of language-guided supervision

If concept supervision is not available at all, then it is unclear how to train CBNMs “independently” (that is, concept extractor first, inference layer later).

We train CBNMs with joint training [1] using only label supervision: both the concept extractor and the inference layer are trained to minimize the cross-entropy loss on the labels.

We do not consider weak supervision obtained from VLMs like CLIP, as done by recent architectures. While exploring JRSs in this setup is interesting future work, we stress that this setup does not resolve issues with concept quality. Although VLMs keep improving, recent works [2,3] show that the concepts they produce are not necessarily high quality even for widely used datasets (like CUB), meaning that it is unclear whether VLM-based concept supervision can be safely integrated to avoid joint reasoning shortcuts.

We will discuss this in the revised manuscript.

Question: Evaluation of the inference layer

The procedure we follow is described in Appendix A5. Intuitively, we assess whether the learned inference layer can handle the ground-truth concept annotations properly. For this to work, we first permute the g-t annotations to reflect the position of the concepts learned by the model, using the (inverse of the) permutation returned by Hungarian matching. We will clarify the intuition in the text.

---

References

[1] Koh et al. Concept Bottleneck Models. ICML 2020.

[2] Srivastava, Yan, Weng. VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance. NeurIPS 2024.

[3] Debole, Barbiero, Giannini, Passerini, Teso, Marconato. If Concept Bottlenecks are the Question, are Foundation Models the Answer? arXiv 2025.

[4] Zheng, Xie, Zhang. Nonparametric Identification of Latent Concepts. ICML 2025.

[5] Rajendran et al. From Causal to Concept-Based Representation Learning. NeurIPS 2024.

We thank the reviewer for the positive assessment of our contribution and for pointing out elements that need improvement. We will use the extra page allowed for the camera-ready to move the experiments on BDD-OIA in the main text, and to clarify the mathematical notation.

Notation is difficult

We amend this by adding a table of notation in Appendix A and complementing the text with additional explanations of the quantities in use, wherever necessary (See detailed comments below). We hope this improves the readability and are eager to iterate further based on the reviewer’s feedback.

L117 and L83

Thank you, we will improve the text based on your suggestions.

The importance of disentanglement for intended semantics

We agree with the reviewer that disentanglement is a central notion, and our definitions of intended semantics and joint reasoning shortcuts are based on it. We will complement the text around Def 3.3 specifying that Eq. (5, left) guarantees disentanglement of the learned concepts.

Mismatch between theory and practice

Definitions 3.3 and 3.4 in our theory are intentionally strict.

Our key insight is that the existence of JRSs implies that the learned concepts and inference layers can drastically differ from the ground-truth ones, while still yielding accurate (in fact, perfect) label predictions. This is precisely the issue we highlight in our experiments (Table 2 and Table 3), where models achieve high label accuracy despite significantly low F1(C) and F1(β) scores. These observations reflect a substantial mismatch between the learned and intended semantics, providing evidence that models are prone to learning JRSs.

In theory, when CBM concepts are even slightly entangled or do not perfectly match the ground-truth concepts, they do not satisfy Definition 3.3. Similarly, models that achieve nearly optimal label predictions but lack the intended semantics do not qualify as JRSs under Definition 3.4. However, we emphasize that our practical conclusion does not hinge on these definitions being satisfied exactly. Rather, the theory helps explain and anticipate the failure modes we observe: high task performance can mask severe representational mismatches. Conversely, when deterministic JRSs are mitigated (as shown in Table 1), models exhibit much closer alignment with the intended semantics, both in terms of concept quality and reasoning, which validates our theoretical framing and its practical relevance. Questions:

Question: L7: What are low-quality concepts?

We completely agree that concept quality is difficult to define and human-centric in general. As we mention in the introduction, at the bare minimum, “high quality” concepts ought to be interpretable and not compromise OOD generalization. This is precisely what our definitions of intended semantics and joint RSs are meant to capture.

Question: How conflicting are statements that CBMs can deal with ood (L20) even if they are identifying correlations (statistical machines)?

Good point! CBNMs are learned from observational data, just like other neural networks. Fitting such data is insufficient to guarantee any sort of OOD generalization when the target domain is different enough from the source domain. This applies equally well to the label predictions and to the concept predictions. If a CBM is affected by JRSs, however, we can construct target domains in which we know it won’t generalize. For instance, imagine it has learned concepts that do not identify the ground-truth ones (e.g., in MNIST-Addition, it conflates “0” and “2”). Then we can always design a ground-truth data-generating process whose concept distribution is the same as the source domain, but the inference layer is different (say, MNIST-Multiplication) such that the learned concepts will yield poor label predictions (all products involving “2” will yield “0” as output).

In short, while the absence of JRSs gives no guarantees that a CBM will generalize OOD, the presence of JRSs substantially increases the chances that it will not.

We will clarify what we mean by OOD in Section 3.

Question: L46: What exactly is meant by discrete concepts?

We mean categorical concepts, that is, random variables that can take only discrete values, as opposed to continuous concepts like angles or distances. We will be more precise in the text.

Question: How is Eq. (1) related to the usual factorization of CBMs?

It is identical, except we marginalize over the ground-truth concepts $g$. This enables us to reason in terms of ground-truth concepts in the derivations and it is justified by our choice of data generating process. The usual factorization of CBMs [1] can be reobtained considering that $p^* ( \mathbf G \mid \mathbf X ) p^* (\mathbf X ) = p^* ( \mathbf X \mid \mathbf G ) p^* (\mathbf G )$. In which case, we have:

$$p^{\ast}(\mathbf{X}, \mathbf{Y}) = \mathbb{E}_{\mathbf{g} \sim p^{\ast}(\mathbf{G} \mid \mathbf{X})} \left[ p^{\ast}(\mathbf{X}) \, p^{\ast}(\mathbf{Y} \mid \mathbf{g}; \mathrm{K}) \right]$$

Question: L81: Ground-truth concept vocabulary is assumed to be unique

Great point. We agree that there might be different choices of concept vocabulary (or domain $\mathcal G$) that all work fine for any given task, e.g., at different levels of abstraction. We do not make any assumptions about how this vocabulary is chosen.

To see this, fix any such vocabulary. This, in turn, it determines the ground-truth concept extractor $\mathbf{f}^\ast$ and inference function $\mathbf{\omega}^*$. Compatibly, our Theorem 3.9 explains when the NeSy-CBM will learn concept extractor $\mathbf f$ and an inference layer $\mathbf \omega$ equivalent to these.

Our analysis holds for any proper choice of ground-truth vocabulary (i.e., as long as it allows to predict the ground-truth label perfectly, and assumptions 4.1 and 4.2 hold). In practice, we expect that, with more fine-grained concepts joint reasoning shortcuts can increase, whereas they can decrease with coarser concepts.

We will discuss this interesting point in the paper.

Question: All the methods in Table 2 are deemed interpretable. Consequently, it should be possible to analyze the hypothesized simplicity bias. What rule has the SENN learned to solve the problem, and how does this result relate to the established theory?

SENNs provide local explanations for a given prediction, i.e., they do not provide an interpretable set of rules holding in the whole domain (i.e., the inference layer $\beta$ depends on the input). In contrast, NeSy-CBMs like DSL and DPL provide global explanations that we report in our experimental evaluation (see Appendix B).

We computed the concept confusion matrices for SENNs. We will report them in Appendix B, together with a comparison with the other approaches. In short, we didn’t observe any simplicity bias in SENNs. We hypothesize that this depends on both the flexibility in modelling local rules and on the reconstruction penalty applied to concepts during learning.

We will clarify this in the text.

Question: L341: "Our work partially fills this gap." Why partially? Again, the limitations should be clearly stated.

We partially fill the gap because we focus on NeSy-CBMs. While our results are likely to transfer to other NeSy models, doing so properly warrants further work.

Question: L357: Why is contrastive learning helpful? Once more, using the interpretability of the methods should provide an answer.

In our experiments (see Fig 3 for MNIST-SumParity and Fig 6 in the Appendix for CLEVR), contrastive learning reduces concept collapse but does not improve the other metrics. We remarked this point in the Conclusion (L327).

This is likely because contrastive losses encourage the concept extractor to assign similar (resp. dissimilar) concepts to similar (resp. dissimilar) inputs, i.e., the concept extractor can not collapse concepts together without increasing the contrastive loss. In practice, we would expect the same to happen also when a reconstruction term is in place, but this does not work as well (Fig 3) as it is more challenging to optimize. There are also theoretical reasons to believe contrastive learning should help with identifiability [2].

Some general notes not related to my assessment

We thank the reviewer for the detailed comments. We plan to include all the aforementioned changes in the final version of the manuscript.

---

References:

[1] Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., & Liang, P. Concept bottleneck models. In ICML 2020.

[2] Zimmermann, R. S., Sharma, Y., Schneider, S., Bethge, M., & Brendel, W. Contrastive learning inverts the data generating process. In ICML 2021.

We thank the reviewer for finding our theoretical framework novel and appreciating the depth of the analysis. Below, we address the points raised in their review.

Conditions under which the count of deterministic JRSs can be reduced. Question: How do different strategies affect this quantity?

Thank you for raising this important point. We agree that reducing the number of deterministic JRSs should be the primary goal and the starting point for designing new mitigation strategies. We stress that some of the existing strategies already allow for reducing the count in Thm 3.6; we analyze the impact of multi-task learning, concept supervision, knowledge distillation, and reconstruction explicitly in Appendix D. For example, introducing a reconstruction term avoids all those JRSs in which multiple concepts are collapsed together, leading to a smaller count. We will make sure to clarify this before the forward pointer to Appendix D in Section 4.

Most experiments are based on variants of MNIST

In order to assess the impact of joint reasoning shortcuts, we need concept-level annotations and prior knowledge (e.g., the rules of traffic in BDD-OIA), which are not commonly available. Our experiments rely on the most extensive existing benchmark for reasoning shortcuts [1], and, besides variants of MNIST-Addition, they also evaluate CLEVR and BDD-OIA. The former two are easy to control and evaluate, the latter is a challenging autonomous driving task with real-world images, four binary labels, and 21 binary concepts.

Comparison with other NeSy algorithms

We consider two NeSy baselines: DPL and DSL.

LTN is not designed for learning the knowledge from data. [Marconato et al. 2023] have already shown that NeSy predictors with fixed prior knowledge, including regular DPL, LTN, and the Semantic Loss [2], suffer from regular Reasoning Shortcuts.

Our goal is to go beyond the “fixed prior knowledge” setting and understand what happens in CBMs and NeSy predictors that learn the knowledge (or, more generally, the inference layer) from data. This is why our choice of competitors covers mainstream baselines like CBNMs, SENNs, and DSL, which all learn the inference layer. We also include a hybrid of DPL and DSL that does the same.

It is not straightforward to adapt LTN to learn the knowledge, as it would require handling the fuzzy logic in the inference layer, and designing appropriate data structures and losses for doing so is non-trivial. Doing so would be a standalone contribution.

Question: Distinction between vanilla shortcuts and joint reasoning shortcuts.

Regular shortcuts [Geirhos et al. 2020] can be understood as unintended input-to-label assignments that achieve high performance. E.g., the model might rely on confounding features (like watermarks) to output good predictions. They compromise generalization to domains where the confounders are not present.

Joint reasoning shortcuts are failure modes where either or both the concepts and the inference layer are not equivalent (in the sense we specify in Def. 3.4) to the ground-truth concepts and inference layer, and they can compromise interpretability and generalization.

The two issues are related but different. While shortcuts can induce JRSs, the converse is not true: a model can be affected by a JRS (i.e., it might have learned “bad” concepts or a “bad” inference layer) even if it does not rely on confounders. An example is reported in Fig 2. This also means that remedies to regular shortcuts do not necessarily resolve JRSs.

We will clarify this distinction in the Related Work.

Question: Why not using SotA vision models to extract concepts?

Good point. Our work focuses on “regular” CBMs and NeSy-CBMs that do not leverage VLM concepts, but it is true that recent CBM/NeSy architectures do use them [3,4]. However, although VLMs keep improving, recent works [5,6] show that the concepts they produce are not necessarily high quality even for widely used datasets (like CUB), meaning that VLMs may be affected by JRSs or similar issues too. We plan to extend our theory to this case in future work. We will include this note in the revised manuscript.

---

References

[1] Bortolotti, Marconato, Carraro, Morettin, van Krieken, Vergari, Teso, and Passerini. A neuro-symbolic benchmark suite for concept quality and reasoning shortcuts. NeurIPS 2024.

[2] Xu, Zhang, Friedman, Liang, van den Broeck. A semantic loss function for deep learning with symbolic knowledge. ICML 2018.

[3] Cunnington, D., Law, M., Lobo, J., & Russo, A. The role of foundation models in neuro-symbolic learning and reasoning. In International Conference on Neural-Symbolic Learning and Reasoning 2024.

[4] Oikarinen, T., Das, S., Nguyen, L., & Weng, L. Label-free Concept Bottleneck Models. ICLR 2023.

[5] Srivastava, Yan, Weng. VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance. NeurIPS 2024.

[6] Debole, Barbiero, Giannini, Passerini, Teso, Marconato. If Concept Bottlenecks are the Question, are Foundation Models the Answer? arXiv 2025.

We thank the reviewer for appreciating the novelty of our work and the thoroughness of the analysis.

Novelty of the work

We acknowledge that the work of Marconato et al. (2023) inspires ours, but we go beyond it by proposing a common framework to analyze NeSy-CBMs and CBNMs. We differ in that:

• Scope Marconato et al. study NeSy-CBMs where the inference layer is fixed. Lifting this constraint is technically challenging but it allows us to substantially generalize the notions of reasoning shortcuts and concept identifiability to CBNMs, a class of models that were never analyzed under this lens.
• Analysis How to properly learn concepts without concept supervision is an open question. Our results answer when to expect concepts to be learned correctly by avoiding joint reasoning shortcuts.
• Mitigation We test existing mitigation strategies for NeSy-CBMs with a fixed inference layer and consider two novel strategies: knowledge distillation and contrastive learning. Our results suggest that preventing JRSs is still an open question.

Clarity

We use the functional notation for ease of manipulation (see 100-104). This formalism also captures regular CBNMs. We consider CBNMs where concepts are modelled by applying a sigmoid (or a softmax) activation on input embeddings in the bottleneck, meaning they are conditionally independent given the input, as usual. In this setting, the CBNMs’ concept extractor is precisely a function that maps inputs to a simplex with one dimension per concept: the $j$-th coordinate is the conditional probability that the $j$-th concept appears in the input. The same setup can also express NeSy-CBMs like DPL and LTN, which consider the same activations on the bottleneck.

Question: NeSy is meant for reasoning, CBMs for classification

NeSy encompasses a wide family of approaches that integrate learning and reasoning, but reasoning shortcuts have so far only been studied in “NeSy predictors”, which happen to share much of the structure of CBNMs: both include a concept extractor mapping inputs to concepts and an inference layer, and both are trained via gradient descent. The main difference is the inference layer: in NeSy predictors, this is typically fixed and encodes known rules for deriving the label from the concepts (e.g., the rule of addition in MNIST-Addition); in CBNMs, it is typically a learned linear layer. This structural similarity is what enables us to extend RSs from NeSy predictors to CBNMs. At any rate, in NeSy, reasoning is always a means of computing predictions, as done in MNIST-Addition [4].

Question: NeSy-CBMs vs general NeSy models

We distinguish between NeSy-CBMs, which combine a concept extractor and an inference layer through a bottleneck of concept activations, and general NeSy models, which can have very different architectures and, in particular, may not model a bottleneck of latent concepts. Examples include grounding-specific Markov logic networks [1], SATNet [2], and the neural theorem prover [3].

Question: Why these baselines? There are only two CBMs, all others are NeSy

For both CBMs and NeSy-CBMs, we evaluated the most representative architectures:

• CBNMs are the most well-studied architecture, and they also serve as a recipe for other concept-based models [see references in lines 53, 54, and 55];
• SENNs are also well-known but leverage a slightly different architecture designed to learn concepts in an unsupervised manner.
• DSL learns both knowledge and concepts, and implements the inference step as a lookup table.
• a modified version of DeepProbLog that is similar to DSL but, like regular DeepProbLog, uses probabilistic logic for inference.

Question: If the CBNMs have no concept supervision, how are the concepts learned?

We train CBNMs using a regular cross-entropy loss on the labels, just like regular feed-forward neural networks. The concepts are treated as latent variables: the model will learn “concepts” that allow it to achieve low training loss. CBNMs also include a cross-entropy loss on the concept bottleneck, which we don't use unless otherwise specified.

Question: Why introducing the desiderata if you don’t evaluate them?

Thank you for raising this point. We want to clarify that the metrics we use to evaluate the accuracy of learned concepts (F1(C)) and of the learned inference layer (F1(beta)) take both desiderata into account.

• Disentanglement: F1(C) evaluates how close the map $\alpha$ (encoding the concept extractor) is to the identity, up to permutation and element-wise invertible transformations. To account for these symmetries, we apply the Hungarian matching algorithm to map learned to ground-truth concepts. This indirectly measures whether the learned concepts are disentangled.
• Generalization: We use the permutation and element-wise transformation obtained via Hungarian matching to evaluate whether the learned inference layer $\omega$ predicts the same labels as the ground-truth knowledge $\omega^*$. In summary, F1(\alpha) and F1(\beta) together tell us whether the two desiderata are respected.

We will clarify in the text how the experimental verification accounts for the two desiderata.

---

References

[1] Lippi and Frasconi. "Prediction of protein β-residue contacts by Markov logic networks with grounding-specific weights." Bioinformatics, 2009.

[2] Wang, Donti, Wilder, and Kolter. Satnet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. ICML 2019.

[3] Rocktäschel and Riedel. End-to-end differentiable proving. NeurIPS, 2017

[4] Manhaeve, R., Dumancic, S., Kimmig, A., Demeester, T., & De Raedt, L.. Deepproblog: Neural probabilistic logic programming. NeurIPS 2018