Controlling Information Leakage in Concept Bottleneck Models with Trees

(continuing the previous response...)

Identifying such instances can then assist our decision-making when deriving concept-based explanations per decision path. This corresponds to the analysis of section 5.3. For better clarity, we update this section by defining two scenarios:

• If a group (leaf node) cannot be further split using leakage, i.e. a sub-tree is not found: This shows that the particular group is not affected by information leakage, which is desirable. In addition, if the task accuracy for this particular group is high, we may consider this an ideal classification example, because the available concept annotations are sufficient for an accurate distinction. The fact that we can identify and isolate such groups is highlighted as a key advantage of our work. Unlike a purely soft CBM, leakage will not impact these groups, and thus the concept explanations for those groups are both leakage-free and accurate. If, on the other hand, the task accuracy is not sufficient, we may flag this group to an expert to either annotate additional concepts or perform an intervention (future work).

• If a group (leaf node) can be further split using leakage, i.e. a sub-tree is found: Then this group can be flagged for additional analysis. We describe a detailed case study for the Woodpecker example in page 10, paragraph: “Our tree-structure allows for meaningful group-specific explanations”. We urge the user to investigate if this difference in likelihoods (leakage) might be intuitive or not, similar to the cat-dog example we described in the beginning. Then the user has the following options: a) rely solely on the decision process of the global tree to derive a perfectly understandable, leak-free explanation, b) extend the decision process with the sub-tree of MCBM-Seq if this likelihood difference seems intuitive, c) use MCBM-Joint’s less intuitive probabilities for maximum accuracy, d) flag this group to an expert to either annotate additional concepts or perform an intervention (future work).

In conclusion, identifying leakage is useful because it provides these analysis tools to a decision maker when deriving explanations, which are not available to a standard CBM. We currently develop future work providing interventions and concept discovery strategies specifically when leakage is observed. However, future leakage mitigation strategies first require a method that controls and inspects leakage for specific sub-groups, thus we believe this work may be used as a very useful analysis tool while not sacrificing the task performance of a standard CBM (the task performance is comparable).

Question: Why do the authors focus on decision trees (e.g. as opposed to a linear model CBM)? The paper only compares against Entropy Net & a black-box baseline.

Response: We focus on decision trees because they offer us the advantage of inspecting leakage, which is not provided by a linear model or a neural network. More specifically, they allow us to inspect leakage defined in Eq. (4) using our formulation of Eq. (5) and Appendix A.2. Quantifying Leakage is not straightforward using a linear model or the Entropy-Net, i.e. it is not evident how the mutual information $I(y; \hat{c}|c)$ in Eq. (4) could be approximated using such models in a similarly efficient way. Moreover, trees allow us to inspect leakage for specific groups and derive group-specific explanations in the form of decision paths by controlling the “minimum samples per leaf (msl)” constraint, as highlighted in section 5.3. In contrast, linear models as well as the Entropy-Net model only allow us to form either instance-specific or class-specific explanations. In terms of task and explanation accuracy, we compare our method directly with the Entropy-Net model because it is currently the state-of-the art method for concept-based explanations, outperforming simple linear models [2].

[2] Barbiero, P., Ciravegna, G., Giannini, F., Lió, P., Gori, M., & Melacci, S. (2022). Entropy-Based Logic Explanations of Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence, 36(6), 6046-6054.

We sincerely thank the reviewer for reading our work and providing comments. We will plan to respond to the highlighted weaknesses and questions in a sequential manner.

Weakness 1: "the main issue seems to be a lack of rationale for definition 3.1, which forms the basis for the paper. ... more reliable."

Response: First, we would like to clarify that our method is designed especially for categorical concepts, since the vast majority of real-world concept-annotated datasets have categorical and not continuous concepts, including CUB and MIMIC which are explored in this paper. Morpho-MNIST is an exception and has continuous concepts, which we could have indeed leveraged differently. However, in our case this particular dataset is used as a toy example to demonstrate how are method works. Thus, we perform the binning operation to create the “small”, “medium” and “large” categorical concepts.

We would also like to clarify again the definition of a “hard” and a “soft” categorical concept, following the existing formulation of [1]. A hard categorical concept is a boolean concept, where “1” indicates its presence and “0” indicates its absence. A soft categorical concept refers to a concept probability, indicating the confidence of the concept predictor about its presence. Thus, a “soft” concept is different from a continuous concept.

[1] Havasi, M., Parbhoo, S., & Doshi-Velez, F. (2022). Addressing leakage in concept bottleneck models. Advances in Neural Information Processing Systems, 35, 23386-23397

Weakness 2: "It is also unclear how quantifying this information leakage is useful....metrics (Table 1)"

Response: We believe that quantifying leakage is a very important tool for analyzing the interpretability of Concept Bottleneck Models.

To make this more clear, let us first adapt a practical example of leakage in CBMs described in [1]. Assume we have a CBM performing animal classification. When the concept predictor recognizes an image as a dog, it may predict a slightly higher likelihood to the concept ‘tail’ compared to an image of a cat. Let us assume for example that the concept predictor gives a likelihood around 0.8 to the majority of dogs for the concept ‘tail’, and around 0.6 for the majority of cats. The ground truth concept ‘tail’ for both classes is 1 (meaning that both animals have a tail). Even though cats and dogs are indistinguishable based on this ground truth concept, the label predictor may still compare the two soft concept probabilities (0.6 and 0.8) and make an accurate classification. However, is this concept-based explanation useful? The label predictor relies on a difference in likelihoods that may (or may not) be interpretable to a human. For example, it might be the case that in most images of cats their tail is not clearly shown due to the orientation of the cat, whereas in most images of dogs the tail is clearly visible. This phenomenon is called “leakage” because unintended information from the input is captured in the concept likelihood (the orientation of cats and dogs). Thus, the concept predictor assigns a higher likelihood to dogs having this concept. However, this may be just an assumption. Regardless of whether this difference is intuitive or not, we believe it is crucial to first find a systematic way to isolate instances where the label predictor takes advantage of such differences, in order for us to then proceed to further analysis. In our paper, a similar case study is performed for the Woodpecker example of Figure 2.

Thus, our work first provides a systematic way to identify those instances affected by leakage using our tree structure. Specifically, we first train a tree using only the ground truth concepts, which we name the “global tree” (step 1 in Figure 1). Each leaf node in the global tree corresponds to a subset of examples that are indistinguishable based on their ground truth concepts. For example, if we observe the leftmost decision path in the toy MNIST example of Figure 3, there is a leaf node corresponding to group of 926 digits (278 “6”s, 33 “8”s and 615 “9”s), and all these digits have small length and small thickness. We see that this group was not extended with leakage after performing our algorithm, i.e. a sub-tree was not found (step 2 in Figure 1). This means that the label predictor could not take advantage of any differences in the likelihood of concepts “length:small” and “thickness:small” to further split this group. On the other hand, the third decision path from the left, which corresponds to the group of digits having small length and large thickness, was extended with two decision rules corresponding to differences in concept likelihoods (leakage): “length:small < 0.8” and “thickness:large < 0.8”.

(The response continues in the next comment)

(continuing the previous response...)

• “Addressing leakage in concept bottleneck models. NeurIPS 2022” [3]. This work is closer to our method, in the sense that a) it uses scalar-valued concepts and b) specifically addresses leakage. However, as we mention in our related work, “Havasi et al. (2022b) tackle missing information with a side channel and an auto-regressive concept predictor, but these approaches struggle with interpretability and disentanglement of residual information (Zabounidis et al., 2023)”. In more detail, all works that use a residual layer or side-channel (including this one) aim to let missing concept information pass directly from inputs to targets, letting the ground truth concept representations intact and not influenced by leakage. Yet, Zabounidis et al., 2023 highlight that the residual information is not guaranteed to capture this intended missing information, and the two representations may be entangled. We argue that the lack of transparency in the residual channel does not make these methods convincing enough for the specific problem.

In conclusion, while we understand that the three-phase method may seem more complex, we argue that it is more effective and provides some novel advantages compared to existing methods, such as the ability to perform group-specific leakage examination in the form of decision paths and to identify the exact decision rules based on leakage. Thus, our work cannot be quantitatively compared with these previous works. Moreover, we argue that the three-phase method is not that complex in practice, because essentially it only involves the training of one global tree and individual sub-trees for the leaf nodes of this tree, along with an independently trained concept predictor like in all CBMs.

We sincerely thank the reviewer for reading our work and providing comments. We will plan to respond to the highlighted weaknesses and questions in a sequential manner.

Weaknesses 1-2 and Question 1: "(Major) There might be, in my opinion, a potential discrepancy between the information leakage metric defined (Definition 3.1) ... (Major) Related to the above point, it seems that the information leakage problem addressed in this paper differs slightly from that studied in existing works..."

Response: Thank you for this great question. You are correct. The definition 3.1 we provide is general and indeed refers to $\hat{C}$ and $C$ being random variables. Indeed, we calculate the metric per leaf node and per decision split by conditioning the random values $\hat{C}$ and $C$ appropriately. Refer to Appendix A.2, page 14 for the full description. We indeed denote $I_{Leakage}$ as $I_{Leakage}( \hat{c}_k )$ in Eq. 8, showing that this is the Information leakage induced by the specific split. We do not estimate the mutual information of definition 3.1 specifically, since we observed that it is more useful in practice to quantify leakage in specific groups rather than providing a global leakage estimate, as shown in the per-path analysis of section 5.3. We believe that conditioning on the random variables does not introduce a discrepancy, but rather makes this information metric more specific.

Weaknesses 3-4 and Question 3: "The work has not been compared to state-of-the-art methods for addressing information leakage, such as CEM [1], PCBM [2] and [3];"

Response: The three cited papers are included in our section 2 “Related Work”, with a short justification of why we believe they do not sufficiently address information leakage. Here, we elaborate for each one:

• “Concept Embedding Models. NeurIPS 2022” [1]: This paper introduces the idea of a “concept embedding”, which was later adopted by more CBM papers such as PCBM [2]. In our work, we use scalar-valued concepts instead of concept embeddings, following the original CBM paper*. While concept embeddings achieve excellent task performance because they lead to more expressive concept representations, the authors do not comment on information leakage, i.e. they do not provide a justification about whether these concept embeddings also capture unintended (“leaked”) information from the inputs to improve the task performance. Instead, they propose the Concept Alignment Score (CAS) to measure how much learnt concept embeddings can be trusted as faithful representations of their ground truth concept labels. Their intuition is that clustering samples based on a faithful concept embedding would result in coherent clusters. While they show that CEMs achieve high CAS scores, we argue that their method may not be sufficiently interpretable because: i) They achieve CAS scores of around 80% in certain datasets, such as CUB and CelebA, which may imply the presence of leakage. ii) This approach does not indicate which subsets may suffer from the imperfect concept alignment, or how does this imperfection affect concept-based explanations. In contrast, our tree-based method allows for group-specific leakage examination in the form of decision paths, and gives the exact decision rules based on leakage. iii) The information captured in a high-dimensional concept embedding is unintuitive compared to a scalar-valued concept, which directly represents the probability (confidence) of the concept predictor.

• “Post-hoc Concept Bottleneck Models. NeurIPS 2022” [2]. Similar to CEMs, this work uses concept embeddings but in the form of concept activation vectors (CAVs). Also, the authors do not address the issue of concept faithfulness or that of leakage, since they rely on multi-modal models to learn concepts that may be not annotated. While they effectively deal with the problem of missing concept annotations, their concept quality and faithfulness relies on the fidelity of their multimodal model.

(The response continues in the next comment)

I admire the authors' effort to address my and other reviewer's concerns. All my questions have been answered, and some of the issues raised have been addressed. However, the work still lacks a direct empirical comparison to existing works for controlling information leakage (e.g. PCBM and CEM). At the same time, I still feel the three-stages unnecessarily heavy, especially that there is currently no clear (empirical) evidence showing the advantages against existing methods. All these make me uncertain whether the work is borderline acceptable or not.

I thereby maintain my current scores at this stage, and will determine my final score after discussing with other reviewers. My current score should be interpreted as a 5.5.

Once again, thank your for your noticeable efforts in improving the work. Wish you the best of luck with the submission.

Weakness 2 and Question 2: From Table 2, I do not see how MCBM is better. It has worser task accuracy than EntropyNet, but perhaps lower leakage? (which is not apparent). Given that their method has advantages when the number of concepts is small, their evaluation too should bring out more readily. From the argument in L460-462, decision trees with soft concept scores must have led to more leakage (or poor fidelity?), but that's not the case in Table 1. Please explain.

Response: We apologize if the interpretation of Tables 1 and 2 is not immediately clear. We will attempt to describe them in this response in an intuitive and detailed manner.

The purpose of Table 2 is to show that MCBM-Seq is comparable in task performance compared to existing CBMs, or lower in performance compared to Joint CBMs which however suffer from information leakage (refer to the work of [3]) and Black-Box neural networks which are inherently uninterpretable. The Table does not show the advantage of our method by itself, but shows how it performs compared to standard methods in order for our analysis to be complete.

The important take-away from Table 2 when looking at the numbers is that the relationship of task accuracies in CBM modes is roughly the following: Hard, Independent < MCBM-Seq <= Sequential < MCBM-Joint (for small $\lambda_C$) < Joint (for small $\lambda_C$) < Black-Box. In contrast, in terms of leakage, they follow the opposite trend: Hard, Independent (No Leakage by definition [3,4]) > MCBM-Seq (has leakage, but this is inspectable and controllable by the decision maker) > Sequential CBM (has leakage according to [3,4], which is uninspectable, uncontrollable and affects all samples as stated in L460-462) > MCBM-Joint (has more leakage but this is again controllable) > Joint-CBM (has a lot of uncontrollable leakage, based on [3,4]). The two reverse trends show the trade-offs of CBMs.

Table 1 was constructed to highlight the advantages of MCBM-Seq. First, the column named “Leakage Inspection” emphasizes that MCBM-Seq is the only method that allows for Leakage Inspection, which is the novel property we aim to introduce in this work. The existing completely soft sequential CBMs, regardless of their type of label predictor (Entropy-Net, Simple Decision Tree), typically have leakage, as indicated in previous works [3, 4] but this leakage is neither easily inspectable nor controlled, which motivated our work. We also provide an intuition for this claim with a practical example in Appendix A.3, page 14. Hard and Independent CBMs are not included in this table because they do not have leakage, so leakage inspection is not applicable. Secondly, the table reveals another advantage of MCBM-Seq when compared explicitly with a Purely Soft Sequential CBM using an Entropy-Net as a label predictor, which is that MCBM-Seq also achieves higher Explanation Accuracy and does not raise Fidelity issues, as explained in lines 416-426. This second advantage does not hold when compared to purely soft sequential CBMs using traditional decision trees, but the first main advantage of leakage inspection still remains.

The reasons why our leakage inspection is useful are those highlighted in section 5.3: a) we can analyze our model for specific decision paths (groups), and thus b) we can derive more meaningful group-specific explanations, since bi) the decision-maker has the flexibility to control the concept explanation based on the length of the decision path (lines 515-518) and bii) leakage will not impact all decision-making paths in a mixed CBM (lines 518-519).

Based on the above clarifications and going back to Question 2, the answer is that decision trees with purely soft concept scores do not raise fidelity issues but they also do not allow for Leakage Inspection, unlike our new MCBM-Seq method. We again encourage the reviewer to refer to Appendix A.3, page 14 which shows a decision tree with purely soft scores and how its inefficiency motivated us to develop MCBM-Seq.

In conclusion, the argument of our work is the following: If MCBM-Seq has a task accuracy between those of a Hard and a Sequential CBM (Table 2) but is superior in terms of explainability due to its leakage inspection property, which is shown in Table 1 and section 5.3 (pages 9 and 10), then we believe it is a useful training method for CBMs. We hope this clarifies the use of our Tables.

[3] Mahinpei, A., Clark, J., Lage, I., Doshi-Velez, F., & Pan, W. (2021). Promises and Pitfalls of Black-Box Concept Learning Models. ArXiv, abs/2106.13314.

[4] Addressing leakage in concept bottleneck models. NeurIPS 2022

Question 1: Please explain the metrics at length. Concept accuracy, fidelity and explanation accuracy. I believe explanation accuracy is mentioned but never used (in which case it can be dropped).

Response: As stated in lines 383-388, these are all metrics introduced in existing works. More specifically:

• The concept accuracy introduced in [1] refers to the accuracy of the concept predictor when predicting all categorical concepts (refer to Step 1 of the method, Figure 1, page 2). Since each concept is predicted independently, the total concept accuracy reported is the aggregation of all predicted concepts. We can observe in Table 2, page 9, that the concept accuracy is the same for all CBM types where the concept predictor is trained independently. For Joint CBMs, we observe the trade-off between concept and task accuracy based on the parameter $\lambda_C$. This is explained in detail in the original CBM work of [1]. The Black-Box model does not have a concept accuracy since it does not use concept supervision (inputs are directly mapped to targets in a single neural network).

• The task accuracy refers to the accuracy of the label predictor. For example, the task accuracy of the Hard CBM refers to the accuracy of the global decision tree (refer to Step 1 of the method, Figure 1, page 2), while the task accuracy of MCBM-Seq refers to the accuracy of the same tree but extended with all potential sub-trees (refer to Step 3 of the method, Figure 1, page 2).

• The explanation accuracy introduced in [2] measures the task performance of a model when using its extracted explanation formulas instead of the model’s predictions, which would correspond to the task accuracy. In case of Entropy-Net, their method approximates the predictions of a neural network using simplified logic rules. Thus, the task accuracy is always greater or equal than the explanation accuracy when using Entropy-Net, as can be observed from Table 1, page 9 as well as the original results of the paper [2]. However, in decision trees, the explanation accuracy is the same as the task accuracy by default, since a decision path from the root to a leaf node serves as both the classifier and the explanation.

• The fidelity of an explanation is also introduced in [2] and measures how well the extracted explanation matches the predictions obtained using the explainer. In practice, the authors calculate it as the accuracy score between the labels predicted from the neural network and the labels predicted from the logic rules that serve as explanations. In decision trees, we denote the fidelity score as 100% to indicate that no fidelity considerations occur, since a decision path from the root to a leaf node serves as both the classifier and the explanation.

[1] Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., & Liang, P. (2020, November). Concept bottleneck models. In International conference on machine learning (pp. 5338-5348). PMLR.

[2] Barbiero, P., Ciravegna, G., Giannini, F., Lió, P., Gori, M., & Melacci, S. (2022). Entropy-Based Logic Explanations of Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence, 36(6), 6046-6054.

We sincerely thank the reviewer for reading our work and providing comments. We will plan to respond to the highlighted weaknesses and questions.

Weakness 1: I expect to see stronger evaluation of MCBM's utility for information leakage, and their implications in improving task accuracy or explanation quality. Perhaps through human-studies or mining new concepts on one of the tasks such as CUB.

Response: We believe that section 5.3 focuses specifically on the implications of leakage on a) task accuracy and b) explanation quality, and c) it also provides a detailed case study on the CUB dataset. More specifically:

• Impact on Task Accuracy: As explained in paragraph “Our method enables inspecting Information Leakage per decision path”, the task accuracy increased in all three decision paths that were affected by leakage, for the reduced Morpho-MNIST example of Fig. 3. This increase shows exactly the impact of leakage, i.e. the task accuracy increases when a decision path of the Hard CBM is extended with additional decision splits relying on leakage. These numbers and paths are shown in Fig. 5 and Table 3. As an example, consider the decision path with number 14 in Figure 5. In the global tree of the Hard CBM, the decision path would end at the light gray node, which has 571 samples all having large length, large width and medium thickness. Due to majority voting, the hard CBM classifies all those digits as “8”s. As shown in Table 3, this leads to an accuracy of 44.91% for this path with number 14. When MCBM-Seq is then applied to investigate if this group is prone to leakage, the algorithm discovers that the group can be further split using the soft concept representation of “length:large”. Specifically, it is observed that the concept predictor is less than 90% confident that the majority of “6”s have a large length, while it is more than 90% confident that the majority of “8”s have this attribute. Thus, the tree-based label predictor takes advantage of this difference in predicted likelihoods, and is able to further split the group of 571 samples into two new groups using the decision rule “length::large <= 0.9”. With this new “leaky” split, the task accuracy improves to 57.70% for the path, as shown in Table 3.

• Impact on Explanation Quality: As explained in paragraph “Our tree-structure allows for meaningful group-specific explanations”, identifying groups affected by leakage can assist our decision-making when deriving concept-based explanations per decision path. For better clarity, we update this paragraph by defining two scenarios:

  • If a group (leaf node) cannot be further split using leakage, i.e. a sub-tree is not found: This shows that the particular group is not affected by information leakage, which is desirable. In addition, if the task accuracy for this particular group is high, we may consider this an ideal classification setting, because the available concept annotations are sufficient for an accurate distinction. The fact that we can identify and isolate such groups is highlighted as a key advantage of our work. Unlike a purely soft CBM, leakage will not impact these groups, and thus the concept explanations for those groups are both leakage-free and accurate. If, on the other hand, the task accuracy of the group is not sufficient, we may flag this group to an expert to either annotate additional concepts or perform an intervention (future work).

  • If a group (leaf node) can be further split using leakage, i.e. a sub-tree is found: Then this group can be flagged for additional analysis. The user has the following options: a) rely solely on the decision process of the global tree to derive a perfectly understandable, leakage-free explanation. In the example of the decision path with number 14 described above, this means terminating the explanation at the light gray node and characterizing all 571 samples as “8”s. b) extend the decision process with the sub-tree of MCBM-Seq if this likelihood difference seems intuitive. In the same example, this means incorporating the path extension into our explanation, using the decision rule “length::large <= 0.9”. c) use MCBM-Joint’s less intuitive probabilities for maximum accuracy, d) flag this group to an expert to either annotate additional concepts or perform an intervention (future work).

• Case study on a real-world setting: We also provided a detailed case study in Appendix A.10, page 24 on the CUB dataset, for distinguishing between the Red Bellied and the Red Headed Woodpeckers. Figure 15 shows the complete decision path (explanation) of the case study both before and after incorporating the path extension, and also highlights that the test accuracy again increases due to leakage for this path.

Are there any specific additional experiments you would like to see? We would be happy to consider them.

Question 3: When the concept scores are mixed, I do not see why only the concepts on the path are softened. What happens when all the concepts are softened (when expanding the tree) or only the leaf concept is softened.

Response: This is a very important detail. We specifically refer to this in lines 258-268. If all concepts are softened, the mutual information for the leaf node: $I(y;\hat{c}_k|c_s)$ would not be satisfied, thus our Definition 3.1 of Leakage in line 176 would not hold since Equation (8) in the Appendix would be incorrect. Refer to Appendix A.2, page 14 for further details. Intuitively, the reason is that we treat Information Leakage as “The amount of unintended information that is used to predict label $y$ with soft concepts $\hat{c}$ that is not present in hard representation $c$. Thus, we need to make sure first that all samples of a group have the hard concept $c$, in order to then investigate if the soft representation of this concept provides extra, leaky information. We cannot investigate the impact of a soft concept in a sample if the sample does not possess the hard concept in the first place.

The concepts appearing in the decision paths are guaranteed to be shared by all samples in the path. Taking the leftmost path in Figure 3, page 6 as an example, the 926 digits ending up in the green leaf node all have small length and small thickness, if we follow their decision path from the root. We then search if the soft representation of any of those two concepts leads to leakage, which does not happen in the particular path because there no nodes with light gray color were found. Concepts not appearing in the path may not be shared by all digits in the group.

Question 4: Please comment on the choice of hyperparam. Table 2 lists out numbers for various hparam $\lambda_C$, but how is it picked?

Response: According to the original CBM work [1], the parameter $\lambda_C$ controls the trade-off between task and concept accuracy in Eq. (3) line 165, which is also evidenced in our results of Table 2 (smaller values of this parameter increase the task accuracy and reduce the concept accuracy, while the opposite holds for large values). For computational reasons, we tested one very small value $\lambda_C = 0.1$, one very large value $\lambda_C = 100$ and one in the middle, to show how the metrics change in the full range of values for this parameter.

[1] Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., & Liang, P. (2020, November). Concept bottleneck models. In International conference on machine learning (pp. 5338-5348). PMLR.

We sincerely thank the reviewer for reading our work and providing comments. We plan to respond to all highlighted points.

Weakness 1: Moreover, it is unclear how MCBMs compare to and how they could be used to analyze much more modern approaches (e.g., Post-hoc CBMs, CEMs, ProbCBMs, Energy-free CBMs, etc.).

Response: We understand this concern. In our Related Work of section 2, we provided a short justification of why we believe these modern CBM approaches (some of which you also mention) either do not address information leakage at all or they partially resolve the issue. Since reviewer MJtJ had a similar concern, we provide below the same detailed justification for three of these works:

• “Concept Embedding Models. NeurIPS 2022”: This paper introduces the idea of a “concept embedding”, which was later adopted by more CBM papers such as PCBM [2]. In our work, we use scalar-valued concepts instead of concept embeddings, following the original CBM paper*. While concept embeddings achieve excellent task performance because they lead to more expressive concept representations, the authors do not comment on information leakage, i.e. they do not provide a justification about whether these concept embeddings also capture unintended (“leaked”) information from the inputs to improve the task performance. Instead, they propose the Concept Alignment Score (CAS) to measure how much learnt concept embeddings can be trusted as faithful representations of their ground truth concept labels. Their intuition is that clustering samples based on a faithful concept embedding would result in coherent clusters. While they show that CEMs achieve high CAS scores, we argue that their method may not be sufficiently interpretable because:

  • They achieve CAS scores of around 80% in certain datasets, such as CUB and CelebA, which may imply the presence of leakage.
  • This approach does not indicate which subsets may suffer from the imperfect concept alignment, or how does this imperfection affect concept-based explanations. In contrast, our tree-based method allows for group-specific leakage examination in the form of decision paths, and gives the exact decision rules based on leakage.
  • The information captured in a high-dimensional concept embedding is unintuitive compared to a scalar-valued concept, which directly represents the probability (confidence) of the concept predictor.

• “Post-hoc Concept Bottleneck Models. NeurIPS 2022”. Similar to CEMs, this work uses concept embeddings but in the form of concept activation vectors (CAVs). Also, the authors do not address the issue of concept faithfulness or that of leakage, since they rely on multi-modal models to learn concepts that may be unavailable. While they effectively deal with the problem of missing concept annotations, their concept quality and faithfulness relies on the fidelity of their multimodal model.

• “Addressing leakage in concept bottleneck models. NeurIPS 2022”. This work is closer to our method, in the sense that a) it uses scalar-valued concepts and b) specifically addresses leakage. However, as we mention in our related work, “Havasi et al. (2022b) tackle missing information with a side channel and an auto-regressive concept predictor, but these approaches struggle with interpretability and disentanglement of residual information (Zabounidis 2023)”. Specifically, all such works which use a residual layer or side-channel aim to let missing concept information pass directly from inputs to targets, keeping the ground truth concept representations not influenced by leakage. Yet, (Zabounidis 2023) highlight that the residual is not guaranteed to capture this intended missing information, and the two representations may be entangled. We argue that the lack of transparency in the residual channel does not make these methods convincing enough for this problem.

Thus, we believe that MCBM-Seq is more effective as an analysis tool and provides some novel advantages compared to existing methods, such as the ability to perform group-specific leakage examination and to identify the exact decision rules based on leakage. Our work cannot be compared with these previous works in terms of how they deal with leakage, since they either do not address this problem, or they indirectly address it using vastly different approaches. Also leakage was not quantitatively defined in these works in order to be properly compared (in contrast to other traditional metrics such as task accuracy).

Regarding the question of whether MCBM-Seq could instead be used to analyze these modern approaches, we believe that it is compatible with many other CBM methods because it does not pose any constraint on the architecture of the concept encoder (lines 536-538). It is promising to combine an Auto-Regressive Concept encoder from the work “Addressing leakage in concept bottleneck models. NeurIPS 2022” with our tree-based label predictor.

(we continue our response to question 3)

Identifying such instances can then assist our decision-making when deriving concept-based explanations per decision path. This corresponds to the analysis of section 5.3. For better clarity, we update this section by defining two scenarios:

• If a group (leaf node) cannot be further split using leakage, i.e. a sub-tree is not found: This shows that the particular group is not affected by information leakage, which is desirable. In addition, if the task accuracy for this particular group is high, we may consider this an ideal classification example, because the available concept annotations are sufficient for an accurate distinction. The fact that we can identify and isolate such groups is highlighted as a key advantage of our work. Unlike a purely soft CBM, leakage will not impact these groups, and thus the concept explanations for those groups are both leakage-free and accurate. If, on the other hand, the task accuracy is not sufficient, we may flag this group to an expert to either annotate additional concepts or perform an intervention (future work).

• If a group (leaf node) can be further split using leakage, i.e. a sub-tree is found: Then this group can be flagged for additional analysis. We describe a detailed case study for the Woodpecker example in page 10, paragraph: “Our tree-structure allows for meaningful group-specific explanations”. We urge the user to investigate if this difference in likelihoods (leakage) might be intuitive or not, similar to the cat-dog example we described in the beginning. Then the user has the following options: a) rely solely on the decision process of the global tree to derive a perfectly understandable, leak-free explanation, b) extend the decision process with the sub-tree of MCBM-Seq if this likelihood difference seems intuitive, c) use MCBM-Joint’s less intuitive probabilities for maximum accuracy, d) flag this group to an expert to either annotate additional concepts or perform an intervention (future work).

In conclusion, identifying leakage is useful because it provides these analysis tools to a decision maker when deriving explanations, which are not available to a standard CBM. We currently develop future work providing interventions and concept discovery strategies specifically when leakage is observed. However, future leakage mitigation strategies first require a method that controls and inspects leakage for specific sub-groups, thus we believe this work may be used as a very useful analysis tool.

Question 3: If the argument to be built is that MCBMs are better for post-hoc analysis than prediction, then I would say the paper should focus more on the analysis part and show how MCBMs can be used to lead to actionable changes/edits/insights that improve the underlying model somehow. … Therefore, do you have any empirical evidence of actionable changes derived from insights from MCBM that led to a model’s update improving its performance under a reasonable metric?

Response

The argument is indeed that MCBMs are better for post-hoc analysis than prediction, which was clarified in more detail in our previous response. Similar to reviewer VU8H, in this response we will first justify more why we consider leakage inspection useful and then explain potential actionable changes proposed in section 5.3 in more detail.

Let us first adapt a practical example of leakage in CBMs described in [1]. Assume we have a CBM performing animal classification. When the concept predictor recognizes an image as a dog, it may predict a slightly higher likelihood to the concept ‘tail’ compared to an image of a cat. Let us assume for example that the concept predictor gives a likelihood around 0.8 to the majority of dogs for the concept ‘tail’, and around 0.6 for the majority of cats. The ground truth concept ‘tail’ for both classes is 1 (meaning that both animals have a tail). Even though cats and dogs are indistinguishable based on this ground truth concept, the label predictor may still compare the two soft concept probabilities (0.6 and 0.8) and make an accurate classification. However, is this concept-based explanation useful? The label predictor relies on a difference in likelihoods that may (or may not) be interpretable to a human. For example, it might be the case that in most images of cats their tail is not clearly shown due to the orientation of the cat, whereas in most images of dogs the tail is clearly visible. This phenomenon is called “leakage” because unintended information from the input is captured in the concept likelihood (the orientation of cats and dogs). Thus, the concept predictor assigns a higher likelihood to dogs having this concept. However, this may be just an assumption. Regardless of whether this difference is intuitive or not, we believe it is crucial to first find a systematic way to isolate instances where the label predictor takes advantage of such differences, in order for us to then proceed to further analysis.

Thus, our work first provides a systematic way to identify those instances affected by leakage using our tree structure. Specifically, we first train a tree using only the ground truth concepts, which we name the “global tree” (step 1 in Figure 1). Each leaf node in the global tree corresponds to a subset of examples that are indistinguishable based on their ground truth concepts. For example, if we observe the leftmost decision path in the toy MNIST example of Figure 3, there is a leaf node corresponding to group of 926 digits (278 “6”s, 33 “8”s and 615 “9”s), and all these digits have small length and small thickness. We see that this group was not extended with leakage after performing our algorithm, i.e. a sub-tree was not found (step 2 in Figure 1). This means that the label predictor could not take advantage of any differences in the likelihood of concepts “length:small” and “thickness:small” to further split this group. On the other hand, the third decision path from the left, which corresponds to the group of digits having small length and large thickness, was extended with two decision rules corresponding to differences in concept likelihoods (leakage): “length:small < 0.8” and “thickness:large < 0.8”.

(the answer to question 3 continues to the next response)

Weakness 1, Questions 1 and 2: The performance difference between MCBM-Seq and joint CBMs, arguably a weak baseline for incompleteness compared to more recent approaches, seems to be large enough that one could construct a very convincing argument that any gains in reductions in concept leakage are not worth it in practice...

Response

Your concern about the significant drop compared to joint CBMs is perfectly reasonable. However, on the other hand joint CBMs highly suffer from information leakage as shown in previous work [3,4]. We believe that having a high-performing CBM with very unreliable explanations due to leakage contradicts the reason CBMs were created in the first place, which is to provide concept-based explanations. Otherwise, it would make more sense to use a high-performing black-box model. Similar to reviewer kyiD, we will attempt to describe the results of Tables 1 and 2 in this response in an intuitive and detailed manner, and we hope our argument will be clarified at the end of this response.

The purpose of Table 2 is to show that MCBM-Seq is comparable in task performance compared to existing CBMs, or lower in performance compared to Joint CBMs which however suffer from information leakage (refer to the work of [3]) and Black-Box neural networks which are inherently uninterpretable. The Table does not show the advantage of our method by itself, but shows how it performs compared to standard methods in order for our analysis to be complete.

The important take-away from Table 2 when looking at the numbers is that the relationship of task accuracies in CBM modes is roughly the following: Hard, Independent < MCBM-Seq <= Sequential < MCBM-Joint (for small $\lambda_C$) < Joint (for small $\lambda_C$) < Black-Box. In contrast, the problem of leakage follows the opposite trend: Hard, Independent (No Leakage by definition [3,4]) > MCBM-Seq (has leakage, but this is inspectable and controllable by the decision maker) > Sequential CBM (has leakage according to [3,4], which is uninspectable, uncontrollable and affects all samples as stated in L460-462) > MCBM-Joint (has more leakage but this is again controllable) > Joint-CBM (typically it has the most leakage and this is uncontrollable, based on [3,4]). The two reverse trends show the trade-offs of CBMs.

Table 1 was constructed to highlight the advantages of MCBM-Seq. First, the last column named “Leakage Inspection” emphasizes that MCBM-Seq is the only method that allows for Leakage Inspection, which is the novel property we introduce in this work. The existing completely soft sequential CBMs, regardless of their type of label predictor (Entropy-Net, Simple Decision Tree), typically have leakage, as shown in previous works [3, 4] but this leakage is neither easily inspectable nor controlled, which motivated our work. We also provide an intuition for this claim with a practical example in Appendix A.3, page 14. Secondly, the table reveals another advantage of MCBM-Seq when compared explicitly with a Purely Soft Sequential CBM using an Entropy-Net as a label predictor, which is that MCBM also achieves higher Explanation Accuracy and does not raise Fidelity issues, as explained in lines 416-426. This second advantage does not hold when compared to purely soft sequential CBMs using traditional decision trees, but the first main advantage of leakage inspection still remains.

The reasons why our leakage inspection metrics is useful are those highlighted in section 5.3: a) we can analyze our model for specific decision paths (groups), and thus b) we can derive more meaningful group-specific explanations, since bi) the decision-maker has the flexibility to control the concept explanation based on the length of the decision path (lines 515-518) and bii) leakage will not impact all decision-making paths in a mixed CBM (lines 518-519).

In conclusion, the argument of our work is the following: If MCBM-Seq has a task accuracy between those of a Hard and a Sequential CBM (Table 2) but is superior in terms of explainability due to its leakage inspection property, which is shown in Table 1 and section 5.3 (pages 9 and 10), then we believe it is a useful training method for CBMs.

In terms of comparing with other modern CBMs, please refer to our previous response. We explain that these methods are only comparable in terms of task performance and not in leakage mitigation. Our method is indeed inferior in prediction accuracy compared to leaky CBMs such as Joint CBMs. However, we believe that leakage inspection and control is a crucial property that has not been thoroughly examined and is equivalently (or even more) important than predictive performance for interpretable models like CBMs.

[3] Mahinpei, A., Clark, J., Lage, I., Doshi-Velez, F., & Pan, W. (2021). Promises and Pitfalls of Black-Box Concept Learning Models. ArXiv, abs/2106.13314.

[4] Addressing leakage in concept bottleneck models. NeurIPS 2022

Dear Reviewer MYyQ,

We once again thank you for your comments. First, we would like to provide some further arguments on the problem of task accuracy drop, by referencing again to the state-of-the-art CBM models, and I hope this confusion will be resolved.

Accuracy drop is empirically observed in practically all CBMs including the state-of-the-art models, when we compare joint to sequential training, and is often even more severe when we use independent training. For example, consider the following paper:

"Post-Hoc CBMs" [1] : Since these CBMs first train the backbone independently, they can be considered an improved form of sequential CBM training with many advantages. The authors compared the performance of PCBM and a simple Joint CBM, and they state themselves in page 15 of the Appendix: "CBMs achieve a slightly better performance than PCBMs, and the original backbone". This fact did not obstruct this paper from achieving a spotlight acceptance at ICLR 2023, due to the numerous advantages of their architecture such as dealing with missing concept annotations.

Carefully reviewing more such papers, the argument that accuracy drops between different modes of training can be claimed for most state-of-the-art CBMs, yet these works were accepted due to dealing with other CBM problems. In this paper, we try to follow a similar logic as well, presenting the advantage of leakage inspection while not observing significant drop between CBMs with the same training mode, e.g. MCBM-Seq with Sequential CBM. The task accuracy comparison of MCBM-Seq with Joint CBM is not fair. We could have put the comparison with joint training only in the Appendix similar to Post-hoc CBMs, if this is such an issue.

Regarding the concerns we did not yet reply, specifically those regarding interventions and lack of error-bars, we absolutely agree and we were working on those experiments during the rebuttal period. However, due to the tight deadline of updating the pdf, we were unable to do so and we decided to withdraw the paper and include these on a future version of our paper.

[1] Post-hoc Concept Bottleneck Models. ICLR 2023