Learning to Intervene on Concept Bottlenecks

We thank you for your time and the valuable review. We are delighted to hear that you find the motivation of our work "superb" and the paper well-written. In the following, we address the concerns and questions posed in your review.

• Why is the Performance of CB2M in Tab 1 worse on full set compared to the identified instances? With identified instances we consider all instances of the validation set to which the CB2M reapplies an intervention. Thus, the identified instances row shows that reapplying an intervention with CB2M does in general result in a good concept and class accuracy on these examples (after reapplying). Thus, the full set includes the samples of this identified set, but also all remaining (mistake) samples. This is the reason for the perceived lower accuracy on the full vs identified set. We have restructured Tab. 1 to make this more clear. Indeed, the concept accuracy of the base CBM on the identified instances seems comparably large. A few concept errors can, however, already result in a large drop in task performance, as observed in class performance of CBM on the identified set. Nevertheless, we have provided explicit FPR and FNR for the generalization task in the appendix (App. A.6). These suggest quite low FPR for CB2Ms, in contrast to a high FNR. However we assume this to be due to hyperparameter tuning for high precision and the fact that negative samples consist of all base model errors, including those that are in principle not correctable via generalizing interventions.
• Comparison to baselines (CooP und Shin et al.): In fact, the goal of our work is not to propose a method that stands in contrast to the mentioned works (CooP and Shin et al.), but rather CB2M represents a method that can be considered orthogonal to these. Specifically, Shin et al. propose several heuristics to estimate which concepts are the most important to intervene on for a given sample. Thus, they do not provide a method to estimate which samples should receive interventions, but instead “only” provide a concept ordering for given samples. Similarly, CooP can provide a subset of concepts to intervene on (even over multiple samples). However, CooP cannot be used directly to identify whether a sample in isolation should receive an intervention. Moreover, neither of these methods tackle the problem of the inefficiency of one-time interventions. In contrast, CB2M can be used to detect samples which potentially require interventions and can then even generalize given interventions to new samples. Thus, overall the mentioned approaches tackle different problem settings such that CB2Ms do not stand in opposition to these, but can in principle be used in conjunction with them (as e.g., Fig. 3 suggests).
• CB2M is not model-agnostic: Indeed, our approach is not in general model-agnostic, but rather agnostic in the context of CBM-like architectures, as long as they have an accessible concept layer and encodings. We have clarified this in the paper and updated the wording where necessary
• High standard deviation in Tab 9 in the appendix: This happens due to two facts: We optimized $t_d$ only once for all augmentations, which is not necessarily optimal given the individual nature of each augmentation. Additionally, the augmentations salt&pepper and speckles have a more disruptive effect on the model encodings, causing fewer samples to be recognized for interventions. We have added a more detailed discussion on this in App. A.6.
• Detection of mistakes for softmax and CB2M is not different (except for C-MNIST): Indeed, our results do suggest the baseline softmax approach to be en par with CB2M. However a major difference between softmax and CB2M is that the softmax approach looks at the model in total. Thus it can, in principle, not distinguish between bottleneck and predictor mistakes. In our investigated data set settings prediction errors are solely caused by bottleneck errors. In more natural and complex settings in which both types of errors occur (bottleneck and predictor error) using the softmax approach will be insufficient. Moreover, softmax and CB2M are not exclusive, as we show in App. A.7 where we combine both approaches into one.
• Assumption of perfect human interventions: Yes, indeed this is an assumption of our work. We have added a discussion on this in a new limitations section (at the end of the evaluations section), linking the mentioned reference as well as to other relevant works.
• Less is more, Fig. 3: Thanks for pointing that out, we have modified the caption.

We hope that this clarifies your remaining concerns. Otherwise, we are happy to answer any remaining questions.

Thank you for reviewing our paper, we are happy to hear that you like our approach and provided valuable feedback.

We hope we can answer your questions in the following:

• What is the concrete goal and potential use-cases of your method? Overall, in this work, we propose to extend CBMs with an inspectable and interactable memory module. This, among over things allows to increase the effectiveness of user interactions on CBMs in two major ways: First, a CB2M is able to propose a preselection of potential errors, which can then be presented to a human user. By effectively guiding the users attention to model mistakes, this preselection potentially reduces the amount of required human inspection and interactions. Second, given the generalization abilities of the CB2M’s memory module, interventional feedback on few samples can effectively be transferred to novel and “unintervened” samples. Thus, overall the goal of CB2Ms is to more efficiently and effectively integrate interventional feedback from expert human users into concept bottleneck models. In fact, as the reviewer hints, the medical domain is a very valid use-case of CB2Ms. Particularly, such high-stake decision domains require extensive and detailed human expert inspection, cf. [1]. However, as high-quality feedback from medical doctors is generally expensive and sparse in nature, it is specifically necessary to make the most of those interactions that can be provided. This is particularly, where CB2Ms can shine. We have demonstrated these abilities on several benchmark data sets, among other things confounded data sets. Specifically, in our updated version we have added a second confounded data set (CUB (conf.)) to make these points more clear.

• CB2M does not achieve SOTA results on CUB: From a performance perspective, the objective of CB2M is not necessarily to achieve new SOTA results, but CB2Ms are rather specifically designed as a flexible extension to increase the effectiveness of human inter- actions (see above) on any CBM-like base model ($g \circ f$). In this work we introduce them in the context of vanilla CBMs based on Koh et al. (2020)’s work. Nevertheless, they can in principle also be combined with more recent architectures which provide a higher baseline performance as the ones which the reviewer suggest. Moreover, the results presented in Tab. 1 in fact depict model performances based on the augmented CUB version, thus explaining the differences which the reviewer hints at this point. We have now marked this as (Aug.) in the paper to make this more clear.

• It is not clear where $x_e$ is taken: In fact, the encoding $x_e$ for a sample $x$ is the input of the last layer of the bottleneck network $g$ (Sec. 2.2, first paragraph).

• Confusion concerning val notation: Indeed, we have updated this to valid(·).

We hope that we have clarified your concerns. Do let us know if you have any further questions or comments.

[1] DeGrave, A. J., Janizek, J. D., & Lee, S. I. (2021). AI for radiographic COVID-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7), 610-619.

We thank you for the extensive review, pointing out valuable strengths of our paper as well as great suggestions for improvement. In the following, we addressed your questions:

• Question 1 & 2: On the fairness of evaluation between CBM and CB2M: Indeed, CB2M has access to more information than CBM in the evaluation. We provide results for CBM finetuned on the respective validation sets (or the test set in case of CUB (Aug.)) in our updated version (App. A.6) where we differentiate between different amounts of finetuning steps (i.e. finetuning epochs). We observe that in all cases, CB2M performs better than doing few finetuning steps, however, for CUB and Parity CMNIST, finetuning for an extended amount of time outperforms CB2M. On the other side, finetuning on Parity-MNIST (unbalanced) does not provide any real benefit, even for an extended number of steps.

  Though we agree to a certain degree with the reviewer on comparing the models with the same amount of data, these comparisons are also unfair towards CB2Ms in the context that CB2M does not perform parameter optimization based on the validation set, whereas the CBM’s parameters in Tab. 10 were specifically finetuned on this. Additionally, a particular benefit of CB2M (in comparison to CBM inetuning) is the potential use-case of dynamic, online learning settings during model deployment. In this case, finetuning on each novel set of data is infeasible and might even lead to issues such as catastrophic forgetting.

  Moreover, this form of finetuning the CBM’s parameters on the interventional data has even more drawbacks in comparison to CB2M in the context of interpretablity and interactability. Specifically, it is very difficult to remove already applied interventions from the finetuned model, if it turns out these interventions were bad/incorrect. On the other hand it is difficult to inspect the representation of the finetuned model, where in CB2M a user can simply inspect the model’s memory.

• Question 3: Comparison of Fig.3 with a CBM: This figure should rather highlight that findings of the related literature (concrete from Shin et al. in this case) also hold when using CB2M. In other words, Fig.3 shows that it is not necessary to provide CB2M with interventions to all concepts to receive significantly improved class accuracies.

• Question 9: Memory of CB2M is not changed during evaluation? Yes, this is correct.

• Question 10: How many interventions have been used for tab 4 (distribution shift)? We used a validation split as for MNIST, consisting of 20% of the training data of SVHN (14000 images). We have added this information to the paper.

We hope that the remaining questions (4 to 8) have been answered within the general remarks. If that should not be the case, we will be happy to provide further clarifications.