We thank the reviewer for their detailed comments and the insightful feedback on the novelty of our method and the formality in deriving and justifying our loss function for learning to defer on Concept Bottleneck Models. In the following, we hope to address and clarify the questions raised by the reviewer.

CUB Concept Annotations. The CUB dataset reports a measure of label uncertainty (Koh et al., 2020), which we use to simulate possibly-faulty human experts. In detail, let $c$ be the label of a concept for a given sample and $u$ be the corresponding label of uncertainty. The uncertainty labels have the following semantics: not visible ($u=1$), guessing ($u=2$), probably ($u=3$), and definitely ($u=4$). Koh et al. (2020) translate the uncertainty labels in the following probabilities, which we use to sample the value $\hat{c}$ of the concept provided by a human expert. We will improve the discussion of this procedure, mentioned in lines 195‒197, by extending our section on the experimental details in Appendix B.

CIFAR-10h. The performance of the human expert labels against the ground truth in CIFAR-10h is $\approx 0.95$.

Concept Embeddings. Yes, the black-box baselines and the CBM models, including our DCBM, adopt the same architecture and the same common frozen representation. Then, the task and concept classifiers are trained independently for the black box, the standard CBM, our DCBM, and its ablations (DCBM-NoTask and DCBM-NoConcepts). We will remark this information in Appendix B, where we already discuss the architectures and the training procedures for the CUB and CIFAR-10h datasets.

Concept Completeness. In a concept-incomplete scenario, when concepts are not sufficient to distinguish between two or more classes, we cannot train a task classifier over those with the same concept-level representation — such as “cat” and “deer” in the CIFAR-10h dataset. The right choice would then be to defer to a human, which can distinguish between the two classes by also employing input data or additional information. By implementing this idea, DCBMs provide a mechanism to deploy a model in an incomplete setting, addressing the risks that might arise from deploying a classifier that arbitrarily chooses one of the two classes. We also report additional results on the coverage of the machine learning model on all the task labels of the CIFAR-10h dataset. The results show how, whenever defer is not too costly, a DCBM only defers instances of cats and deer to a human. Coherently with our formulation, when deferring becomes too costly, the model instead prefers to take a guess instead of deferring, hence increasing the coverage.

Costs and Number of Deferrals. The likelihood of a concept or task classifier to ask for a human is regulated at training time by the cost parameter $\lambda$ and is independent of the probability of deferring of the remaining classifiers. Consequently, at parity of cost, we can expect the cumulative number of defers to be larger for larger sets of concepts.

Comparison with UCP on CUB. A concept coverage of 80% means that the model asks for a human intervention one out of five times. When deferring only on the concepts (DCBM-NoTask), the model performs better than our black box and CBM baselines, until the cost gets too high to intervene and converges to the CBM performance. We report the following additional results, where we compare a DCBM-NoTask (i.e. a DCBM deferring on concepts only) and the UCP strategy from Shin et al. (2023) on the CUB dataset with uncertain humans. We compare with UCP instead of LCP since LCP uses information not available at inference time, and thus it constitutes only a theoretical upper bound as discussed in Shin et al. (2023).

The results show that when we allow for a large number of interventions ($\lambda\leq.025$), UCP underperforms because it asks for interventions based solely on model uncertainty, without accounting for whether the human is likely to be more accurate. As a result, it defers to the human on instances where the ML model predictions would have been a better option. On the other hand, when the budget of interventions is limited, DCBMs have a more conservative approach and tend to prefer the use of the ML model, leading to slightly worse outcomes. We thank the reviewer for the opportunity to add this additional comparison, which we will discuss in our paper.

Task Defer. We thank the reviewer for the opportunity to clarify this point. Yes, deferring at the concepts level might be sufficient to avoid deferring at the task level. In particular, the classification of the task occurs after eventual defers are solved in concept-space. The task is then predicted once a value has been assigned to all concepts. This can be clearly seen once again on the results from the CIFAR10h, when cost is $\lambda=0$ and concepts are perfectly predicted. In this case, as discussed previously, defer occurs only for deer and cat, the only two classes that the ML model cannot distinguish when looking at concepts. For all the other classes, coverage is one, meaning that correcting concepts suffices to avoid defer on the final task, even when it would not be costly to do so. We will add this discussion in the final version of the paper.

Independent Scheme. We train all our DCBM models using the independent scheme, i.e., training the concept predictors from the ground-truth inputs and the task predictor on the ground-truth concepts, as illustrated in Koh et al. (2020). As the only difference, we pre-train an image encoder to avoid parameter sharing among concept predictors (Lines 173–181).

Limitations. We thank the reviewer for the suggestion, and we will add a paragraph dedicated to the limitations of our method. In particular, we will remark that particular care should be taken in assigning deferral costs depending on the application and that the advantages of learning-to-defer hold only when the human distribution is similar enough between training and testing. On this last point, we would like to remark that, to the best of our knowledge, the problem of learning to defer under distribution shifts has not yet been studied in the literature. A plausible solution to that would be to use continual learning strategies, i.e., adapting the training process to a stream of data with possible distribution shifts. The integration of learning-to-defer and continual learning constitutes a promising direction that we will report as future work. We will also take into account the minor points on formatting and readability, for which we thank the reviewer.

We thank the reviewer again for their constructive and helpful suggestions. We will incorporate this discussion, additional experimental results, and not-previously-mentioned limitations in the final version of the paper. Please let us know if there are some remaining concerns the reviewer would like us to discuss.

We are glad to address the remaining questions in the following.

I assume that the results on CUB in the paper are based on class-based concepts for CUB [...] how does this correspond to the use of uncertainty labels, as these (at least to my knowledge) are given based on the actual concepts the human labelers assigned to the sample and not to the class-level concepts?

We thank the reviewer for raising this question about the CUB dataset, as it allows us to better detail how we implemented human uncertainty in the experiments. The reviewer is correct in identifying that our model is trained on the class-based concepts of the CUB dataset as ground-truth. However, to better realistically simulate the uncertainty of a human labels, we used the sample-level labels to define the human predictions w.r.t. the class-level (ground-truth) concepts. Notably, this does not constitute a requirement of our model — which could handle class-based human annotation —, but instead generates a more challenging scenario to learn the human distribution, as we comment in the next paragraph.

If these are not aligned, then it seems to me like the uncertain human provides more false labels than intended.

We agree with the reviewer that it is possible that a human expert may provide less accurate labels in this setting, both for the final task and for the concepts. Indeed, the core idea of the learning-to-defer paradigm is that, in those cases where the human expert provides less accurate predictions, the system will learn to rely more on the machine prediction, thus it is able to avoid likely useless interventions at inference time. This was also the main aim of our experiment with the uncertain human (cf. Fig. 3).

Concept Completeness

We agree with the reviewer, and we thank them for the suggestion. In the revised version of the paper, we will add the discussion of the concept completeness mitigation research question with the content of our rebuttal.

Is the coverage aligned between DCBM and UCP?

Yes, in our additional results, we set the budget interventions for UCP to be aligned with the coverage of a DCBM on the concept classification problems. In practice, for a fair comparison between our approaches, we proceed as follows. For each defer cost (i.) we train a DCBM, (ii.) we compute the coverage on the concepts for the DCBM, and (iii.) we train a CBM using UCP setting the intervention budget to the coverage of the DCBM. For instance, if the cost $\lambda$ translates into $CovConc=.80$, we allow UCP to intervene on the $1-CovConc=20\\%$ most uncertain concepts.

And, regarding the question regarding the CUB uncertain human information, how does this affect these results?

We use uncertain human information to simulate annotations from an imprecise human expert, i.e., an expert whose answers might differ from the ground truth. Intuitively, the more a human expert might misclassify some instances, the more it makes sense to use a learning-to-defer approach. If the human was to always be right, intervention strategies such as UCP would be appropriate too, as they only consider model uncertainty and an intervention would always be beneficial.

We will be happy to answer more questions or to delve into more details for any further concern or curiosity.

We thank the reviewer for their feedback. We appreciate the reviewer recognizes that "exploring human-in-the-loop interactions with interpretable systems is highly useful for real-world deployment of these models" and having appreciated the clarity of our writing.

In the following, we address the reviewer’s concerns on the weaknesses of our work:

Baselines. We thank the reviewer for the opportunity to clarify this point. Intervention strategies, such as those proposed by Shin et al. (2023), allocate a number of admissible interventions and then choose for each instance, or for a batch of instances, on which concepts to intervene. Using learning to defer methodologies, we instead equip a CBM with the capability to (i.) autonomously ask for human intervention and (ii.) acknowledge the capabilities of the expert, i.e., we consider fallible human beings with variable performance. Intervention selection strategies instead only consider the uncertainty of the model, which however might disregard that a human would not be better than the ML predictor in classifying a particular instance. Practically, a DCBM autonomously calls for human intervention on the concepts on which there is the higher probability that the human would outperform the model itself. We report the following additional results where we compare DCBM-NoTask (i.e., without the option to defer on the final task) and the UCP strategy from Shin et al. (2023) on the CUB dataset with uncertain humans.

The results show that when we allow for a large number of interventions ($\lambda \leq .025$), UCP underperforms because it asks for interventions based solely on model uncertainty, without accounting for whether the human is likely to be more accurate. As a result, it defers to the human on instances where the ML model predictions would have been a better option. On the other hand, when the budget of interventions is limited, DCBMs have a more conservative approach and tend to prefer the use of the ML model, leading to slightly worse outcomes. We thank the reviewer for the opportunity to add this additional comparison, which we will discuss in our paper.

Claims on Human Uncertainty. In our experiments with the CUB dataset with uncertain labels, we evaluate our claims that (i.) humans might introduce errors when intervening and that (ii.) DCBMs correctly handle this scenario. Concepts annotated by humans do not necessarily correspond to the ground truth, as we discussed in the previous question and whose construction we report in the following “Concept Annotations” paragraph of the rebuttal. As shown in Figure 3b, we can see that even when the cost of deferring is equal to zero, the model still directly classifies 19% of the concepts at inference time, i.e., $CovConc\approx0.185$. This is because the DCBM is able to detect instances where the human will provide an inaccurate answer. Compared to intervention strategies, cost zero of intervening would correspond to allocating an intervention budget on all concepts. Without knowing the human distribution, this strategy would intervene also on concepts where it would have been better to use the ML classifier, as it is also visible in the additional results provided in the previous answer.

Having already tackled the question on intervention strategies when discussing the reported weaknesses of our work, we now address the remaining questions of the reviewer.

Accuracy-interpretability trade-off. The CEM paper shows that there is an inherent trade-off between the interpretability of concepts and the accuracy of the task: if one is willing to sacrifice part of the interpretability of concepts (e.g., by resorting to concept embeddings), the performance of the concept-based model gets closer to the one of a black-box model. In other words, using embeddings is generally less interpretable than a plain-vanilla concept-bottleneck model, but still more interpretable than a black box model.

Concept Annotations. The learning-to-defer paradigm assumes to model the predictive distribution of a human expert. Therefore, the annotations made by the human expert reported in the dataset and the decisions made at inference time should approximately follow the same distribution. Intuitively, they should be inaccurate in a similar way. Concerning the procedure to account for human uncertainty in the CUB concepts (lines 195‒197), we adopt the following strategy. Let $c$ be the ground-truth label of a concept for a given sample and $u$ be the corresponding label of uncertainty, as provided by Koh et al. (2020). Uncertainty labels have the following semantics: not visible ($u=1$), guessing ($u=2$), probably ($u=3$), and definitely ($u=4$). Koh et al. (2020) translate the uncertainty labels in the following probabilities, which we use to sample the value $\hat{c}$ of the concept provided by a human practitioner. We will improve the discussion of this procedure, mentioned in lines 195‒197, by extending our section on the experimental details in Appendix B.

Concept Uncertainty. No, a DCBM does not use concept uncertainty as an estimate to whether it should ask for intervention or not. Instead, it models human uncertainty by training over a dataset containing both ground-truth values and human predictions, as done in standard learning-to-defer. In this way, a DCBM models whether a human will be able to provide a correct answer or not. Depending on the defer cost, it then determines whether to defer the answer to the human or directly answer the classification problem.

We thank the reviewer again for their constructive feedback. We will incorporate these clarifications, additional experimental results, and revised explanations in the final version of the paper. Please let us know if there are some other concerns the reviewer would like us to discuss.

[...] the cost of intervention hyperparameter seems directly analogous to specifying the number of interventions.

The cost hyperparameter influences the number of interventions. However, it is a parameter that affects each classification task independently. The number of interventions therefore depends on the cost and on the particular instances at inference time. On the other hand, the number of interventions of UCP is set a priori. In other words, DCBMs are adaptive to the inference time distribution, while intervention strategies such as UCP are not. For instance, if we fix $\lambda$ this does not imply that we are always going to intervene $n$ times on a batch of instances, as UCP would instead do. As shown in the paper experiments and in the previous controlled scenario, intervening is not always a good strategy assuming a non-oracle human. Furthermore, costs can also be differently specified for each concept, if a concept is more expensive to be intervened on compared to others. This can not be expressed using UCP, which assumes concepts to have the same importance.

The higher-cost regime for interventions is more realistic in the real world, where UCP outperforms the proposed approach

We agree that a higher-cost regime can be more realistic in some contexts. However, several high-stakes settings (see e.g., healthcare or finance) often require the human overseeing the ML model due to responsibility of the final decision. Hence, it is not unusual to have large amount of human supervision (read lower cost-regime), as making a mistake is very costly. In this context, defer can decrease the cognitive load of the human, specifying on which concepts to focus on. We argue that in these contexts, DCBMs constitute a possible solution to build better models for human-AI collaboration.

It seems to me that the approach requires uncertainty labels in addition to the concept labels to build a distribution from which to sample from.

No, our approach does not require uncertainty labels additional to the concept labels. By following the learning-to-defer paradigm, DCBMs require predictions from an expert, which we stress can be different from the ground-truth concepts — e.g., the ground truth might require time to be discovered, as for health outcomes, or relies on multiple annotators majority votes. Uncertainty labels are just a particular feature of the CUB dataset, which we employed to simulate experimentally the case of imperfect human annotations.

We thank the reviewer for the additional comments, which we are glad to address in the following.

I do not see how UCP is not also autonomous, could the authors clarify this point?

UCP assumes that humans never make mistakes when intervening, and requires the deployer to define beforehand an intervention budget. This implies that the same number of interventions is applied to each batch of instances by selecting the top-k most uncertain concepts. Differently, learning-to-defer based approaches autonomously choose whether to defer one or more concepts at inference time on each input sample. Suppose a scenario where a batch contains only “easy-for-humans” instances — that might be deferred — and a batch containing only “difficult-for-humans” instances — that should not be deferred. UCP would intervene on both batches, possibly decreasing the performance of the overall human-AI collaboration, as reported in the additional results from our rebuttal and in the following. On the other hand, DCBMs would intervene only when strictly necessary, i.e., on concepts from the first batch.

I do not see inherent value in acknowledging the capabilities of the expert if the subsequent performance does not improve

Acknowledging the capabilities of an expert is fundamental to learn whether it is valuable to defer an instance, i.e., requiring intervention, or not. We stress that a good intervention policy should not only improve performance, but also avoid performance to decrease if wrong interventions were to be performed. That is exactly why it is important to acknowledge what are the capabilities of the human expert to whom we require interventions. With this respect, DCBMs improve the performance of standard CBMs only if the human is sufficiently good at solving a classification task. Indeed, by modeling the human expert distribution, DCBMs defers only when the human is likely to be right and the ML model wrong. On the other hand, by only considering model uncertainty, UCP might wrongly intervene and degrade the overall performance of the human-AI team.

To better motivate this point with a fully-controlled toy example, we consider a slight modification of the completeness data we used in our synthetic approach: we consider a scenario where the human is always correct, on the first 5 concepts (out of 10) and always wrong on the remaining 5 concepts, i.e., the concept predictions for these last 5 concepts always differ from the ground truth concepts.

We think this controlled experiment clearly motivates why it is important to model human fallacies. To the best of our knowledge, our approach is the first to tackle the unrealistic assumption of a human oracle in the context of CBMs. Hence, we think this is an important first step to deploy safer models.

[...] I believe a more robust evaluation of how this approach compares/contrasts to these prior works would be very beneficial

We thank the reviewer for raising this point, which we think has contributed to widen the experimental evaluation with competing methods. In the revised version of the paper, we will take the opportunity of the extra-page to report and discuss the full comparison with UCP, among the other improvements suggested by all reviewers.

We are glad the reviewer appreciated the novelty in being the “first work to combine deferral with concept bottlenecks” and introducing coverage into the interpretability-accuracy trade-off. We now address the weaknesses and the question posed by the reviewer.

Concept Independence. We agree with the reviewer that assuming concept independence might be a strong assumption in some scenarios, which however we share with most of the literature on CBMs [1,2,3]. The reviewer is correct also in stating that errors might be propagated and would require particular attention, which is precisely the reason we discussed in our conclusive section this intuition as a future work. Still, assuming to know the relationship between concepts might be an even stronger assumption than independence for openly available data. Therefore, in this paper, we focus instead on a two-layer structure where concepts only influence the final task label and not each other. Finally, our characterization of DCBMs can be straightforwardly extended to group together sets of mutually exclusive binary concepts (thus not independent) into a single multi-class concept, on which one could easily apply the defer option. As we mentioned in our conclusive section, extending the L2D paradigm to non-independent concepts and acknowledging their interactions is an interesting and challenging research question left for future work.

Human-Error Distribution Shift. We thank the reviewer for highlighting an interesting point on the robustness of DCBMs. To the best of our knowledge, the problem of learning to defer under distribution shifts has not yet been studied in the literature. A plausible solution to that would be to use continual learning strategies, i.e., adapting the training process to a stream of data with possible distribution shifts. Learning to defer systems, therefore, are thought to be used, in a real-world context, with humans that have similar expertise to the ones that provided the additional feedback for training. Concerning the strategic behavior, attacks would be possible from experts resulting to be more accurate than the ML model in the training set, but that strategically choose to misclassify instances at inference time. Detecting this behavior or other distribution shifts is out of the scope of the current work, but we thank the reviewer for the suggestion, which we will mention in the future work references of the paper. To the best of our knowledge, there are no works investigating this specific adversary model in the literature.

Adaptable Defer Cost. Our current framework covers theoretically this scenario. Our formulation separates the deferral loss for each concept, i.e., we have one separate deferral system for each concept and task variable. Therefore, it is possible to specify different costs for each concept depending on how important or difficult that given concept is expected to be. However, the cost must be fixed at training time.

Latency and Provisional Decisions. We thank the reviewer for the interesting suggestion, which provides a straightforward extension in our formulation. In cases where a prediction needs to be given without waiting for human intervention, the machine’s response to concepts and tasks can always be evaluated to formulate an immediate prediction, albeit at a high risk of error. The model predictions are produced as the log-probability of each class, including the additional special class for deferral $\bot$. If the model chose to defer to the human, the DCBM could select the class corresponding to the second-largest value and return that as a provisional response with a warning. This approach is solid as the model is trained jointly to increase the activations of the ground-truth labels and, when the human is correct, of the deferral label $\bot$.

Coverage and Explanations. Indeed, higher coverage rates are reached when defer cost is high and therefore the system is trained to rely less on human experts. This is an instance of the trade-off between coverage, interpretability, and accuracy that underlies our work. The choice of the cost of deferring a given concept or the final task is application dependent and should be a main design choice when deploying a DCBM. We will improve the discussion on this point by adding a subsection on the limitations of DCBMs in the conclusion of our paper, where we already hinted at the problem of having different costs.

Computational Overhead. The cost of training a DCBM against a CBM is negligible, as they have essentially the same architecture, apart for an additional feature on each concept or task classifier. In detail, on our hardware, for a training epoch it takes $\approx 9.0$ seconds on CUB for a DCBM and $\approx 7.5$ for a standard CBM. For CIFAR-10h, given the smaller number of concepts, the difference is even more negligible, with both taking approximately one second per epoch. We report epoch time as, due to early stopping strategies, the overall training time might vary across runs. We will improve Appendix B, which already reports training architecture and hardware details, by reporting training times of DCBMs and CBMs.

We thank the reviewer again for their positive feedback. We will incorporate these clarifications, suggested limitations and additional discussion in the final version of the paper. Please let us know if there are some other concerns the reviewer would like us to discuss.

1. Tuomas P. Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In ICLR. OpenReview.net, 2023.
2. Danis Alukaev, Semen Kiselev, Ilya Pershin, Bulat Ibragimov, Vladimir Ivanov, Alexey Kornaev, and Ivan Titov. 2023. Cross-Modal Conceptualization in Bottleneck Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5241–5253, Singapore. Association for Computational Linguistics.
3. Rémi Kazmierczak, Eloïse Berthier, Goran Frehse, Gianni Franchi: CLIP-QDA: An Explainable Concept Bottleneck Model. Trans. Mach. Learn. Res. 2024 (2024)

We thank the reviewer for their useful feedback. We are glad to see that the reviewer states ours to be “a great work that attempts to add human-in-loop setting for CBMs”, appreciates the overall organization of the paper, and praises the soundness of our theoretical results.

First, we address the weaknesses highlighted by the reviewer:

Defer on Task. Deferring on the final task is a useful strategy to mitigate risk in incomplete scenarios where concepts alone are insufficient to determine the task label. For instance, in the CIFAR-10h dataset, concepts do not allow distinguishing cats from deer, since they have the same concept-level representation. Clearly, standard CBMs with no leakage would fail in such classification task. Conversely, in these cases, incomplete concept combinations trigger the defer option of a DCBM on the task, highlighting instances that can not be classified by looking at concepts only. We show this experimentally on CIFAR-10h, where we can see that deferring on the final task improves the final accuracy (Figure 4). Instead, the accuracy of the ablated DCBM-NoTask (i.e., a DCBM with no possibility to defer on the final task) plateaus at around 90%. This is due to DCBM-NoTask randomly guessing between deer and cats, while correctly classifying other classes. Furthermore, we report the following additional table showing how, whenever defer is not too costly, a DCBM correctly defers only instances of cats and deer to a human. Coherently with our formulation, when the cost increases, the model instead prefers to take a guess instead of deferring. The table reports the coverage on all task labels for increasing cost $\lambda$.

Concept Independence. We agree that assuming independence among concepts can be thought of as a strong assumption. Nevertheless, we stress that (i.) this is a fundamental and widely accepted assumption in the CBM literature [1,2,3]; (ii.) assuming to know the relationship between concepts might be an even stronger assumption than independence for openly available data; (iii) our characterization of DCBMs can be straightforwardly extended to group together sets of mutually exclusive binary concepts (thus not independent) into a single multi-class concept, on which one could easily apply the defer option. As we mentioned in our conclusive section, extending the L2D paradigm to non-independent concepts and acknowledging their interactions is an interesting and challenging research question left for future work.

Concept Interventions. We thank the reviewer for this question, which we however might have not fully understood. The primary objective of our work is to equip CBMs with the capability to autonomously call for human intervention when appropriate. Therefore, the amount of required interventions is chosen by the model itself, depending on an application dependent hyperparameter that represents the cost of deferring. All results show how decreasing the cost of deferring, reduces the coverage of the machine learning model by increasing the number of defers and hence improves the performance of the overall DCBM.

Having already addressed the question on task deferrals, we answer the remaining questions:

Accuracy Trends. We are sorry if Figures 3 and 4 were not clear enough. The figure reports the contrary of what puzzles the reviewer: since on the x-axis we have the cost of deferring (larger cost, means more coverage/less defer), we can see a positive effect of defer on both Task accuracy and Concept Accuracy: in all the Figures indeed we have the largest accuracy at $\lambda=0$, i.e., when deferring is not costly, while increasing the defer cost (i.e., more coverage of the ML model) reduces the number of defers and the accuracy.

Concept Interpretability. We fully agree that whenever the concepts are difficult to predict and a human is likely to make a mistake, performing an intervention is counterproductive. We argue that this is precisely why our DCBMs can be useful in practice compared to existing CBMs, which often assume the human expert is an oracle, limiting their applicability in real-world contexts. Conversely, by explicitly taking into account human performance, DCBMs avoid asking for a human intervention when the expected human performance is worse than that of the ML model. Moreover, letting the model call for human intervention autonomously improves the Human-AI collaboration by partially offloading human oversight to the machine.

Human Study. We thank the reviewer for the suggestion of performing human studies. However, in this paper, we focused on methodological aspects to define a novel method to combine CBMs and the L2D literature. User studies are still rare in the L2D literature: to the best of our knowledge, only a couple of works [4,5] implement human validation for learning-to-defer. In particular, these works featuring user studies only focus on the user study itself due to the non-trivial effort in defining and executing them over L2D peculiarities. As they are definitely not immediate, they constitute a promising future work. We will discuss the feasibility of human studies when discussing limitations in the conclusive section of our work.

We thank the reviewer again for their constructive comments. We will incorporate these clarifications, additional experimental results, and not-previously-mentioned limitations in the final version of the paper. Please let us know if there are some remaining concerns the reviewer would like us to discuss.

1. Tuomas P. Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In ICLR. OpenReview.net, 2023.
2. Danis Alukaev, Semen Kiselev, Ilya Pershin, Bulat Ibragimov, Vladimir Ivanov, Alexey Kornaev, and Ivan Titov. 2023. Cross-Modal Conceptualization in Bottleneck Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5241–5253, Singapore. Association for Computational Linguistics.
3. Rémi Kazmierczak, Eloïse Berthier, Goran Frehse, Gianni Franchi: CLIP-QDA: An Explainable Concept Bottleneck Model. Trans. Mach. Learn. Res. 2024 (2024)
4. Bondi, Elizabeth, Raphael Koster, Hannah Sheahan, Martin Chadwick, Yoram Bachrach, Taylan Cemgil, Ulrich Paquet, and Krishnamurthy Dvijotham. "Role of human-AI interaction in selective prediction." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 5, pp. 5286-5294. 2022.
5. Hemmer, Patrick, Monika Westphal, Max Schemmer, Sebastian Vetter, Michael Vössing, and Gerhard Satzger. "Human-AI collaboration: the effect of AI delegation on human task performance and task satisfaction." In Proceedings of the 28th International Conference on Intelligent User Interfaces, pp. 453-463. 2023.