Concepts' Information Bottleneck Models

The integration of the Information Bottleneck framework adds complexity to the CBMs. A more detailed discussion of the computational overhead and implementation challenges associated with the proposed method would improve the paper.

The inclusion of the IB mostly adds training overhead due to the need to compute the additional losses, at inference time there are almost none (only for the variational approximation computation like in a VAE).

In big-O complexity the overhead is not present (as it is a constant multiplier, since for the MI estimator, on each training forward step, we just need to compute the MI over a random subset of training samples. We set the size of this random subset to batch_size of 2 * batch_size. So where a basic CBM does forward+backward in O(batch_size) operations, we do so in O(2 * batch_size) operations.

---

The performance of the proposed method may be sensitive to the choice of hyperparameters, such as the Lagrangian multiplier β. A more systematic approach to hyperparameter tuning could be explored to optimize performance.

We show the changes in the proposed CIBMs with different $\beta$ parameters (0.25 and 0.50) in the updated manuscript in Table 2.

---

you claim to achieve significant improvement in performance compared to vanilla CBMs and related advanced architectures. How do you support this claim of being significant? Is this meant to be statistically significant?

Yes. As requested by the reviewer, we performed a Studetn’s paired t-test between the CBM and $\text{CIBM}_E$, and obtained that they are different with a significance p=0.022.

Dear reviewer xX2U,

We were wondering if our reply addressed your concerns.

We will appreciate to hear from you since the time to make updates to the paper is running out.

In the practical scenario, the architecture they employ is not entirely clear to me. [...] (see Q1).

We show a diagram of the architecture used and their link to the variational model in Fig. 1 of the revised manuscript. Moreover, we added implementation details in the Appendix to make it easier to understand what our architecture is.

---

Are the concepts being supervised, and is the same set of concepts used as in traditional CBMs? A simple visual representation of the model, highlighting the differences introduced compared to CBMs, would be very helpful.

Yes, they are being supervised in the sense that we have ground truth labels for them. However, we do not have access to the the distribution $p(c | z)$ that corresponds to them. Moreover, the concepts set is taken as is from the original CBMs work by Koh et al. (and this was mentioned in Section 4.1).

We also added Figs. 1 and 2 to illustrate the model’s architecture and the variables relationships.

---

The rationale behind dropping certain baselines (as seen in Table 2) is not well explained. [...] (see Q2). Q2: Why did you drop certain baselines in some experiments, retaining only a few (e.g., dropping CEM in all experiments except CUB)? I would prefer a comparison with the strongest model, such as CEM, instead of weaker models like PCBM, to ensure a fair performance evaluation.

We appreciate the reviewer’s observation regarding baseline comparisons across datasets and would like to clarify our approach.

Due to limited computational resources, we were unable to train all baseline methods on all datasets ourselves. Instead, we followed existing experimental protocols and relied on the reported figures for baseline methods whenever they were available in the literature. This approach allowed us to focus our computational resources on implementing and evaluating the proposed methods across multiple datasets.

CUB is the most well-established benchmark in the CBM literature, and comprehensive results for various baseline methods, including CEM, ProbCBM, and PCBM, are publicly available. This made it a natural choice for demonstrating performance comparisons across methods. For other datasets, such as AwA2 and aPY, comparable metrics for these baselines were not readily available, which restricted us from including them in our comparisons.

We emphasize that comparisons were made in good faith and to the best of our ability to be fair and comparable, given the available data. For methods like PCBM, where intervention-specific training is not applicable, we included results to provide additional context but did not claim superiority over them in those aspects.

The inclusion of CEM and ProbCBM on CUB allows readers to evaluate our framework against strong baselines in a fair manner. While the comparisons are not exhaustive, they illustrate the practical advantages of our framework in addressing concept leakage and improving interpretability.

Moreover, our results highlight the flexibility and generalizability of the proposed framework across diverse datasets, which is a significant contribution given the limited availability of baseline metrics for non-CUB datasets.

To address this concern further, we propose to explicitly acknowledge this limitation in the manuscript and provide a detailed explanation of the rationale for our choice of comparisons. We will also highlight that resource constraints prevented us from training all baselines on all datasets and invite future work to extend these comparisons.

Q3: Could you clarify whether the trend or the higher value of I(C;Y) is more significant, and explain why this matters? Additionally, what does a lower I(X;C) represent in practical terms? Moreover, please standardize the x-axis range across all plots to avoid misleading comparisons between methods.

The higher values on $I(C;Y)$ represent less uncertainty in the predictions due to the two variables being more dependent on each other. This translates into better classification since the variables are informative to each other. In the case of our experiments, we found that our CIBMs, after training, give us approximately more nats for both the mutual information of the predictive variables $I(C;Y)$ and $I(Z;C)$ in comparison to the CBM.

While having the lowest compression rates by having lower $I(X;C)$ and $I(Z;C)$ is desirable. It is not guaranteed that the compressions don't get rid of predictive information for the concepts and the class labels. In our case, we hypothesized that the higher values in CIBMs in comparison to the CBM are due to the need for predictive information for the tasks. As shown, CIBMs have higher predictive values than the CBM.

---

Q4: The plots in Figure 3 all appear quite similar, and it’s unclear what specific differences I should focus on. Could you explain your claims more clearly and point out the key takeaways?

The objective of these plots is similar to the previous ones with the difference that they show the interaction between the latent variables and the concepts and the data. In this case, the latent variable in CIBM gives us more information about the concepts in contrast to the CBM. However, in the CIBMs the information gain between the latent variables and the input remains similar as between the input and the concepts.

---

Q5: Why was CBM not included as a baseline in Figure 4? Given that CBM likely exhibits a similar trend to CIBM, the statement that “CIBM does not suffer from concept leakage” feels unsupported. Could you strengthen this claim with further evidence or comparative results?

It was an oversight on our part. We now include a full comparison of the baseline and our proposed models in the original Fig. 3. In the revised manuscript, we include a full comparison between the CIBMs and CBM in Fig. 4.

---

Q7: Regarding Table 3, why is the performance on CUB so low when there are no corrupted concepts? [...] Furthermore, do you have any insights into why your model’s AUC drops more than CBM’s as the number of corrupted concepts increases (at some point, CBM even surpasses CIBM)? Additionally, why did you choose to corrupt only one concept in AwA2, using a different evaluation setup compared to CUB? Please also specify the strategy used for intervention (uncertainty or random).

The differences in performance are due to changes in the experimental training needed due to our limited computational resources. In particular, we used frozen encoders (i.e., $p(z | x)$) for all the models to reduce the computational load. Then, we proceed to train them using a random intervention strategy. Regarding the difference in runs, we prioritized CUB given its widespread usage, and decided on using one corrupted concept in AwA2 due the limited runs we could performed.

---

Q8: At L524, what do you mean by “trained with different concept annotation”? Were CBM and CIBM trained using the same concept annotations, or were there differences in the annotations used?

The phrase on L524 means that if one happens to have two concept annotations for some dataset, then it is possible to use CIBM performance to determine which concept set is better. We reviewed our writing to make this clear.

For completeness, in all our comparisons between CBM and CIBM, we train the two models on the same concept annotations for a fair comparison.

---

Curiosity: Did you happen to evaluate CEM in the experimental setup used for Figure 2? It would be interesting to observe the trend of a concept-based model with higher expressive power, such as CEM, in comparison to the models you presented.

No, we didn’t re-train a CEM to get the mutual information values for it.

Several claims are either missing supporting information (Figure 1), lack proper motivation (L426-431), or are somewhat misleading (L467-469). [...] (see Q5).

The objective of the information plane is to show the mutual information on the model variables after training. In particular, we expect to see a model with high $I(Z;C)$ and $I(C;Y)$ such that these variables are dependent on each other (maximally expressive), and simultaneously, low $I(X;C)$ to show that these variables are independent (maximally compressive). However, the compression of the variables does not necessarily measure that the important parts of the variables are being compressed. We present a detailed explanation below.

In contrast to CBMs, which exhibit lower mutual information between inputs and representations ($I(X;Z)$ and $I(X;C)$), CIBMs achieve higher mutual information while maintaining alignment with the target $I(C;Y)$. This behavior reflects the fact that CIBMs are optimized to retain task-relevant information while removing irrelevant or redundant bits. Lower mutual information in CBMs does not necessarily indicate better compression; instead, it may reflect a failure to capture meaningful input features, resulting in noisier or less predictive concepts.

To demonstrate this, we evaluate the alignment between representations and the target $I(C;Y)$ and show that CIBMs consistently outperform CBMs, indicating that the retained information is both relevant and predictive. Additionally, CIBMs achieve better interpretability and concept quality, reinforcing that the higher mutual information is a reflection of meaningful expressiveness rather than leakage. This is further supported by the proposed intervention-based metrics ($AUC_{TTI}$ and $NAUC_{TTI}$) which highlight the importance of retaining task-relevant information in the concepts $C$. While CBMs exhibit lower mutual information between inputs and representations ($I(X;C)$ and $I(X;Z)$), their poorer performance on these metrics, particularly under concept corruption, suggests that this lower information content stems from a failure to capture sufficient relevant features. By contrast, the higher $I(X;C)$ and $I(X;Z)$ in CIBMs reflect the retention of meaningful bits that contribute to better concept quality and downstream task performance. These findings demonstrate that reducing concept leakage requires selectively preserving relevant information rather than minimizing mutual information indiscriminately.

---

If the goal of the model is to reduce leakage, why isn’t it compared against other models that tackle the same issue, such as those cited in the paper (e.g., “Addressing Leakage in Concept Bottleneck Models”)? Including a comparison with at least one of these models would strengthen the experimental validation (see Q6). Q6: Why did you choose not to compare your model with other approaches specifically designed to reduce leakage, such as “Addressing Leakage in Concept Bottleneck Models”?

In the original manuscript, the intervened groups in Fig. 3 correspond to two different strategies for our proposed method. Now, in Fig. 4 of the reviewed manuscript, we now include a comparison of the CBM and CIBMs using these strategies that shows the direct and full comparison between the methods.

Regarding the lack of other comparisons, we do not have them due to our limited computational resources. We decided to evaluate and perform experiments on our models instead of using the compute to perform extensive runs on the baselines.

---

Q1: Is z simply a hidden representation extracted from a neural network (e.g., the output of a ResNet)? Does your model follow the structure x->z->c->y ? Clarifying this would help improve understanding of the overall architecture.

Yes, we have that directed graph on the variational approximation of the distribution. We now added a diagram to better exemplify the relation between the generative model and its variational approximation in Fig. 2 of the revised manuscript. Moreover, regarding the architecture we added Fig. 1 to illustrate what the model is doing, and we added implementation details to the Appendix.

The experimental evaluation is insufficient, primarily relying on comparisons against a vanilla CBM with unspecified training parameters. The results are not compelling, as CEM appears to either outperform or match the proposed methods on the CUB dataset.

Regarding the training parameters, we have included additional details about the parameters used in the experiments in the Appendix.

We highlight that our focus is not only on improving the accuracy of the concept and label predictions but also on the reduction of concept leakage. Thus, for us, it is interesting to see that we can maintain similar performance on the prediction tasks while heavily reducing the dependance of the variables and reducing the concept leakage. Thus, our learned representations (both for the data and the concepts) are better than the baselines as shown by the higher mutual information, $I(C;Y)$ and $I(Z,C)$, in the information planes in Fig. 3 (in the revised manuscript).

Given that we are the first ones to introduce the IB and CBM, as also noted by R.GBnn, we believe that showing comparable results in terms of accuracy while higher coherence between the concepts and the labels and their respective latents (through the higher information gains shown in Fig. 3 in the revised manuscript) is an important result.

Nevertheless, we are currently computing results using CEM. However, due to our limited computational resources, we focused on performing other experiments that the reviewers suggested. We will update our response and submitted manuscript when the results from CEM are done.

---

The experiment section is not comprehensive enough to be reproducible, no code is supplemented.

We significantly increased the details about the implementation in the Appendix. As requested, we shared the code in an anonymized git repository for your consideration https://anonymous.4open.science/r/CIBM-4FE3. We will release the code when the paper is accepted.

---

The Uncertainty Based (UB) concept interventions fail to demonstrate meaningful improvements. The method's performance is comparable to or worse than random baselines. The paper lacks crucial comparisons with contemporary intervention strategies from recent literature.

In the original manuscript, the intervened groups in Fig. 3 correspond to two different strategies for our proposed method. Now, in Fig. 4 of the reviewed manuscript, we include a comparison of the CBM and CIBMs using these strategies that shows the direct and full comparison between the methods.

Regarding the investigation of other strategies, due to our limited resources and time for the rebuttal, it is infeasible to do such comparisons now. However, we will consider them for our future work.

---

The proposed metrics lack novelty and present existing concepts in a potentially misleading way: [...]

While concept intervention trends are not new, our proposed metrics provide a quantitative summary that complements graphical trends and allows direct comparison across models (as highlighted by R.miqn and R.xX2U as well). Unlike previous works, we use these metrics to analyze robustness under corruption, highlighting the connection between concept leakage and downstream performance.

$AUC_{TTI}$ is intentionally simple to provide an interpretable global measure of intervention effectiveness. Its trends align with $I(X;C)$ behavior in $\text{CIBM}_{E}$, demonstrating that higher mutual information values correspond to better intervention robustness and overall performance.

$\text{NAUC}_{\text{TTI}}$ is designed to capture differences between baseline and intervention performance, reflecting the degree of concept leakage. While this can penalize models with higher baseline performance, our analysis shows that $\text{CIBM}_E$ maintains high NAUC even under corruption, indicating reduced concept leakage and more robust information retention.

In other words, the superior intervention performance (based on the proposed metrics) of $\text{CIBM}_E$, demonstrates that its higher $I(X;C)$ and $I(X;Z)$ reflect task-relevant information, not leakage. The metrics quantify how well $\text{CIBM}_E$’s higher mutual information translates into practical benefits: better robustness, stronger baseline performance, and improved intervention response.

Moreover, the behavior of $I(X;C)$ and $I(X;Z)$ in $CIBM_{E}$ defends the proposed metrics: Higher mutual information values validate why $CIBM_E$ achieves better $AUC_{\text{TTI}}$ and $\text{NAUC}_{\text{TTI}}$. The metrics effectively capture the advantages of retaining task-relevant information in $C$ while reducing leakage, reinforcing the value of the metrics as meaningful evaluation tools.

Rather than introducing potentially confusing metrics, intervention results would be better presented through graphs showing performance across multiple concept groups, providing clearer and more interpretable results.

In our original experiments, we computed the aggregated values and do not have access to the individual values. Due to the limited resources, we focused on the other experiments requested in the reviews. We have started the experiments again, and as soon as we get them we will update the manuscript with the suggested plots.

---

Line 278: Which CBM training scheme (joint, sequential, or independent) is used for comparison? Given that sequential training is known to reduce concept leakage (as per the pitfalls paper https://arxiv.org/pdf/2106.13314), why wasn't a comparison made against CBM using hard concept representations and independent training?

We trained our models jointly and used soft concepts. Our experiments focused on other methods with a similar training scheme to have fair comparisons. Due to our limited resources, we couldn’t re-do all the experiments for different training schemes and concept representations.

---

Line 149: Its not clear where Z is coming from under your formulation, presumably some layer before the concept bottleneck?

In the traditional Information Bottleneck theory, the data $X$ and its labels $Y$ are interpreted through a latent variable $Z$. In this sense, in our extension of the IB, the reviewer is right that the latent variable comes before the concept bottleneck and is the result of encoding the data. We now show a diagram to better understand the latent variables relationship and their variational approximation in Fig. 2 of the reviewed manuscript.

---

Line 300: "We use ResNet18 embeddings provided by the dataset authors and train FCN on top of them." For this dataset and the others, are the backbone networks further tuned during training?

We now have details in the Appendix. In summary, the backbone is only trained for the CUB dataset, and for AwA2 and aPY it is frozen where we rely on the features provided.

---

Line 324-377 (Table 2): Why are baseline comparisons inconsistent across datasets? [...]

We appreciate the reviewer’s observation regarding baseline comparisons across datasets and would like to clarify our approach.

Due to limited computational resources, we were unable to train all baseline methods on all datasets ourselves. Instead, we followed existing experimental protocols and relied on the reported values for baseline methods whenever they were available in the literature. This approach allowed us to focus our computational resources on implementing and evaluating the proposed methods across multiple datasets.

CUB is the most well-established benchmark in the CBM literature, and comprehensive results for various baseline methods, including CEM, ProbCBM, and PCBM, are publicly available. This made it a natural choice for demonstrating performance comparisons across methods. For other datasets, such as AwA2 and aPY, comparable metrics for these baselines were not readily available, which restricted us from including them in our comparisons.

We emphasize that comparisons were made in good faith and to the best of our ability to be fair and comparable, given the available data. For methods like PCBM, where intervention-specific training is not applicable, we included results to provide additional context but did not claim superiority over them in those aspects.

The inclusion of CEM and ProbCBM on CUB allows readers to evaluate our framework against strong baselines in a fair manner. While the comparisons are not exhaustive, they illustrate the practical advantages of our framework in addressing concept leakage and improving interpretability.

Moreover, our results highlight the flexibility and generalizability of the proposed framework across diverse datasets, which is a significant contribution given the limited availability of baseline metrics for non-CUB datasets.

To address this concern further, we propose to explicitly acknowledge this limitation in the manuscript and provide a detailed explanation of the rationale for our choice of comparisons. We will also highlight that resource constraints prevented us from training all baselines on all datasets and invite future work to extend these comparisons.

---

Line 431: The claim about CIBME's training stability needs validation loss curves for support.

In the updated version we tone down the claims about the higher stability since the losses show that the methods are as stable or slightly better than CBM. Nevertheless, for completeness, we now added the loss plots on the Appendix.

---

Line 522: Why are concept interventions varied for CUB but not for AWA2?

Due to our limited resources, we had to prioritize which experiments to perform. Thus, we focused on CUB given its prevalence in the literature.

Is the results for CBM in Table 2 corresponding to the case where you use hard (i.e. binary) concept label? If so, it would be beneficial to explicitly mention this;

The ground truth concept labels are indeed binary. However, as to concepts predictions passed to the label classifiers, we are only training (and comparing only against) models that use soft concepts for class prediction.

In detail, the concepts' predictor can be seens as a multi-label task classifier. In practice, we compute $C$ logits, then, we compute binary cross-entropy (BCE) for each of these logits with binary labels. Finally, we backpropagate them through the means of BCEs.

We include these experimental details now in the final version of the manuscript.

---

The proposed IB-based CBM framework for controlling information leakage appears quite general. [...] could alternative methods [...] also be applicable? These methods may be more effective in removing information from the learned concept representation. I feel the paper could benefit from a discussion on the generality of their framework.

We agree with the reviewer's observation that our proposal is more general. In our experiments, we evaluated the MI estimator based on Kawaguchi et al. However, other methods to estimate the MI will be equally applicable as long as there is a gradient based method that allows us to train the neural networks. While we would like to evaluate different estimators, our limited computing resources prevented us from doing so. We have a brief discussion on this point at the end of Section 3.2.

(Major) Despite the elegant framework proposed, some implementation details may lack clarity and require further justification; [...] (Minor) Reproducibility: despite the very interesting and elegant proposal, no code repo is shared. Together with the missing technical details mentioned above, this weakens the reproducibility of the work.

We significantly increased the details about the implementation in the Appendix. As requested, we shared the code in an anonymized git repository for your consideration https://anonymous.4open.science/r/CIBM-4FE3. We will release the code when the paper is accepted.

---

(Major) The technical method for minimizing mutual information (MI) in the proposed IB-based CBM method is actually not so novel and largely relies on existing methods such as [1];

We show that the inclusion of IB into the CBM generalizes the way MI can be used into the CBMs. One of these ways, the $\text{CIBM}_E$, is similar to a naive extension of the work by Kawaguchi et al. [1]. However, our proposal goes beyond applying Kawaguchi et al.’s idea. We actually show that existing work fits within our proposed generalization framework.

Thus, our technical contribution is the generalization of the inclusion of the IB principle into CBMs and we show how two different approaches can be extracted which can, in turn, be implemented through different approximators of the MI ($\text{CIBM}_E$) and the entropy ($\text{CIBM}_B$).

---

(Major) The comparison between the two IB implementations appears somewhat simplistic and may provide only limited insights. What makes the estimator-based implementation more useful than the other?

Theoretically, the main difference between the $\text{CIBM}_B$ and $\text{CIBM}_E$ is the estimator they will rely on, their difficulty of computation, and the gradient information each provides to the different stages in the neural network.

In $\text{CIBM}_B$, we need to compute the entropy of the concepts which requires an estimator over the true concepts distribution $p(c)$ (note that this is different from the variational distribution $q(c)$). In contrast, $\text{CIBM}_E$ relies on the mutual information between the data and the concepts. Thus, we require a MI estimator to train this model.

These two approaches, in our experiments, showed similar results (once the entropy gradients are regularized), but given other estimators the results may diverge.

---

(Minor) While the presentation is generally good, some content could be more concise and structured. [...]

We streamlined Section 3.1 and moved the derivations to the appendix as suggested.

---

(Minor) The main experimental results are based on only three runs. While I appreciate the author’s transparency in reporting this, more runs could be considered for better robustness of the results;

Due to our limited resources, we couldn’t report more runs on the models. In the revised manuscript, we included an average of 5 runs instead.

---

(Minor) When assessing intervenability, a comparison between the proposed CIBM method and the original CBM is lacking. How CIBM exactly helps in improving intervenability does not seem apparent.

In Fig. 4 of the reviewed manuscript, we now include a comparison of the CBM and CIBMs that shows the direct comparison between the methods.

---

How is the ground truth probability p(c|z) in the conditional entropy-based implementation computed, is it available from the data?

We do not have access to the ground truth probability distribution. However, we do have access to the ground truth concept labels. Thus, our implementation uses the cross-entropy as a supervised method using the ground truth labels of the concepts during training. We added these details into the Appendix.

---

Regarding the estimator-based implementation mentioned in Sec 3.2, what is the exact routine for optimizing I(X; C)? Do you employ an approach similar to adversarial training, where you first estimate I(X; C) before each gradient step for optimizing C?

In summary, before each backward step on some batch $B_1$, we first estimate $I(X;C)$ on some other batch $B_2$, which is randomly collected from the training dataset.

We use the MI estimator from Kawaguchi et al. We rely on the fact that concepts logits are computed with variational approximation to get an estimate for $E[\log p(c|x) - \log p(c)]$.

We include the implementation details in the appendix now.

Dear reviewer GBnn,

We were wondering if our reply addressed your concerns.

We will appreciate to hear from you since the time to make updates to the paper is running out.

Thank you for your detailed response in the rebuttal. I admire the noticeable actions taken to address my and other reviewers’ concern, which significantly improves the quality of the paper. Many of my concerns have been addressed. Thank you!

My final comments are as follows:

• Please consider citing the two referenced works [2, 3] (along with any other relevant literature studying the MI minimization problem) when discussing the various methods for minimizing I(X;C) at the end of Section 3. Both sources appear to be highly relevant to the mutual information minimization problem you are tackling. Additionally, it would be helpful to discuss why [1] is a better choice compared to [2, 3], though this is just a recommendation.
• Please consider performing the comparisons requested by Reviewer U1w5.

I will keep my already positive score at this stage and will determine my final score after discussing with other reviewers. Wish you all the best in the submission.

The improvement of the proposed method compared to existing methods is marginal (Table 2), especially given that prediction accuracy is a primary evaluation metric, making the experimental results less compelling.

We agree with the reviewer about not having substantial improvement in terms of accuracy. But we argue that there are other factors to consider such as the reduction of information leakage while retaining the same prediction accuracy. We highlight that the bottleneck from the leakage limits the representation power and, thus, the final accuracies. In our case, we achieve both.

Moreover, our setup only introduces a constraint in the learning framework, and doesn’t introduce additional components or streams as other methods do. Thus, our proposal introduces a framework that generalizes previous methods, and allows for future expansion.

---

The process of incorporating the IB into CBMs is not clearly explained; adding a diagram to illustrate this process would improve clarity.

We now included a diagram, Fig. 1 in the revised manuscript, that illustrates the data processing pipeline and also illustrates where our IB regularizes the variables.

---

The core idea of applying the established IB framework to CBMs limits the novelty of this work.

We disagree with the reviewer about the limited novelty of our approach. As far as we know, and as highlighted by R.GBnn, we are the first ones to include the idea of Information Bottleneck to regularize the variables’ (data, $x$, latent representations, $z$, concepts, $c$, and labels $y$) mutual information, which in turn regularizes the compression and expressiveness of their relations.

Thus, our work not only establishes a general and extensible framework for using the IB framework to the CBM setup, but it also demonstrates two ways (our two CIBMs) to effectively train the CBM while reducing data leakage. Our proposal can be extended to explore different mechanisms of estimating the mutual information and the entropy of the concepts. Thus, we claim that this exploration poses possibilities for future work and delineates interesting work that moves from the existing body of work.

---

In Table 2, the improvements in prediction accuracy on most datasets are very limited compared to the baseline models. Could you provide more explanation on this? What are your thoughts on these limited improvements, and given this, how can we conclude the effectiveness of the proposed CIB method?

Our focus is not only on improving the accuracy of the concept and label predictions but also on the reduction of concept leakage. Thus, for us, it is interesting to see that we can maintain similar performance on the prediction tasks while heavily reducing the dependence of the variables and reducing the concept leakage. Thus, our learned representations (both for the data and the concepts) are better than the baselines as shown by the higher mutual information, $I(C;Y)$ and $I(Z,C)$, in the information planes in Fig. 3 (in the revised manuscript).

---

Additionally, since the $\text{CIBM}_B$ model in Section 3.1 performs worse than almost all baselines, is it still necessary to devote so many pages to this method? More explanation on this could be helpful to understand the contribution of this section.

The $\text{CIBM}_B$ that we reported on the paper is a fair and direct comparison with $\text{CIBM}_E$. However, we found out that the main reason for the drop in performance is that the gradients from $H(C)$ affect negatively the feature encoder $p(z | x)$. We evaluated different ways of solving this problem, and found out that performing a stop gradient operation on the feature encoder solves the problem. We hypothesized that the problem is due to computing the entropy of the concepts based on the data distribution $p(c)$ while the concept encoder depends on its variational counterpart $q(c)$.

In the original submission, we limited ourselves to the exploration of the estimated version. Now, for completeness, we also show results for the $\text{CIBM}_B$ with the stop gradient version. We detailed this in Section 4.2 of the updated version of the manuscript.

---

The variational inference derivation is relatively straightforward and could be moved to the appendix.

We thank the reviewer for the suggestion. We streamlined the presentation of our proposal in the reviewed version, and moved the derivations to the Appendix.

As a final update about the requested experiments, we were able to perform the experiments regarding hard and soft CBMs trained jointly and independently in CUB and AwA2 datasets. We added those results into the overall performance in Table 2, as well as the intervention results in Fig. 4.

We highlight that our results show better performance in comparison to the hard CBMs. In the interventions evaluations, there is a bump in performance in the hard CBMs due to their nature, but they suffer in coarser datasets. We added a discussion about the differences of these training setups and the proposed CIBMs in Appendix D.

In summary, after the exchanges with the reviewers, and after our previous update, we were able to perform the following experiments and added them to the paper:

1. Inclusion of CEM results across multiple datasets (AwA2 and APY) (Table 2).
2. Inclusion of CEM intervention results on both CUB and AwA2 (Figure 4).
3. Addition of hard-concept CBM (CBM HJ in Figure C.1).
4. Expansion of the analysis on corrupted concepts (Figure C.2) to evaluate robustness and leakage mitigation.

Our evaluation of CEM corresponds to the basic setup since we couldn’t implement the full version during the rebuttal time. Nevertheless, our implementation matches the one reported from the authors, but the basic version doesn’t perform well in the interventions.

Moreover, we give individual replies to the reviewers comments in their respective threads.