We thank the reviewer for their feedback and helping us improve our work.

Consider visualizing gradient behavior over time...

We agree with the reviewer that this figure can aid in illustrating our claim. A version of it is already in App D, where we empirically confirm our theoretical results.

1 Sequential CBMS

We chose to focus on joint CBMs as they are shown to generally be the best-performing models for a variety of tasks. Below, we provide some results for the unnoised setting using sequential CBMs. We generally observe that CPO provides better results for both the sequential and joint setting, specifically in task accuracy, where we find that the added uncertainty estimates encoded into the concept representation allows for more expressive modeling to be picked up by the decoder.

Sequential CBM results: https://ibb.co/bMJKXcLP

2. What happens when empirical data is systematically biased?

1.1 We agree that incorporating experiments with more structured noise would strengthen our conclusions. Specifically, in our response to reviewer sJF1 we describe the following experiments:

One where noise is introduced at the concept group level (e.g., flipping only wing-related attributes in birds or altering beak colors).

Another where noise is introduced based on the confidence level provided by the labeler (as available in the CUB dataset).

3.Strong assumptions

While our results do rely on the assumption of a uniform prior, this is the weakest prior one can have. We show in Sec 5.3 how using a prior improves the performance of the model through the added flexibility. This ability to leverage a prior (which BCE cannot) but still work using an uninformative one is beneficial. Thus, we believe that while it is an assumption that our results rely on, it is not impairing.

While we acknowledge that concepts are often correlated, assuming conditional independence is a common approach in CBM literature, with dedicated works exploring dependency modeling (e.g., stochastic CBMs [1]). Given this, we opted for a model that assumes conditional independence.

This feedback has prompted us to reflect on whether conditional independence is necessary for the correctness of our analysis. Upon review, we found that it is not required, meaning our results can generalize to more expressive models.

To illustrate this, consider modeling the joint distribution of a concept with all other concepts, $\pi(c_i, c_{0:i-1}|x)$ . Using the chain rule, this can be factorized into an autoregressive formulation:

$$\pi(c_i | c_{i-1}, x)\pi(c_{i-1} | c_{i-2}, x) \dots$$

With this factorization, we can always compute the joint distributions $\pi(c | x)$ and $\pi(c' | x)$, ensuring that our results in Appendix C.1 still hold even without only assuming conditional independence on x.

The main reason for the independence assumption was to simplify the derivation of the CPO loss in Eq. (6). Since we find this assumption unnecessary, we will drop this assumption from our work.

Here, we present preliminary results using this autoregressive formulation. Specifically, we implement this approach with two networks: $\pi_{\phi}(c_{i} \mid c_{0:i-1}, x)$ and $h_{\omega}(\tau \mid c, x)$, where $h_{\omega}$ determines the order of the autoregressive decomposition. In this process, $h_{\omega}(\tau \mid c, x)$ takes as input the concept probabilities predicted by $\pi$ and produces a softmax distribution over concepts. The concept with the highest probability is then selected and predicted by $\pi$, and this procedure repeats in an autoregressive manner until all concept probabilities have been obtained.

AR CBM Table: https://ibb.co/BH9s5jcm

Here, we find that our autoregressive model beats conditionally independent models for BCE, slightly underperforming for CPO. Regardless, we find that CPO also performs better for this type of model.

4. How does LCPO perform in early training phases? Proposition 4.2 suggests LCPO’s gradient updates are more conservative than BCE’s, potentially leading to slower convergence...

In practice, CPO can be slower in fitting training data, but we observe that this does not negatively impact generalization. Instead, it suggests that CPO may be less prone to overfitting, which is a known issue in likelihood-based training. To illustrate this, we provide anonymized links for sample runs on the CUB dataset across two seeds.

https://ibb.co/Z67qYySF (Validation Task Accuracy)

https://ibb.co/4wkNn1WW (Validation C AUC)

https://ibb.co/Xx8WNLnZ (Training task accuracy)

https://ibb.co/chxX81Jf (Training C AUC)

Regarding adaptive learning rates or entropy-based updates, any improvements seen from these strategies should also apply to CPO, as its updates are fundamentally based on data likelihood.

Computational Analysis

Please see our response to reviewer sJF1

[1] Stochastic Concept Bottleneck Models https://arxiv.org/html/2406.19272v1

We sincerely thank the reviewer for their time and effort in helping us improve our work.

Weaknesses

1 Diverse Experiments to Strengthen Conclusions

1.1 Thank you for this valuable feedback. We agree that incorporating experiments with more structured noise would strengthen our conclusions. In our response to reviewer sJF1, we discuss two additional experiments:

One where noise is introduced at the concept group level (e.g., flipping only wing-related attributes in birds or altering beak colors).

Another where noise is introduced based on the confidence level provided by the labeler (as available in the CUB dataset).

1.2 If the model still performs almost as well when the label flips with 0.4 probability, it is deviating from its empirical evidence and relying more on the posterior. For (2) - should we actually ignore empirical evidence?

While it is possible that the model’s strong performance is partially due to reliance on the posterior rather than empirical evidence, we do not believe this fully explains the results. For example, in some settings, particularly for CEMs trained with BCE, models can still perform well even under high levels of noise. We attribute this to the fact that, despite 40% of the labels being noisy, the remaining 60% provide sufficient empirical evidence for the model to fit the data—albeit imperfectly.

2. Improving the Introduction for Clarity

We appreciate the reviewer’s suggestion to include a dedicated paragraph explaining Figure 1 in the introduction. While we agree that this addition would improve clarity, space constraints may limit our ability to introduce an entirely new paragraph. Instead, we propose enhancing the introduction by adding context around Figure 1, making it more digestible for the reader. Here, we provide the modified text we would add to the introduction to address this issue

We propose Concept Preference Optimization (CPO), a policy optimization-inspired objective loss for CBMs. Figure 1 illustrates how CPO leverages pairwise comparisons of concept preferences to guide updates toward preferred concepts while mitigating the impact of incorrect gradients. Unlike traditional likelihood-based learning, which updates on all samples regardless of correctness, CPO selectively adjusts based on sampled preferences. This reduces sensitivity to noise by being able to mitigate incorrect gradient updates when incorrect concepts are sampled. Our analysis shows that CPO is equivalent to learning the posterior distribution over concepts, leading to more robust training. Empirically, we demonstrate that CPO not only improves CBM performance in noise-free settings but also significantly alleviates the impact of concept mislabeling.

Question

1. How are the hyperparameters tuned for the model?

We discuss these details in Appendix A.1, which we provide here for reference:

From App A. We use a batch size of 512 for the Celeb dataset and 256 for CUB and AwA2. We train all
models using RTX8000 Nvidia-GPU. In all datasets, we train for up to 200 epochs and early stop if the validation loss has not improved in 15 epochs. For fair evaluation across methods, we tune the learning rate for CEMs, CBMs, and ProbCBM. Specifically, for CUB and AwA2 datasets, we explore learning rates ∈ {0.1, 0.01}, while for CelebA, we expand the search to ∈ {0.1, 0.01, 0.05, 0.005} due to the observed instability of CEMs at higher learning rates. Additionally, we set the hyper-parameter λ ∈ {1, 5, 10} for all methods. For CEMs and models trained using LDPO, we found RandInt beneficial, which randomly intervenes on 25% of the concepts during training. ProbCBM introduced a few extra hyperparameters, which we did not tune in this work, and we directly used the hyperparameters provided by the original authors. Similar to other models, ProbCBM employs RandInt at 50%, making it particularly sensitive to interventions, especially in concept-complete tasks such as AwA2 and CUB. The only model for which we tune additional hyper-parameters is Coop-CBM, where we adjust the weight parameter for the auxiliary loss.

We hope our additional experiments and clarifications can help alleviate your concerns.

We thank the reviewer for their feedback and encouragements. Here we provide some detailed responses to some of the questions you have raised.

In Eq (6), should $D$ be $\mu$ ?

Yes, that is a notational mistake on our end. We will fix it.

1. In Eq. (5), are c and c′ scalars or k-dimensional vectors?

In our work, Eq. (5) uses scalars. However, this is not a strict requirement—one could use vectors to structure concepts in this way. That said, scalars are significantly less computationally demanding, as modeling structured objects would lead to $2^k$ possible concept representations, which could become impractical.

2. In the later training stages, could c′ ≻ c occur? If so, would continued training hurt the model's ability to distinguish preference pairs?

Yes, this situation can arise later in training. To mitigate this, we apply early stopping across all models. This ensures that the model does not overfit to later-stage preferences in a way that degrades its ability to distinguish concept quality.

3. Is the reason CPO is more robust against concept-level noise that DPO behaves like BCE when c ∼ μ+, but for c ∼ μ−, DPO yields smaller gradients?

This intuition is correct. In Appendix D, we show that when noise is low (e.g., 0.1), CPO’s gradients are indeed smaller than BCE’s. This is consistent with our experiments, where BCE performs reasonably well under p_noise = 0.1. However, at p_noise = 0.3, CPO’s gradients are orders of magnitude smaller than BCE’s. This aligns with our empirical findings—CPO remains relatively unaffected by high noise levels, whereas BCE struggles significantly.

4. Could CPO be used to identify mislabeled concepts or even recover correct labels?

This is an interesting question. While we have not explored this explicitly, we have qualitatively found that CPO’s uncertainty estimations seem more grounded to the target object in the image. To show this, we provide examples of images before and after various augmentations, along with the concept being analyzed. Our uncertainty score is based on the variance of the concept, normalized to a [0,1] range, where 0 indicates complete certainty and 1 represents maximum uncertainty. Results show that when the target object becomes obscured, CPO more effectively increases its concept uncertainty, while both BCE and ProbCBM perform significantly worse at this task. Additionally, we observe that in certain scenarios, cropping the image creates a zooming effect that makes models more confident in their assessments.

https://ibb.co/JR9DZvfd

https://ibb.co/219TTtLN

https://ibb.co/23Rkwqqr

We sincerely thank the reviewer for their time and effort in helping us improve our work.

W 1

We agree. We will address these concerns in the camera-ready version. We discuss the details in comment 2 of our response to reviewer 2knp.

W 2

We also agree (thank you). As you and other reviewers have pointed out, experiments with more structured noise would strengthen our results. To address this, we have designed two experiments with more structured noise to the concepts than our original setting.

Noising by Group Level

As you suggested, we now study mislabeling similar concepts based on concept groups in the CUB and AwA2 datasets. This means we introduce noise to semantically similar concepts, such as switching red wings to brown wings (as in your example). In this experiment, we apply noise at the group level with different noise levels, $p\in$ {$\{0.1,0.2,0.3,0.4\}$}.

Full results are below. We observe that models trained using BCE experience a substantial drop in performance, whereas CPO models remain much more robust. For instance, in CUB task accuracy in CBM CPO remains relatively stable even at high noise levels. Likewise, in AwA2 we observe CPO models are able to preserve Concept AUC performance much better compared to their BCE counterparts.

Unfortunately, due to character limits in our response, we are restricted to providing links to these results. If any reviewer is uncomfortable clicking the link, we will try to provide a version of the full table in our next (final) response.

CUB and AwA2 Noising by Group:

https://ibb.co/NgDZ9sBC

Uncertainty Experiment

Additionally, we leverage a useful property of the CUB dataset: labelers provide confidence scores for their annotations. These scores range from $\{1,2,3,4\}$, where higher values indicate greater confidence. We introduce noise proportionally to these scores—labels with lower confidence are more likely to be flipped. Specifically, we apply noise at the following rates:

• Confidence 1 → 40% noise
• Confidence 2 → 30% noise
• Confidence 3 → 20% noise
• Confidence 4 → 10% noise

CUB confidence-based results:

https://ibb.co/KcfB6qHp

As in other experiments, BCE-based models are heavily impacted by noise, leading to significant drops in task accuracy and concept AUC. In contrast, CPO-based models consistently exhibit robustness to noise, regardless of their structure.

We thank all reviewers for suggesting experiments with more structured noise, as we believe these results significantly improve the quality of our work.

W 3

This is a good point; improving visual clarity is important. We are experimenting with converting Figure 4 into a table. However, we initially chose a plot for its space efficiency. At a minimum, we will include a table with all metrics in the appendix for the camera-ready version.

Additionally, if we retain the figure, we may remove Prob-CBMs from the plot to enhance clarity, restricting their results to the appendix given their underperformance in this setting.

Questions

What exactly is the ground truth concept distribution that the preference is calculated against?...

The preferred concepts are indeed taken from the empirical dataset. We discuss it in lines 165-170:

"To circumvent this issue, we can leverage the empirical dataset and state its preference over a concept set sampled from $\pi_\theta$. The preference over a pair of concepts should hold specifically early on in training where the policy is suboptimal compared to the empirical data."

What are the computational overheads, if any...

We thank the reviewer for suggesting improvements to our computational analysis. Below, we provide a detailed breakdown of the computational overhead associated with each model. The following table reports the average time per epoch (in minutes) for all models, measured over a full run on the CUB dataset. Additionally, we include the number of trainable parameters for each model.

https://ibb.co/8gL3ZNLq

While CPO introduces a small computational overhead, the increase is minimal—approximately 0.05 minutes per epoch compared to BCE. This is in contrast to CEM and ProbCBM, which significantly increase runtime. Moreover, we compare an unoptimized implementation of CPO against an optimized BCE implementation built specifically for efficiency in PyTorch. This suggests that further optimizations could reduce CPO’s overhead even more.

On the other hand, ProbCBM has 4× the number of parameters compared to CBM and nearly doubles the training time, despite underperforming CPO in most settings.

A potential limitation of CPO is that its gradients tend to be smaller—particularly in the early stages of training—compared to BCE. This could lead to slower convergence in some cases. However, as we also mention in our response to reviewer dJqj, we do not observe this empirically.