Narrowing Information Bottleneck Theory for Multimodal Image-Text Representations Interpretability

Response to Weaknesses:

W1: Thank you for highlighting this concern. We would like to clarify that our main text does not exceed the 10-page limit set by ICLR. As per the official instructions, sections such as the Code of Ethics and Ethics Statement and the Reproducibility Statement are permitted to appear beyond the 10th page and do not count toward the main text length.

W2: We acknowledge the importance of application-specific experiments, particularly for high-risk domains such as healthcare and content moderation. However, due to ethical and privacy constraints, obtaining approval for medical datasets is a lengthy process. We are actively pursuing access to these datasets and will promptly update our repository with the experimental results once permissions are granted. To address the current limitations, we have included two additional public datasets, ImageNet and Flickr8k, which complement the experiments and provide further evidence of our method’s effectiveness compared to M2IB.

W3: We respectfully disagree with the characterization of our experimental scope as limited. The M2IB paper primarily conducts quantitative analysis on only two datasets (Conceptual Captions and MS-CXR), while the remote sensing dataset is used as a demonstration example without quantitative evaluation. Due to the same ethical and privacy constraints associated with MS-CXR, we are in the process of obtaining usage permissions but have not yet received approval. To compensate, we included two additional publicly available and diverse datasets (ImageNet and Flickr8k), which we believe sufficiently demonstrate the robustness and versatility of our method.

W4: Thank you for pointing out the inconsistency in significant figures in Tables 2 and 3. We will ensure that this is addressed in the revised version for improved readability.

W5: We appreciate your attention to minor typographical errors. We will correct the formatting of "blackbox" and ensure that all appendix references are properly linked in the updated version of the manuscript.

---

Response to Questions:

Q1: We confirm that the current content adheres to ICLR’s page limitations. The sections Code of Ethics and Ethics Statement and Reproducibility Statement are explicitly allowed to extend beyond the 10-page limit and do not count as part of the main text.

Q2: We are actively seeking approval for the use of healthcare datasets. Once access is granted, we will promptly publish the results in our code repository to validate the method’s relevance for high-risk applications. In the meantime, we have included experiments on two additional datasets, ImageNet and Flickr8k, to strengthen the evidence supporting our approach.

Q3: To provide further insights into generalization, we have generated qualitative examples using the RSICD remote sensing dataset. These examples can be found in our code repository at [[https://anonymous.4open.science/r/NIB-DBCD/rebuttal/]]. While these results are illustrative, they indicate that the proposed method could generalize effectively to other domains.

Q4: We will revise Tables 2 and 3 to ensure consistent significant figures, which will enhance readability and reduce any potential confusion when comparing results.

Dear Reviewer nFC1,

With the rebuttal phase nearing its conclusion, we would like to express our gratitude for your thoughtful comments and suggestions. We have ensured the manuscript adheres to the ICLR page limit and included additional datasets (ImageNet, Flickr8k) to strengthen our experimental evaluation. Formatting inconsistencies in Tables 2 and 3 have also been resolved.

If there are any other aspects requiring clarification, we are happy to provide further details. We hope our detailed rebuttal and added experiments demonstrate the robustness of our approach and kindly request you to reconsider your evaluation.

Best regards, Authors of Submission 11505

Response to Weaknesses:

W1: Thank you for your comment. We respectfully clarify that all our derivations are tightly connected and follow a coherent progression. The core narrative of our work is based on improving and optimizing the bottleneck theory. Each theoretical analysis and algorithmic component is systematically aligned to support the central premise of narrowing the bottleneck to enhance interpretability.All our inferences are to prove the validity of equation 10. You can also refer to our open source code link, which is exactly the same as what is proposed in equation 10.

W2: We appreciate your suggestion to make the methodology clearer. The essence of our approach lies in setting up a bottleneck to identify the most critical features that preserve the model's predictions. As we progressively narrow the bottleneck, the feature set diminishes, allowing us to iteratively pinpoint and prioritize the most influential features. This process ensures that only the most crucial information flows through the bottleneck.

---

Response to Questions:

Q1: Thank you for raising this question regarding randomness in our method. We emphasize that $z_{ic}$ is deterministically obtained by the model, ensuring there is no inherent randomness in its computation. As discussed in Equation 8, our method remains valid as the variance approaches zero. When the variance is zero, the noise term vanishes, eliminating any potential randomness. This theoretical grounding confirms that our method does not rely on stochastic sampling in its final implementation.

Dear Reviewer Ycuc,

As the rebuttal phase approaches its end, we would like to thank you for your valuable feedback. We have clarified the connection between our theoretical analysis and algorithm, refined the narrative for better accessibility, and provided further justification for the determinism of our method.

Should there be unresolved concerns, please feel free to communicate them. We would greatly appreciate it if you could reconsider your score in light of our clarified responses and the improvements made.

Best regards, Authors of Submission 11505

Response to Weaknesses:

W1.1: Thank you for highlighting this concern. We would like to clarify that while the scalar $\lambda$ serves as a universal update parameter for each layer, the flow of information for each feature dimension is determined independently. Although $\lambda$ remains consistent across dimensions within a layer, the actual updated values vary depending on the magnitude of each feature. For example, if a feature has a magnitude of 8 and $\lambda$ is set to $1/4$, the final updated value will be 2. In contrast, if another feature dimension has a magnitude of 6, the updated value will be 1.5. These variations ensure that our theoretical properties hold and that the bottleneck effect is applied dynamically based on the specific characteristics of each feature.

W1.2: We appreciate your concern regarding the Gaussian noise assumption. In practice, any distribution independent of the original feature distribution is valid, as detailed in [1]. We chose Gaussian noise to simplify computational complexity, a rationale similar to its adoption in diffusion models. This choice does not impact the generality of our method but significantly reduces its implementation complexity.

W2.1: As discussed in response to W1.1, the information varies for each feature dimension based on iterative updates.At the same time, our theory is to reduce redundant information while observing task-relevant, so there is no need to make such trade-offs.

W2.2: This concern has been addressed in prior works, including Grad-CAM and M2IB, where the validity of the assumption regarding spatial feature consistency is demonstrated.

W3.1: We adopted Gaussian noise for reasons analogous to its use in diffusion models, as it simplifies computation while maintaining robustness, as discussed in [1]. Additionally, the theoretical guarantees of our method rely solely on the independence of the noise distribution, which significantly reduces complexity. These assumptions have been rigorously validated and proven effective in various scenarios.

W3.2: We agree that incorporating additional datasets could further validate our method. To address this, we added experiments on two additional datasets, ImageNet and Flickr8k, compared to M2IB. While we actively seek approval to use medical datasets, ethical and privacy constraints have delayed access. Once permissions are granted, we will supplement our results with medical datasets in our public repository. Moreover, as shown in Theorem 2, the reduction of $\sigma$ eliminates randomness, making the model more controllable without compromising robustness.

W3.3: While the derivation may appear complex, it ensures the rigor and robustness of our theory. In practice, the key equation is Equation 10, which directly relates to feature attribution. Regarding $\lambda$’s dependency, as the bottleneck input becomes independent noise, the model naturally loses its ability to discern relevant features. This phenomenon aligns with theoretical expectations and is discussed in detail in [1].

---

Response to Questions:

Q1: We acknowledge the significance of including diverse datasets, particularly in sensitive domains such as healthcare. However, due to ethical and privacy considerations, obtaining approval for medical datasets is a lengthy process. We are actively pursuing permissions and will add experiments on medical datasets in the future. Meanwhile, to enhance diversity, we included two additional datasets, ImageNet and Flickr8k, which complement the Conceptual Captions dataset and provide a broader evaluation compared to M2IB.

Q2: We appreciate the suggestion to include more illustrative comparisons. Additional attribution visualizations for our method compared to various baselines can be found in the results folder of our public code repository [https://anonymous.4open.science/r/NIB-DBCD/rebuttal/] and [https://anonymous.4open.science/r/NIB-DBCD/results/].

Q3: The substantial improvement in text interpretability arises because text data often contains more irrelevant or negative features, such as unrelated words, which our method effectively identifies and excludes. This is supported by the metrics, which demonstrate that the attribution values for text data show a greater degree of improvement compared to image data.

Q4: Identifying negative features enhances interpretability by explicitly highlighting features detrimental to model predictions. While this characteristic could potentially aid in debugging models or addressing biases in training data, these applications extend beyond the scope of this paper and would require further investigation.

Q5: Our method is theoretically rigorous and experimentally validated across multiple datasets. Thus far, we have not identified any scenarios where the method fails. However, we welcome further exploration to identify potential limitations in specific edge cases.

Additional Points:

• Regarding computational efficiency, as outlined in Table 1, NIB achieves superior efficiency in terms of forward and backward passes compared to other methods. The efficiency of our method scales independently of model complexity and dataset size, as demonstrated in our evaluation.

Table 1 forward and backward passes of NIB compared to other methods

• Statistical significance testing was not conducted as our method does not introduce randomness, and the superiority of our results over other methods is consistent and evident from the metrics. We adhered to the experimental protocols of prior works for fair comparison.

• Negative feature attribution is a unique capability of our method. Unlike M2IB, which only identifies positive features, NIB preserves the ability to recognize and attribute negative features, as demonstrated by the significant improvements in evaluation metrics.

Response to Weaknesses:

W1: The identical dimensionality of image and text representations is a design choice inherent to CLIP, intended to facilitate contrastive learning by mapping both modalities into a shared embedding space. Regarding Theorem 1 and Theorem 2, we acknowledge the reviewer’s observation. We will adjust the presentation in the manuscript to describe them as remarks rather than theorems to better align with their fundamental nature.

Response to Questions:

Q1: We appreciate the reviewer’s request for additional details. Our experiment setup follows the protocol of M2IB, where 2,000 images are used for testing. The computation involves converting boolean values into binary (0 or 1), which limits the precision to fewer decimal places. The robustness of our method is inherently guaranteed by the attribution axioms it satisfies. We have provided additional visual examples in the supplementary material (accessible at [https://anonymous.4open.science/r/NIB-DBCD/rebuttal/] and [https://anonymous.4open.science/r/NIB-DBCD/results/]).

In Section 6.2 of our manuscript, we also present an ablation study on other layers. The rationale for choosing the 9th layer is based on the Bottleneck principle, which hypothesizes the intermediate layer as the most effective bottleneck for representation learning. Through extensive ablation experiments on various datasets, the 9th layer consistently demonstrated optimal performance, balancing detailed feature extraction with high-level semantic representations, but there are also advantages to choosing other layers