Enhancing Pre-trained Representation Classifiability can Boost its Interpretability

W3: Discussion of IIS on models with random weights.

Your comments are valuable for us to further improve the completeness of IIS.

Reasons for high IIS of random representations. In Figure 5, the models are not initialized with pre-trained weights. The high IIS during the initial phase of pre-training is because the IIS is designed with the Accuracy Retention Ratio (ARR), which is the ratio of prediction accuracy based on interpretations to that of the original representations. Therefore, for a model with random weights, the IIS may be high because both interpretations and representations have similarly low accuracy.

Influences. The primary objective of this paper is to investigate the interpretability-classifiability relationship of pre-trained representations, which consistently exhibit high accuracy. Therefore, this phenomenon does not affect the application of IIS in measuring the interpretability of pre-trained representations.

Inspirations. However, we appreciate your findings of this phenomenon. Incorporating random representations can further complete the IIS definition. In future work, we plan to refine the definition of IIS by considering factors such as the structural entropy [1] of representations. For example, dividing IIS by entropy values can assign a lower IIS to random representations that have high entropy.

Revision. We have incorporated reasons for the high IIS of random representations in L312-314 and added the target of refining IIS by considering the structural entropy in Section 6 of the revision.

W4: Addition of the standard deviation

We agree with your opinion that the standard deviation is important for validating the method's effectiveness.

Difficulty of providing standard deviation in large-scale datasets. However, it is a prevailing issue in the field of computer vision that the standard deviation is often omitted due to the large scale of datasets and models [2,3]. Given the substantial size of ImageNet data, conducting repeated experiments on it demands excessive resources.

Adding Standard deviations of Table 4 in the revision. Nevertheless, we make efforts to provide the standard deviation for Table 4 experiments with 5 different random seeds in the revision.

W5: Insights of Table 3

Thank you for your valuable suggestions that help improve our descriptions of experimental results.

Explanations of conclusions in Table 3. The sparsity ratio $s$ plays a crucial role in balancing the loss of sparsity and classifiability during the fine-tuning process, i.e., larger $s$ means stronger sparsity constraints. As the experimental results show, the optimal results occur at $s=0.1$, a slight constraint strength. Any larger value of $s$ would result in performance degradation.

Your question about sparsity. This is exactly what our findings can address. While sparsifying the less interpretable pre-trained representations with a large sparsity ratio leads to significant information loss, our investigations in Figure 7 demonstrate that as the classifiability of representations increases, they can be converted to sparser concepts with reduced information loss.

Revision. For a better analysis of the experimental results, we have rewritten this sentence in L430-431 of the revision.

W6: Re-drawing the Figure 1

Thank you for your suggestion! We have re-drawed Figure 1 with real data and experiments in the revision.

[1] Wu, J., et al. (2022). Structural entropy guided graph hierarchical pooling. ICML.

[2] Yang, Y., et al. (2023). Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. CVPR.

[3] Oikarinen, T., et al. (2023). Label-free concept bottleneck models. ICLR.

Thank you very much for the detailed comments and suggestions, which have greatly benefited our research. We hope the following responses can address your concerns.

W1: Clarification of L96-L99

Thank you for your careful review. However, we may not fully understand your concern but we will do our best to address it. The clarification of this part is provided as follows.

Clarification. The "higher classifiability implies more interpretability" is not an assumption but a conclusion of our experimental results in Figures 3, 4, and A12. We observe that the improvement in classifiability is accompanied by enhanced IIS. Enhanced IIS denotes higher interpretability, where representations can preserve more task-relevant semantics for interpretations.

Please let us know if your concerns are related to the other points. We are standing by to address them.

W2: Different methods to increase the sparsity of interpretations.

We appreciate your valuable suggestions. We will clarify the order of concept removal and explore alternatives to reduce the number of concepts.

Concept removal orders. For increasing sparsity, the concepts are removed in ascending order based on their correlation to the image representations in Equation 3. Concepts with limited correlation to image semantics are removed first.

Regarding your assumption of removing similar sub-concepts, firstly, we have conducted deduplication when building concept libraries to avoid the existence of excessively similar sub-concepts. Secondly, the removal is performed based on the similarity between concepts and representations. Therefore, even if there are multiple similar concepts, they may be eliminated simultaneously due to their lack of relevance to image semantics.

For your assumption of removing erroneous concepts, although we cannot guarantee the elimination of all errors, our sparsification strategy helps remove concepts that are absent in images through semantic similarities between concepts and images.

Alternatives for removing concepts. Inspired by your suggestions, we design another way to reduce the number of concepts by aggregating them into a limited number of clusters. For IIS computation, the representations are projected into these cluster centers rather than sparsified concepts.

• Experiments. The comparison results between our sparsification strategy and the cluster-based strategy are provided as follows.

  Model	ResNet-18	ResNet-34	ResNet-50	ResNet-101	ResNet-152
  IIS-Cluster	0.664	0.675	0.721	0.732	0.742
  IIS	0.695	0.719	0.766	0.776	0.780
  Acc@1	63.39	64.22	67.01	67.13	68.00

• Analysis. First, we still observe the positive correlation between classifiability and interpretability with the clustering strategy. Second, replacing our sparsification mechanism with clustering leads to the degradation in IIS. This is because our sparsification mechanism selects available concepts according to the input image while clustering first reduces the number of concepts and employs all the remaining concepts for images.

Revision: Inspired by your question, we have provided a detailed discussion in Section A.2.1 regarding the impact of different sparsification methods on IIS values.

Dear Reviewer, we sincerely appreciate your support and recognition. We will follow your suggestions to implement our future work!

Q2: Sparsify concepts by only masking the small absolute values

Thank you for your insightful suggestions for improving the sparsification process.

Justification: The adjustment of the remaining elements by their distance to the threshold follows the soft-threshold function, which is widely employed for sparsification [3].

Experiments: Following your suggestions, we explore the influence of ``only masking out small absolute value elements without changing others" on IIS (masking-only). Experiments are conducted with the ResNet series on CUB-200.

Analysis. First, we surprisingly observe an increase in IIS with only the masking. This implies that using hard thresholds can improve the accuracy of interpretable predictions in comparison to the commonly employed soft thresholds for sparsification. Second, it is noteworthy that interpretability remains positively correlated with classifiability within the same sparsification method. Our conclusions are not constrained to certain sparsification methods.

Revision: Based on the aforementioned results, we have provided a detailed discussion in Section A.2.1 regarding the impact of different sparsification methods on IIS values.

Q3: Details of Table 4

Accuracy calculation. The accuracy in Table 4 is calculated based on the predictions of linear classifier $g_{\text {cls}}$ in L206-210 of the revision.

Sparsity ratios used in experiments. We leverage the sparsification mechanism in Equation 4 to control the number of concepts available for prediction. The sparsity ratio $s$ is set as 0.5 and 0 for models with 123 concepts and 247 concepts, respectively.

Revision. We have mentioned the usage of classifier $g_{\text {cls}}$ in Section 4.2 of the revision.

[1] Wu, J., et al. (2022). Structural entropy guided graph hierarchical pooling. ICML.

[2] Oikarinen, T., et al. (2023). Label-free concept bottleneck models. ICLR.

[3] Chen, Y., et al. (2023). A unified framework for soft threshold pruning. arXiv.

Thank you very much for the valuable and insightful feedback, which has greatly contributed to refining our manuscripts. We hope the following responses can address your concerns.

W1: IIS for representations with extremely low classifiability

Your comments are valuable for us to further improve the theoretical properties of IIS.

IIS is designed for pre-trained representations. We would like to clarify that the purpose of IIS is to investigate the interpretability-classifiability relationship of classifiability-oriented pre-trained representations. Therefore, we did not consider representations with extremely low classifiability in the design of IIS.

Taking random representations into consideration. We recognize and appreciate your suggestions to complete the IIS definition and will make further improvements in future work. For example, we can incorporate structural entropy [1] into IIS and assign low IIS values for high entropy representations (such as random representations).

Revision. We have mentioned this phenomenon in L312-314 and added the target of refining IIS by considering the structural entropy in Section 6 of the revision.

Q1: Derivation of Equation 2

Thank you very much for your careful feedback, which helped us identify and correct the typo in Equation 2.

Fixing the typo. We inadvertently wrote the quadratic term as a cubic term in the original manuscript, and this has been addressed in the revised version.

Discussion with cos-cubed loss. The relationship between Equation 2 and the cos-cubed loss in [2] can be summarized as:

• Similarities. The objective of both loss functions is to map a set of concept vectors $[\mathbf{c}_1, \mathbf{c}_2,...,\mathbf{c}_M]$ in the representation space to textual concepts $[c_1, c_2,...,c_M]$.

• Dissimilarities. Given the soft-label $\mathbf{y}_x=[y_x^{c_1},y_x^{c_2},...,y_x^{c_M}]$ of image $x$ and the inner products between concept vectors and image representations $\mathbf{q}_x=[f(x)^\top \mathbf{c}_1, f(x)^\top \mathbf{c}_2,...,f(x)^\top \mathbf{c}_M]$. The cos-cubed loss normalizes vectors $\mathbf{y}_x$ and $\mathbf{q}_x$, and utilizes the inner product of their cubic powers as a similarity measure. In contrast, we directly employ the mean square error based on the normalized vectors.

• Experiments with cos-cubed loss. We conduct additional experiments with cos-cubed loss. As the following table shows, replacing Equation 2 with cos-cubed loss does not impact the positive correlation between IIS and classifiability, which verifies the generality of our conclusion.

  Model	ResNet-18	ResNet-34	ResNet-50	ResNet-101	ResNet-152
  IIS	0.756	0.774	0.789	0.802	0.805
  Acc@1	63.39	64.22	67.01	67.13	68.00

Revision: We have fixed the typo and added the discussion of the relationship between Equation 2 and cos-cubed loss in Section A.2.2.

Thank you very much for your feedback, which has been of great help to our work. We hope the following responses can clarify the missing points and address your concerns.

W1&Q1: Correlations between IIS and segmentation performance

Thank you for your valuable comments. Based on the (i) positive correlation between segmentation performance and representation classifiability [1,2], (ii) positive correlation between representation classifiability and IIS (our manuscript), we can derive the positive correlation between IIS and segmentation performance.

Experiments. Given the pre-trained representations of ResNet50, we perform two sets of comparative experiments. (i) We directly fine-tune the representations on segmentation tasks (Origin). (ii) We first adjust the representations through IIS maximization in Equation 8 and then fine-tune them on segmentation tasks (Ours).

Analysis. Compared to original representations, enhancing the IIS of representations leads to an improvement in their classifiability (Acc.). The improvement leads to an increase in the segmentation performance (mIoU).

Revision. We have incorporated the experiments in Section A.2.4 of the revision.

W2: Details of the textual concept extraction.

Thank you for your careful review that helps us refine the illustration of implementation details.

Overlapping concepts have been removed. Following previous work [3], given a concept, if there is another concept already in the concept set with a cosine similarity larger than 0.9, the concept will be removed. The similarity between concepts is measured in the embedding space of CLIP.

Revision. We have added detailed descriptions of the textual concept extraction in Section A.1.1 of the revision.

W3&Q2: More comparisons with ECBM.

Improvement compared to ECBM. Our methods demonstrate superior performance by solely training the classification head with automatically generated concept annotations. In contrast, ECBM employs manual concept annotations to train the entire model.

Comparing to ECBM with the same concept library. Following your suggestion, we re-implement the ECBM on our textual concept library.

• Experiments. We provide the performance comparisons with ECBM on the CUB-200 dataset.

  Model	#Concepts	Acc@1
  ECBM	247	83.21
  Ours	247	83.50

• Analysis. Within the same concept library, our method achieves better performance than ECBM by 0.29%.

Revision. We have incorporated comparisons with ECBM using the same concept library in Table 4 of the revision.

Q3: The utilization of different concept types

There may be some misunderstanding here. Different concept libraries are not concatenated but utilized separately.

According to L134-138, we conduct experiments with different concept libraries separately to ensure the general applicability of our conclusions. Concepts from different libraries are not concatenated during the experiment.

For example, all types of concepts are utilized separately in Figure 3&6. Only textual concepts are utilized in Figure 4&7 and Table 4&5.

Revision. In the revision, we have illustrated the separate usage of different concept libraries in L137-139.

Q4: Comparisons with DEAL

Thank you for mentioning this interesting work that further supports our findings.

Performance comparisons. Following your suggestion, the performance comparisons on ImageNet are provided as follows.

Within the same pre-trained model, our method achieves better performance than DEAL.

Then, we discuss the differences between our method and DEAL.

Training strategy.

• DEAL is specifically designed for the CLIP architecture and employs contrastive learning methods to fine-tune the entire model.
• Our method can be applied to arbitrary architectures and only fine-tune the visual encoder.

Prediction mechanism.

• DEAL predicts based on the average embedding similarity between the image and concepts within the category.
• Our method inputs the image representations to a prediction head without the text encoder.

Interpretability-classifiability relationship.

• DEAL solely observed a qualitative impact of interpretability on classifiability in CLIP.
• In comparison, we find the mutually promoting relationship between both factors on a wide range of models in a quantitative manner.

Revision. In the revision, we have added the discussion of differences between our methods and DEAL in Section A.5.4.

[1] Liu, Z., et al. (2022). A convnet for the 2020s. CVPR.

[2] Liu, Z., et al. (2021). Swin transformer: Hierarchical vision transformer using shifted windows. ICCV.

[3] Oikarinen, T., et al. (2023). Label-free concept bottleneck models. ICLR.

Thank you for your detailed responses to my questions. I have a few additional inquiries:

Q1. Could you specify the dataset used in Table A14?

Q2. In Section A.1.1, was the description for the textual concept included in the original experimental setup? If not, could you clarify any changes observed in the experiment and provide an explanation of them?

We would like to thank you for the valuable comments! We will diligently follow your guidance to further enrich our experiments.

W1: Comparisons between IIS and AVG Acc. in VLG-CBM

Thank you for introducing this work to us. The main differences between IIS and VLG-CBM can be summarized in three aspects:

Metric Definition.

• The IIS is defined as the area under the curve formed by the sparsity and accuracy.
• The metric in VLG-CBM is solely defined based on the average accuracy at different sparsity levels.

Objective.

• The objective of IIS is to quantify the interpretability of arbitrary vision pre-trained representations.

• The average accuracy in VLG-CBM serves as a metric for evaluating the performance of CBMs, a specific type of interpretable model.

Sparsification.

• Our sparsification mechanism can accurately control the number of concepts for prediction during forward propagation.

• VLG-CBM adjusts the number of concepts by the sparsity regularization in the loss function, which requires complex strategies including gradually reducing the regularization strengths during training and post-training weight pruning.

Revision: We have incorporated the discussion of IIS and the metric in VLG-CBM in Section A.5.3 of the revision.

W2: Experiments on additional datasets

Following your suggestion, we conduct experiments on UCF-101, DTD and HAM10000.

Interpretability-classifiability relationship. The investigation results in our manuscript, i.e., the positive correlation between interpretability and classifiability, can be generalized to these three datasets.

Performance comparisons. We provide comparisons with LaBo on these three datasets. Our method achieves better performance than LaBo.

Revision: We have incorporated the additional experiments in Section A.2.3 of the revision.

W3: More comparisons with existing methods

Thank you for your suggestions that enrich our experiments.

Experiments. Following your suggestion, we add comparisons with LF-CBM [1] on CUB-200 and ImageNet-1k datasets based on different backbones.

Analysis. Our method exhibits better performance than LF-CBM. Moreover, the performance of LF-CBM is significantly improved by leveraging representations with higher classifiability. This phenomenon aligns with the discussion in Section A.5.2.

For [2], its performance has only been reported in few-shot scenarios, which deviates from our experimental settings. Nevertheless, the impact of training data volumes on the interpretability-classifiability relationship is an interesting question for investigation in future work.

Revision: We have incorporated the additional experiments in Section A.2.3 and [2] in Section A.5.2 of the revision.

[1] Oikarinen, T., et al. (2023). Label-free concept bottleneck models. ICLR.

[2] Yan, A., et al. (2023). Learning concise and descriptive attributes for visual recognition. ICCV.

Thank you very much for your valuable comments, which help us better clarify our contributions and inspire our future work! We hope the following responses can address your concerns.

W1: Completeness of evaluation

Your concern is very insightful and prompts us to discuss the trustworthiness of concept-based interpretations.

Directly incorporating trustworthiness limits the usability of IIS. We agree with you that similar to the sparsity, trustworthiness serves as another important factor for evaluating the interpretation quality. However, existing evaluations of trustworthiness heavily rely on concept annotations and cannot be applied to arbitrary datasets. Therefore, we did not incorporate this factor into IIS as it would confine the application scope of IIS solely to datasets with concept annotations, which contradicts our objective of applying IIS to arbitrary datasets.

Future plans for incorporating trustworthiness. Nonetheless, we recognize the crucial role of trustworthiness in concept-based interpretations. In future work, we will try to evaluate the trustworthiness automatically by incorporating an auxiliary visual language grounding model, and incorporate this factor into the measurement of representation interpretability.

Revision: In the revision, we have added the discussion on future work of refining IIS by incorporating trustworthiness in Section 6.

W2: Removing orders of concepts

Ascending in the manuscript. As Equation 4 shows, the concepts are removed in ascending order based on their similarity to the image representations. Concepts with the least similarity to image semantics are removed first.

Descending order. Following your suggestion, we report the IIS of different representations by removing concepts in descending order based on their similarity to the image representations.

Analysis. When reversing the order of concept removal, IIS demonstrates a significant decrease due to the projection of representations onto less relevant concepts for predictions.

However, representations with higher classifiability still demonstrate higher IIS under the descending sparsification order. This phenomenon further demonstrates that the positive correlation between interpretability and classifiability is an inherent characteristic of pre-trained representations, rather than a consequence of specific sparsification methods.

Revision. In the revision, we have incorporated the discussions of concept removal orders in Section A.2.1.

W3: Experiments with CLIP

Experiments. Following your suggestion, we provide the comparison results between ViT-B/16 and CLIP-ViT-B/16 on the CUB-200 dataset.

Analysis. The experimental results remain consistent with our investigations in Section 3.2, i.e., representations with higher classifiability exhibit enhanced interpretability. Representations pre-trained with large-scale vision-language datasets have stronger generalization ability than representations of the standard ViT, thereby resulting in improved classifiability on this dataset and subsequently enhancing their interpretability.

Revision. We have added these experiments in Section A.2.3 of the revision.

Thanks for the authors' rebuttal. For the evaluation of trustworthiness, could the authors consider using visualization approach for qualitative validation? For instance, visualizing the activation locations of each concept and comparing them with the input images.

We sincerely thank you for your valuable feedback on our manuscript. Your comments have been instrumental in guiding our future work.

W1: Theoretical analyses of the proposed method

Thank you for your valuable suggestions! We agree with you that adding theoretical analysis can further enhance the generality of our experimental results.

Inspired by the concept disentangle phenomenon observed in [1], we will take the representation disentangle theory [2] in pre-trained representations as a starting point. Then, we will theoretically analyze whether the vectors corresponding to concepts exhibit enhanced disentanglement, thereby enabling descriptions of pre-trained representations with less information loss in a sparse manner.

Revision. In the revision, we have added the discussion of future work on theoretical analysis for IIS in Section 6.

W2: Applicability in real-world scenarios

Thank you for your constructive comments! We want to clarify that our concept libraries are not difficult to acquire and can be applied to real-world scenarios.

Automated concept acquisition. The construction of concept libraries can be accomplished automatically without human annotations. The visual concepts are constructed by aggregating image patches and segments, and the textual concepts are generated by prompting an LLM and aligned with images using a vision-language model.

Experiments on real-world datasets. We have conducted experiments on real-world datasets without concept annotations, including ImageNet-1k (Figures 3&5&6&7&9, Table 5) and Kinetics-400 (Figure A12).

Q1: Weights in the additional linear classifier

The classifier can be used in two cases in Section 4.1 and 4.2.

In Section 4.1, the output of this classifier is used to compute the auxiliary loss in Equation 8.

In Section 4.2, the classifier provides interpretable predictions based on concepts.

Revision. We have mentioned the usage of this classifier in Section 4.2 of the revision.

[1] Materzyńska, J., Torralba, A., & Bau, D. (2022). Disentangling visual and written concepts in CLIP. CVPR.

[2] Wang, X., Chen, H., Wu, Z., & Zhu, W. (2024). Disentangled representation learning. IEEE TPAMI.