In Pursuit of Causal Label Correlations for Multi-label Image Recognition

We thank reviewer BNe7 for the positive comments on our work. In the following, we present our responses addressing the raised concerns.

(Weakness 1) In this work, we apply K-means clustering on the spatial features extracted from a pre-trained classification network for confounder modeling. This approach is based on our realization that the cofounders for recognizing certain object are often hard to define and enumerate – objects, scene, and even the texture of the environment are all potential confounders. On the other hand, a pre-trained classification CNN tends to activate the discriminative regions (which could be objects, scene and even the texture) in an image. Therefore, K-means clustering on CNN spatial features can produce a compact set of prototypes to represent potential confounders like objects, scenes and textures.

(Weakness 2) Thanks for your suggestion. We conduct additional experiments regarding the number of the clustering centers, as illustrated in Fig. 2 in our attached pdf file. Our experiments show that 80 clusters are sufficient to represent potential confounders in our datasets, and 400 clusters does not bring significant accuracy gain. We point out that this conclusion still remains as an empirical observation. And, we hypothesize that for more complex multi-label image classification datasets (which might be collected in the future), further increasing the number of clusters may be necessary for modeling complex confounders and achieving higher accuracy.

(Weakness 3) As the confounders are pre-computed, only one forward pass is required to obtain the final prediction.

(Weakness 4) With K-means clustering, each spatial feature will be assigned to a certain cluster $c$. We calculate the number of spatial features for each cluster, divide it by the total number of spatial features, and then obtain $P(c)$ for each cluster.

(Question 1) Thanks for your suggestion. We plan to add more explanations about confounders (e.g., our response to Weakness 1) and more experiments (e.g., other implementation choice for confounders like random vectors) in our revised version.

(Question 2) Since the confounders (and the prior $P(c)$) are pre-computed, the inference process of the causal branch (formulated by Eq. 11) only requires one forward pass. As a result, the inference process of our whole pipeline (Eq. 2) also only requires one forward pass. We will further detail the inference process of our method in the revised version.

(Question 3) Please refer to our response to Weakness 4.

(Limitations) As described in Sec A.1, our current approach still has limitations in interpretability and higher-level causal modeling. We consider these as our future works, and will try our best to solve these limitations.

We are thankful for your acceptance and constructive feedback.

We thank reviewer wRCW for the detailed feedbacks on our work. In the following, we present our responses addressing the raised concerns. Should our rebuttal effectively address the concerns, we kindly hope you can raise your score.

(Weakness 1) We agree with you that a detailed analysis of the hyperparameters is crucial to validate the robustness of our method. Regarding the number of clusters for confounders as you suggested, we have already presented the results in our main paper. Please refer to Tab. 4 and Sec. 5.3.2 for details. As for the parameters, in the following Table, we compare our method to the baseline (naive ResNet-50) and Q2L (SOTA method). Our method significantly outperforms the baseline, as well as the SOTA Q2L method which has much larger model size. In particular, our causal module only adds marginal parameters, but can improve the accuracy (mAP) on COCO-stuff from 56.8 to 60.6, demonstrating the effectiveness of our causal label correlation modeling.

(Weakness 2) We agree with you that a discussion about the limitations is valuable for a more complete understanding of our method. Actually, we have discussed the limitations of our current confounder modeling and causal intervention in Sec. A.1 (Supplemental Material). We plan to add such discussions to the main body of our paper.

Regarding the confounders, we have investigated the effects of different confounder modeling methods (see Tab. 6). In addition, we also add the comparisons of our K-means based confounders and random confounders, which clearly show that inaccurate confounders will lead to significant performance degradation.

(Weakness 3) We agree with you that comprehensive ablation study is helpful for understanding the importance and effectiveness of each component of the proposed method. Actually, we have presented ablation experiments to validate the effectiveness of each component of our method. Please refer to Tab 3 and Sec. 5.3.1.

(Question 1) Thanks for suggesting this reference paper. There are two core differences between our method and the method proposed in this paper. Firstly, the meaning of the nodes in the Structural Causal Model is different: we consider two different target objects as $X$ and $Y$, while the reference paper considers the target object as $X$ and model prediction as $Y$. Secondly, our high-level considerations in confounder modeling are different. The reference paper considers co-occurring objects as confounders. While in our paper, we argue that purely modeling confounders with object-level features is insufficient, and the confounders should also include background or text information. We will add discussions about our paper and this reference paper in our revised paper.

(Question 2) Please refer to our response to weakness 1.

(Question 3) Please refer to our response to weakness 2.

(Question 4) In Sec 4.3 (line 203~217) we discuss our understanding about confounder modeling in context of multi-label image recognition. We argue that confounders should not only consider the objects with labels defined by the dataset, but also include other types of objects, image background and even image textures. Based on this understanding, we present to model the confounders by clustering the spatial features extracted by a pre-trained classification network, which may characterize confounders with object-level, texture-level and scene-level concepts. In Tab. 6, we investigate the impact of different confounder modeling methods, where our method achieves the best result.

(Question 5) Our method aims to suppress spurious label correlations and enhance causal label correlation. Therefore, if the correlation between two labels are spurious, our method can always suppress their label correlations for multi-label image recognition. We would like to highlight that this does not mean that our method can always make right prediction, but means that it can always suppress spurious label correlations to reduce the probability of error prediction caused by it.

(Question 6) How to handle noisy or incomplete labels is an important research topic in machine learning. However, this is not the focus of our paper, and our current method cannot address the partial label issue. Thanks for your suggestion, and we consider extending our method to the partial label setting as our future work.

(Question 7) We would like to highlight that pure feature decoupling using the Transformer decoder cannot reduce contextual biases. To reduce contextual biases, we explicitly model causal label correlations based on the decoupled features and confounders, with the guidance of causal intervention in causal theory.

(Limitation) On one hand, confounder modeling is a crucial step in our method, and bad choices may lead to obvious performance degradation. But, on the other hand, we would like to highlight, it remains an open question about confounder modeling for visual recognition tasks. In this work, we present our understanding about confounders (see line 203~217), and then develop a simple but effective modeling approach based on pre-trained classification network and K-means clustering. In our ablation experiments (Tab. 6), we show the advantages of our confounder modeling approach.

In short, what we would say is that for this open question (confounder modeling), we have presented our rational analysis, as well as a simple approach with experiments validating its effectiveness.

We thank reviewer NEe5 for the constructive comments and suggestions. In the following, we present our responses addressing the raised concerns. Should our rebuttal effectively address the concerns, we kindly hope you can raise your score.

(Weakness 1 and 2) Thanks for your reminding, and we will correct these typos in our revised manuscript.

(Weakness 3) To differentiate these two different fully-connected layers, we will use $f_{fc1}$ and $f_{fc2}$ in Eq. 6 and Eq. 11, respectively.

(Weakness 4) Thanks for your comments, which remind us that the current diagram for causal label correlation modeling in Fig. 4 lacks a crucial arrow to “feed” the added features into the cross-attention operation. We update our Fig. 4 (see our attached pdf file) to show this.

(Weakness 5) Thanks for your kind suggestion, and we agree that random confounders should be the simplest baseline for comparison. We conduct experiments for this random version of confounders: it leads to significant performance degradation (measured by mAP), especially on the “Exclusive” subset of COCO-Stuff. We believe that confounders should be modeled at semantic level (which cannot be achieved by random vectors), and thus present our K-means based solution upon semantic features extracted by a pre-trained classification backbone network.

(Weakness 6) We agree with you that our modeling process (as formulated in Eq. 11) should be further clarified. Please refer to our responses to Question 1 in the following.

(Question 1) Regarding the $f_{merge}$ in the first two rows in Eq 11, it represents our high-level idea that “We seek for a model to combine the information of all label-specific features $x_i$ and one potential confounder feature c to predict the logit of label $Y_j$. From the third row in Eq.11, we merge all label-specific features $x_i$ into a feature matrix $X$, and thus $f_{merge}$ can be removed.

Regarding the summation over all confounders, we thank for your kind reminding: it cannot be removed in Eq. 11. We will update Eq. 11 in our revised paper as the following.

where ${Z}\_c$ is the addition-based combination of all label-specific features ${X}$ and a confounder feature $c$, and $f\_{fc2}$ is a fully-connected layer applied upon the cross-attention feature to obtain the logit.

We provide the following pseudocode to illustrate the implementation process of Eq. 11.

Define $X \in \mathbb{R}^{N \times D}$, $C \in \mathbb{R}^{M \times D}$, $P \in \mathbb{R}^{M}$ is the label-specific features, confounders and priors.

Z = X.unsqueeze(1) + C.unsqueeze(0) # $Z \in \mathbb{R}^{N \times M \times D}$

yj = X[j, :]

y_causal_j = 0

for c in range(M):

        y_causal_j += fc(cross_atten(yj, Z[:,c,:], Z[:,c,:])) * P[c]

y_causal_j = y_causal_j.sigmoid()

(Question 2) To ensure fair comparisons, we retrain these models based on the codes released by authors.

(Question 3) Honestly, we present our results on MS-COCO for completeness, rather than for showing the advantage of our method. This is because MS-COCO generally satisfies the i.i.d assumption, while our method aims to improve practical multi-label recognition where the training and test images may not follow the i.i.d assumption, as the co-occurence between objects might change in testing. We follow the basic experimental settings in [1] which also aims to overcome contextual bias (but not from a causal intervention perspective), but also present our results under challenging cross-dataset setup. In this sense, our main experimental results are sufficient to validate the core contributions of this work: our method significantly outperforms existing methods under non-i.i.d settings, and can still achieve competitive results for i.i.d setting.

[1] Singh, Krishna Kumar, et al. "Don't judge an object by its context: Learning to overcome contextual bias." CVPR. 2020.

Thank you for your response. We must emphasize that our approach does not simply add causal intervention to Q2L. Q2L includes both a Transformer Decoder and a Transformer Encoder, and uses a specially proposed ASL Loss [1]. In contrast, without considering the causal intervention branch, we only utilize the Transformer Decoder with less parameters for feature decoupling, and train our model with general multi-label classification loss.

Furthermore, Table 8 aims for performance comparison, rather than ablation study. It does not indicate that our causal intervention method gains no performance improvement on MS-COCO. We conducted an ablation study on MS-COCO as shown in the table below (CMLL was not compared due to lack of experiments at resolution 448x448 and unavailability of code). It can be seen that our causal intervention module still provides some improvements on the MS-COCO dataset. Compared with Q2L, we achieve similar results with fewer parameters (refer to our rebuttal for Weakness 1 of Reviewer wRCW). Additionally, when compared with other methods specifically designed based on causal theory for the MS-COCO dataset, our method still achieves competitive results.

We consider pursuing higher accuracy on MS-COCO as our future work, by combining our causal intervention method with more advanced backbone networks, multi-label loss, or pre-training methods.

[1] Asymmetric loss for multi-label classification, 2020.