Rethinking the Bias of Foundation Model under Long-tailed Distribution

We greatly appreciate your detailed feedback and thoughtful suggestions. To ensure all your concerns are thoroughly addressed, we have summarized your comments in quotes below and provided our responses to each point in detail.

Weakness 1

The writing can be greatly improved. For example, the paper mentions Figures 2 and 3 in the introduction without further explaining what they mean. More explanation on figure 2 and 3 in the introduction or detailed caption for figures 2 and 3 would be preferred. Authors also refer to “D-many,” “D-Medium,” and “D-Few” without explicitly explaining what they are. Given the frequency of these terms, I would expect more details on what they means e.g., the definition and data processing etc.

Re: In the introduction, we provide a brief summary of our findings and approach. A more detailed explanation of these two figures is available in Section 4.1. To clarify, we have added further details regarding these images in the captions in the revised version.

Regarding data imbalance, we follow the approach outlined in OLTR [a] and categorize the classes into "D-Many," "D-Medium," and "D-Few." Specifically, we group classes with more than 100 training samples as "D-Many," those with 20-100 samples as "D-Medium," and those with fewer than 20 samples as "D-Few."

For parameter imbalance, given the unavailability of pre-training data, we use GLA [b] to estimate the label prior and uniformly split the classes. We designate the top 30% of classes as "P-Many," the next 30%-60% as "P-Medium," and the remaining classes as "P-Few." For example, if class $A$ in the downstream task has 1000 samples, it would be grouped as "D-Many." However, if class $A$ has a lower estimated pre-training prior, it would be also classified as "P-Few". Therefore, its performance in Fig.2 should be grouped into "D-Many" (column 1) and "P-Few" (row 3).

To further clarify these concepts, we have added additional details in the appendix.

Weakness 2

Regarding the experimental results, the improvement of the proposed method looks slightly marginal given that fine-tuning based method can also alleviate the bias. Maybe a side-by-side comparison of the improvement of the proposed method and fine-tuning based method would be beneficial. Moreoever, the results are mostly in tables, visual examples similar to those in Figure 4, that illustrate how the method reduces spurious correlations could be beneficial. To be more precise, what does the heatmap looks like after the application of the proposed method? Has the problem of partial semantic factor been alleviated from the visualization?

Re: Thank you for your valuable feedback. Indeed, the results for $M=1$ in Tab.8 can be interpreted as a fine-tuning-based approach. As $M$ increases, the performance gradually improves, highlighting the effectiveness of our method.

We have also visualized the heatmap of our method when $M=3$. As shown in Fig. 10 in the paper, in the first row, OpenCLIP primarily concentrates on the head, while MetaCLIP shifts its attention to the body. In contrast, our method successfully integrates both the head and body, yielding a more holistic understanding of the object. Similarly, in the second row, our method expands its focus to include the head, wings, and tail, offering a more comprehensive representation of the object. Finally, in the last row, our method effectively captures the entire structure of the truck, demonstrating its robustness and adaptability to diverse objects and scenarios. Our method is able to better focus on the object by eliminating the influence of the accessible semantic factors. We have included these results in the revised version of the paper.

Question 1

Could you explain a bit more about the concept of partial semantic factor and the causal graph in Section 5.1? What is a data-balanced representation, and could you provide an example of $C \rightarrow X$

Re: Thank you for your valuable feedback. In response, we have reconstructed the causal graph and introduced $C$ and $U$ as incomplete semantic factors and inaccessible semantic factors. The path $U \rightarrow X \leftarrow C$ illustrates that the dataset can be generated by considering both incomplete semantic factors and inaccessible semantic factors.

As shown in the first row of Fig. 4, CLIP is more influenced by the bird's wing, so $C$ represents the bird's wing in this case. However, we cannot fully generate the bird by relying solely on the wing (i.e., the incomplete semantic factor). We also need other inaccessible semantic factors, such as the bird's head or tail, which we represent using the variable $U$, Therefore $U \rightarrow X \leftarrow C$ indicates that the bird's wing, along with other parts of the bird, contributes to generating the bird object.

The concept of data balanced representation refers to the learned representation being unaffected by the bias introduced by imbalanced downstream data. When a model is trained without incorporating any rebalancing techniques, the learned representation is dominated by head classes, which leads to the underrepresentation of tail classes. Consequently, we denote $B$ as the balanced representation learned from imbalanced downstream data through data rebalancing methods, such as LA.

Question 2

In Section 4.1, the authors claim that although the fine-tuning-based method is able to address parameter imbalance, bias still exists; the results shown in figure 3 for Places365 looks unreasonably bad and could you provide more details on this part of experiments?

Re: Thank you for pointing this out. We conduct experiments on Places365-LT and ImageNet-LT using the CE and LA loss functions. To better investigate the influence of data imbalance and parameter imbalance, we sort the class indices based on the label prior. The x-axis in the left image of Fig. 3 represents the class indices sorted by $P_S(Y)$, while the right image shows the indices sorted by the estimated upstream label prior $\widehat{P}_P(Y)$. Since the pre-training data is inaccessible, we use the method from [b] to estimate the upstream label prior $\widehat{P}_P(Y)$.

As shown in Fig. 3, the performance of tail classes (sorted by $\widehat{P}_P(Y)$) decreases, indicating the presence of parameter imbalance. After applying the LA loss, we observe that performance also decreases, but the rate of decrease weakens. Therefore, we conclude that "fine-tuning-based methods can address parameter imbalance, but bias still exists."

Regarding the performance on Places365-LT, this dataset is inherently more challenging. Typically, we achieve about 79% overall accuracy on ImageNet-LT, but only around 50% on Places365-LT. Thus, when compared to ImageNet-LT, the performance on Places365-LT appears disproportionately poor. However, the absolute lower performance on Places365-LT does not alter our conclusion. We also observe that the LA curve lies above the zero-shot baseline (indicating that parameter imbalance is partly relieved) but still shows a decreasing trend, suggesting that bias remains.

[a] Liu, Ziwei, et al. "Large-scale long-tailed recognition in an open world." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019. [b] Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. Advances in Neural Information Processing Systems, 36, 2024.

Thank you for your insightful feedback and constructive input. We have carefully reviewed your concerns and questions and provided detailed responses to each. Below, we summarize your points in quotes, followed by our corresponding replies.

Weakness 1

The argument that C is a confounder is confusing. The model will predict Y using the representation B, it is hard to believe there is a direct causal effect of C on Y. I would suggest the authors to show the direct causal effect or confounding bias with experiments.

Re:

(1)Experimental explanation of why the incomplete semantic factors obtained from a parameter-imbalanced model are confounders. Thanks for your advice. We conduct experiments to investigate why the incomplete semantic factor acts as a confounder in the causal graph. As illustrated in the first row of Fig.4, the fine-tuned OpenCLIP model predominantly focuses on the head, whereas MetaCLIP primarily focuses on the body. Accordingly, we define $C=0$ to represent the head and $C=1$ to represent the body for this particular class.

To assess the influence of the incomplete semantic factor, we randomly select samples from this class and examine the prediction results of OpenCLIP and MetaCLIP. As shown in Fig.10, when the dog's head is occluded, OpenCLIP tends to produce lower-confidence predictions. Conversely, when the body is occluded and the head remains visible, OpenCLIP is more likely to provide higher-confidence predictions. Since both models are fine-tuned using the same principle, the observed differences in predictions can be attributed to incomplete semantic factors, thereby supporting our hypothesis.

(2)Theoretical explanation of why the incomplete semantic factors obtained from a parameter-imbalanced model are confounders. We first provide the definition of confounders in the causal framework: When a third variable $Z$ influences both $X$ and $Y$, we say that $X$ and $Y$ are confounded. Such a variable $Z$ is referred to as a confounder of $X$ and $Y$ [Pearl et al., 2009, p. 55, Def. 2.7.1 & p. 184, Sec. 6.22]. Specifically, it satisfies the fork structure $X \leftarrow Z \rightarrow Y$. This latent common cause $Z$ can create a spurious correlation between $X$ and $Y$, making the observed statistical relationship $P(Y|X)$ potentially misleading. To address this, the causal relationship $P(Y|\text{do}(X))$ is used as a replacement for the correlation $P(Y|X)$.

Secondly, we explain why the incomplete semantic factor $C$ in our problem setting can create a backdoor path $X \leftarrow C \rightarrow Y$, leading to a spurious correlation. The example in the first row in Fig.4 explains why the parameter-imbalanced model $D$ leads to an incomplete semantic factor $C$. For example, the semantic factor available to OpenCLIP is in the head of the object, while the semantic factor available to MetaCLIP is in the body of the object.

We emphasize that the semantic factors obtained due to model imbalance are incomplete. Therefore, for rigor, we have modified the causal diagram in Fig. 5 to include the inaccessible semantic factor $U$. From the perspective of data generation, $C$ and $U$ together constitute $X$, thus $X \leftarrow C$ holds. On the other hand, the path $C \rightarrow Y$ indicates that the predicted label distributions follow their own parameter imbalance. A parameter-imbalanced model, OpenCLIP, consistently predicts a "head of the object" label distribution (i.e., making more accurate predictions on test samples where the head of the object is not occluded), regardless of the test distributions. Similarly, the parameter-imbalanced model MetaCLIP exhibits a distinct "body of the object" preference across different test distributions.

The relationship between the confounding path $X \leftarrow C \rightarrow Y$ is unstable. When the training set model predicts the label with the help of the body semantics of the object, if the body of a sample of a certain class in the test example is occluded, the model will give an incorrect prediction. From this example, it can be seen that an ideal model needs to learn the causal relationship $P(Y|do(X))$ between the input and the label, rather than fitting the unstable confounding path $X \leftarrow C \rightarrow Y$, in order to achieve generalization between different distributions.

[Pearl et al., 2009] Pearl, Judea. Causality. Cambridge university press, 2009.

Weakness 2

The proposed method utilized backdoor adjustment, which is also used in some previous work in the literature of long tail classification such as [1]. The authors did not point out the difference.

Re: In causal inference, the backdoor adjustment is a crucial tool for identifying and controlling confounding variables to enable causal estimation. It primarily addresses the challenge of estimating causal effects in observational data by adjusting for specific variables.

In [1], the authors analyze the SGD momentum as a confounder and apply the backdoor adjustment to mitigate its influence. In our method, we treat the incomplete semantic factor as the confounder and disentangle its impact. The distinctions are as follows:

(1)Motivation: Our method investigates how the bias of foundation models influences imbalanced downstream tasks. By contrast, in [1], the authors examine how the momentum in the SGD optimizer affects long-tailed learning.

(2)Confounder: We consider the incomplete semantic factor as the confounder, while in [1], the authors identify momentum as the confounder. Consequently, the causal graphs in these two cases are distinct.

(3)Method: We implement the computation of $P(Y|do(X))$ through diverse scenarios and methods. In [1], the authors consider the imbalance can lead to momentum bias, and calculate $P(Y|x,m)$ through the de-confounded training to relieve it. In contrast, our method focuses on incomplete semantic factors, computing $P(Y|b,c)$ by leveraging different foundation models to extract varied semantic information, thereby ensuring the model's inference is not constrained to incomplete semantic information.

Question 1

In Table 8, it seems that as M increases, performance improves. Do the authors think this trend can persist for larger values of M? Is there any theory or insight that can support this observed trend?

Re: Theoretically, we analyze the problem from the perspective of the Completeness of the Available Value Set for the Adjustment Variable. Here, our adjustment variable is $C$. Technically, the possible values of $C$ are infinite and for backdoor adjustment, the sum should account for every possible domain of $C$. A fine-grained definition of $C$ would lead to enormous computational costs, which makes it impractical for training. When pre-training datasets correspond to diverse semantic information, an increase in the number of available semantic factors (i.e., a higher $M$) generally enhances performance [a]. However, when $M$ becomes sufficiently large, indicating that potential confounders are more comprehensively addressed, further increasing $M$ yields slight performance improvements. Additionally, a larger $M$ incurs higher computational costs. Therefore, determining the optimal value of $M$ requires balancing performance gains with computational efficiency.

Question 2

In Table 7, why can the proposed method also improve performance in D-Many? I suppose such methods would sacrifice the performance on D-Many for improvements on long tail classes.

Re: Thank you for highlighting this point. Improving performance in tail classes does not necessarily come at the expense of performance in head classes. For instance, RIDE [b] demonstrates overall performance improvements across the "Many," "Medium," and "Few" classes compared to other rebalancing-based methods on the iNaturalist2018 dataset.

Additionally, our method emphasizes identifying all available semantic factors across different classes. The "D-Many" also faces a spurious relationship due to incomplete semantic factors that do not depend on downstream data imbalance. Therefore, our method not only supports the classification of tail classes but also enhances the performance of head classes. As a result, our method achieves an overall improvement in classification performance.

[a] Li, Jiangmeng, et al. "Hierarchical Topology Isomorphism Expertise Embedded Graph Contrastive Learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 12. 2024. [b] Wang, Xudong, et al. "Long-tailed recognition by routing diverse distribution-aware experts." arXiv preprint arXiv:2010.01809 (2020).

Thank you for your reply. Our backdoor adjustment aims to estimate the probability $P(Y \mid do(X))$, and we present the estimated values as follows. Additionally, to estimate $P(Y \mid X)$, we follow the previous work [a] and train the model from scratch to directly estimate the probability. The results are presented below. For clarity, we refer to the images in Fig. 8 from left to right as Fig. 8(1), Fig. 8(2), Fig. 8(3), and Fig. 8(4).

From these results, we find that $P(Y \mid do(X)) \ne P(Y \mid X)$ and confirm the existence of confounding bias.

If you have any other questions, please feel free to let me know, and I would be happy to respond.

[a] Liu, Ziwei, et al. "Large-scale long-tailed recognition in an open world." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

(1)Experimental explanation of why the incomplete semantic factors obtained from a parameter-imbalanced model are confounders.

Thanks for the reply.

The experiment only illustrates the correlation between $C$ and confidence scores. To show the existence of confounding bias, by definition, the authors need to show concretely that $P(Y|do(X)) \ne P(Y|X)$.

Thank you for your reply. We constructed a dataset consisting of six classes: 'airplane,' 'automobile,' 'bird,' 'cat,' 'dog,' and 'truck.' For each class, we define distinct incomplete semantic factors, as shown below, where $C$ denotes the incomplete semantic factor:

For example, the term "airplane head" refers to the front part of the aircraft, including the cockpit, while "airplane tail" represents the rear section, including the winglets. To construct the dataset, we used Stable Diffusion [a] by providing prompts in the format: "a photo of a {class name}'s {head or tail}." For instance, to generate samples for the airplane class, we used the prompts "a photo of an airplane's head" and "a photo of an airplane's tail," respectively, ensuring sufficient samples for each semantic factor.

The training dataset was created under the following conditions:

1. For each incomplete semantic factor, 50 samples were generated. This dataset is referred to as $A_{train}$.

2. For $C=0$, 90 samples were generated, and for $C=1$, 10 samples were generated. This dataset is referred to as $B_{train}$.

For evaluation, the test dataset was constructed under the following conditions:

1. For each incomplete semantic factor, 50 samples were generated. This dataset is referred to as $A_{test}$.

2. For $C=0$, 90 samples were generated, and for $C=1$, 10 samples were generated. This dataset is referred to as $B_{test}$.

We use ResNet-32 [b] as the backbone and train it from scratch on $A_{{train}}$ and $B_{{train}}$, respectively. The trained models are then evaluated on $A_{\text{test}}$ and $B_{\text{test}}$, respectively. As shown in Fig.12, when the model is trained on $B_{train}$ but evaluated on $A_{test}$, we find the performance drops (compared with testing on $B_{test}$), which indicates the confounder influences the model and cannot make right predictions. Our findings can be summarized as follows:

1. The model trained on $A_{{train}}$, when used for inference, can be interpreted as estimating $P(Y \mid {do}(X))$. We observe that the model trained on $A_{{train}}$ performs similarly on both $A_{{test}}$ and $B_{{test}}$. This result indicates that the trained model does not learn the correlation.

2. The model trained on $B_{{train}}$, when used for inference, can be interpreted as estimating $P(Y \mid X)$. We observe that the model trained on $B_{{train}}$ exhibits different performance between $A_{{test}}$ and $B_{{test}}$. Considering that if the trained model is not affected by incomplete semantic factors, it should present the same performance between $A_{{test}}$ and $B_{{test}}$. However, we find the evaluation results are different, verifying the confounder's influence.

From these experiments, we verify that $C$ is a confounder. More details are in the revised paper.

[a]Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[b]He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

The model trained on $A_{train}$, when used for inference, can be interpreted as estimating $P(Y|do(X))$.

This does not sound correct to me. In $A_{train}$ you do have P(C=1)=P(C=0), however, there is no intervention on $X$, that means $X$ is still causally affected by $C$ in $A_{train}$. The correct way to compute $P(Y|do(X))$ is to maintain $C$ and intervene on $X$.

Thanks for the reply.

Note that, in your experiments, the datasets $A_{train}$ and $B_{train}$ are generated from the same causal graph/data-generating process. The difference is only the number of samples for $P(C=1)$ and $P(C=0)$. This should not affect the identification of the interventional distribution $P(Y|do(X))$, which is still unclear to me (I would assume that the authors rely on backdoor criteria to estimate $P(Y|do(X)) = \sum_c P(Y|X,C=c)P(C=c)$. In both balanced and imbalanced cases, we should be able to estimate both $P(Y|X,C=c)$ and $P(C=c)$ from the data, assuming strong ignorability. While imbalance or not only leads to different levels of estimation errors of $P(Y|X,C=c)$ and $P(C=c)$, which should not affect the identification of $P(Y|do(X))$ itself with same identification and estimation process.

I would be happy to learn any theorem in Pearl's book [b] which claims that training a model on a balanced observational dataset with $P(C=1)=P(C=0)$ would lead to causal identification and unbiased estimate of an interventional distribution while training on an imbalanced dataset with $P(C=1)\ne P(C=0)$ cannot achieve the same.

we can ensure that $P(C=1)=P(C=0)$, meaning that males and females are equally represented in both the treated and untreated groups.

This claim is not correct. $P(C=1)=P(C=0)$ is a necessary but not sufficient condition for that claim males and females are equally represented in both the treated and untreated groups, which is equivalent to $P(C=1|X=x)=P(C=0|X=x), x=\{0,1\}$.

BTW, Simpson's paradox is a fact or an observation that $P(Y=y|X)>P(Y=y'|X)$ but $P(Y=y|X,C)<P(Y=y'|X,C)$ can happen, it is not a theorem. So applying a fact to your example does not make much sense to me.

1. I agree with your description of the example from [Pearl, 2016].
2. If $P(X=1|C=c)=P(X=0|C=c)$ for all $c$, then there is no causal effect of $C$ on $X$, leading to randomization. The key is, in this case, $P(Y|do(X))=P(Y|X)$, so you can estimate $P(Y|do(X))$ without using the backdoor criterion. Under this setting, you might be able to show there exists confounding bias by comparing $P(Y|X)$ in this setting and $P(Y|X)$ with $P(X=1|C=c) \ne P(X=0|C=c)$.

Thanks for your in-depth discussion with us. We are very pleased that you agree with our description of the example. Regarding our experiments, we composed two training sets ($A_{train}$ and $B_{train}$) and two test sets ($A_{test}$ and $B_{test}$) .

1. For each incomplete semantic factor, 50 samples were generated. This dataset is referred to as $A_{train}$. ($P(X|C=0) = P(X|C=1)$)

2. For $C=0$, 90 samples were generated, and for $C=1$, 10 samples were generated. This dataset is referred to as $B_{train}$. ($P(X|C=0) \neq P(X|C=1)$)

The model trained on $A_{{train}}$, when used for inference, can be interpreted as estimating $P(Y \mid {do}(X))$, since $P(X|C=0) = P(X|C=1)$. The model trained on $B_{{train}}$, when used for inference, can be interpreted as estimating $P(Y \mid X)$, since $P(X|C=0) \neq P(X|C=1)$.

For evaluation, the test datasets are constructed under the following conditions:

1. For each incomplete semantic factor, 50 samples were generated. This dataset is referred to as $A_{test}$.

2. For $C=0$, 90 samples were generated, and for $C=1$, 10 samples were generated. This dataset is referred to as $B_{test}$.

We conducted four groups of experiments, as outlined below:

a) The model is trained on $A_{{train}}$ and evaluate on $A_{{test}}$.

b) The model is trained on $A_{{train}}$ and evaluate on $B_{{test}}$.

c) The model is trained on $B_{{train}}$ and evaluate on $A_{{test}}$.

d) The model is trained on $B_{{train}}$ and evaluate on $B_{{test}}$.

As shown in Fig.12 in the revised paper, we observe that the model trained on $A_{{train}}$ performs similarly on both $A_{{test}}$ and $B_{{test}}$. This result indicates that the trained model does not learn the correlation.

However, when the model is trained on $B_{{train}}$, we observe that the model trained on $B_{{train}}$ exhibits different performance between $A_{{test}}$ and $B_{{test}}$. Considering that if the trained model is not affected by incomplete semantic factors, it should present the same performance between $A_{{test}}$ and $B_{{test}}$. However, we find the evaluation results are different, verifying the confounder's influence.

Under this setting, you might be able to show there exists confounding bias by comparing $P(Y|X)$ in this setting and P(Y|X) with $P(X=1|C=c) \neq P(X=0|C=c)$.

Moreover, when evaluated on the $A_{{test}}$, we find that the model trained on $A_{{train}}$ achieves better performance compared with the model trained on $B_{{train}}$. The reason is we constitute $A_{{train}}$ following $P(X=1|C=c) = P(X=0|C=c)$ but $B_{{train}}$ following $P(X=1|C=c) \neq P(X=0|C=c)$. By these experiments, we show "there exists confounding bias by comparing $P(Y|X)$ in this setting and P(Y|X) with $P(X=1|C=c) \neq P(X=0|C=c)$."

Thank you once again for your time and insightful review. We do believe your valuable comments are very important to us. If you have any further questions or require additional clarification or experiments, please don't hesitate to reach out. We would be more than happy to provide whatever is needed to ensure the highest quality of this work.

Question 2

Can you provide theoretical justification for why semantic factors should be treated as confounders in the causal framework?

Re: As mentioned in Weakness 1, we first provide the definition of confounders in the causal framework: When a third variable $Z$ influences both $X$ and $Y$, we say that $X$ and $Y$ are confounded. Such a variable $Z$ is referred to as a confounder of $X$ and $Y$ [Pearl et al., 2009, p. 55, Def. 2.7.1 & p. 184, Sec. 6.22]. Specifically, it satisfies the fork structure $X \leftarrow Z \rightarrow Y$. This latent common cause $Z$ can create a spurious correlation between $X$ and $Y$, making the observed statistical relationship $P(Y|X)$ potentially misleading. To address this, the causal relationship $P(Y|\text{do}(X))$ is used as a replacement for the correlation $P(Y|X)$.

Secondly, we explain why the incomplete semantic factor $C$ in our problem setting can create a backdoor path $X \leftarrow C \rightarrow Y$, leading to a spurious correlation. The example in the first row in Fig.4 explains why the parameter-imbalanced model $D$ leads to an incomplete semantic factor $C$. For example, the semantic factor available to OpenCLIP is in the head of the object, while the semantic factor available to MetaCLIP is in the body of the object.

We emphasize that the semantic factors obtained due to model imbalance are incomplete. Therefore, for rigor, we have modified the causal diagram in Fig. 5 to include the inaccessible semantic factor $U$. From the perspective of data generation, $C$ and $U$ together constitute $X$, thus $X \leftarrow C$ holds. On the other hand, the path $C \rightarrow Y$ indicates that the predicted label distributions follow their own parameter imbalance. A parameter-imbalanced model, OpenCLIP, consistently predicts a "head of the object" label distribution (i.e., making more accurate predictions on test samples where the head of the object is not occluded), regardless of the test distributions. Similarly, the parameter-imbalanced model MetaCLIP exhibits a distinct "body of the object" preference across different test distributions.

The relationship between the confounding path $X \leftarrow C \rightarrow Y$ is unstable. When the training set model predicts the label with the help of the body semantics of the object, if the body of a sample of a certain class in the test example is occluded, the model will give an incorrect prediction. From this example, it can be seen that an ideal model needs to learn the causal relationship $P(Y|do(X))$ between the input and the label, rather than fitting the unstable confounding path $X \leftarrow C \rightarrow Y$, in order to achieve generalization between different distributions.

Question 3

What is the computational overhead of the proposed method compared to standard fine-tuning?

Re: See the response for the Weakness 3

Question 4

How does the method perform when the degree of imbalance varies between pre-training and fine-tuning?

Re: Thank you for your insightful advice. In our paper, we address both data imbalance and parameter imbalance simultaneously, with analyses primarily based on commonly used datasets. To better understand the impact of parameter imbalance, we consider the following scenarios: (1) Consistency: The data imbalance and parameter imbalance are consistent, where the head classes grouped by the data imbalance are also the head classes grouped by the parameter imbalance. (2) Reverse: The data imbalance and parameter imbalance are reversed, where the head classes grouped by the data imbalance are the tail classes grouped by the parameter imbalance. (3)Balance: The downstream data is balanced. To simulate these scenarios, we construct additional imbalanced datasets by downsampling the Places365 dataset, as illustrated in Fig. 7. We evaluate performance using the Places365-LT balanced test set. To reduce the influence of randomness, we create three datasets for each scenario and report the average performance across them.

The results, shown in Tab. 13, indicate that when the downstream data is balanced, the model achieves the best performance, as this setup aligns most closely with the distribution of the downstream task. Interestingly, the Reverse scenario outperforms the Consistency scenario. In the Reverse setting, the tail classes (in terms of parameter imbalance) can be viewed as head classes (in terms of data imbalance), suggesting that parameter imbalance can be mitigated by increasing the number of training samples for tail classes. However, in real-world applications, collecting sufficient data for tail classes is often challenging, and simply adding more data is not a practical solution to this issue.

[a] Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. Advances in Neural Information Processing Systems, 36, 2024.

Thanks for your valuable comments and suggestions! We have carefully addressed all your weaknesses and questions. In the following, your weaknesses and questions are summarized in quotes, followed by our point-by-point responses.

Weakness 1

The definition and quantification of "parameter imbalance" lack rigorous theoretical justification. The causal relationship between pre-training data imbalance and parameter imbalance is not thoroughly explained. The selection of semantic factors as confounders needs stronger theoretical support.

Re:

(1)Theoretical justification. Thank you for pointing this out. In the revised paper Sec.4.1, we have added a theoretical definition of parameter imbalance to help readers better understand the concept. Briefly, we use the imbalanced factor (IF) to quantify the extent to which foundation models are influenced by different pre-training datasets.

(2)Causal relationship between pre-training data and parameter imbalance. Since the pre-training data of the foundation model is inaccessible, its influence is primarily reflected in the pre-trained weights or parameters, which we refer to as parameter imbalance. Consequently, we describe the imbalance in pre-training data as the parameter imbalance when adapting the model to downstream tasks. In the revised paper, we also provide a theoretical justification for parameter imbalance to help readers better understand this concept.

(3)Confounder. We imply that the incomplete semantic factor $C$, caused by parameter imbalance $D$, serves as a confounder. We will provide theoretical support from two aspects: 1) the theoretical definition of confounders, and 2) the spurious correlation induced by the backdoor path $X \leftarrow C \rightarrow Y$.

(3.1)Theoretical definition of confounders. When a third variable $Z$ influences both $X$ and $Y$, we say that $X$ and $Y$ are confounded. Such a variable $Z$ is referred to as a confounder of $X$ and $Y$ [Pearl et al., 2009, p. 55, Def. 2.7.1 & p. 184, Sec. 6.22]. Specifically, it satisfies the fork structure $X \leftarrow Z \rightarrow Y$. This latent common cause $Z$ can create a spurious correlation between $X$ and $Y$, making the observed statistical relationship $P(Y|X)$ potentially misleading. To address this, the causal relationship $P(Y|\text{do}(X))$ is used as a replacement for the correlation $P(Y|X)$.

[Pearl et al., 2009] Pearl, Judea. Causality. Cambridge university press, 2009.

(3.2) Why are the incomplete semantic factors derived from a parameter imbalance model confounders. The example in the first row in Fig.4 explains why the parameter-imbalanced model $D$ leads to an incomplete semantic factor $C$. For example, the semantic factor available to OpenCLIP is in the head of the object, while the semantic factor available to MetaCLIP is in the body of the object. The relationship between the confounding path $X \leftarrow C \rightarrow Y$ is unstable. When the training set model predicts the label with the help of the body semantics of the object, if the body of a sample of a certain class in the test example is occluded, the model will give an incorrect prediction. From this example, it can be seen that an ideal model needs to learn the causal relationship $P(Y|do(X))$ between the input and the label, rather than fitting the unstable confounding path $X \leftarrow C \rightarrow Y$, in order to achieve generalization between different distributions.

Weakness 2

The claim that parameter imbalance has greater impact than data imbalance needs more empirical evidence.

Re: In Fig. 3, the class indices in the left image are sorted based on the label prior of the downstream data, while the class indices in the right image are sorted based on the estimated label prior of the pre-training data. (Since the pre-training data is inaccessible, we use GLA [a] to estimate the label prior of the pre-training data.) After applying Logit Adjustment, we observe that the downward trend in the performance curve is mitigated, indicating that data imbalance can be effectively addressed. However, the right image still exhibits a downward trend, suggesting that parameter imbalance persists. Given that data imbalance can be alleviated while parameter imbalance remains unresolved, we conclude that parameter imbalance has a greater impact than data imbalance.

To further support this conclusion, we train a model from scratch on ImageNet-LT using ResNet-50. Since the parameters of ResNet-50 are randomly initialized, the trained model is only influenced by data imbalance. After training, we calculate the accuracy of each class and sort the classes based on the estimated label prior of the pre-training data, $\widehat{P}_P(Y)$. Because parameter imbalance does not affect the trained model in this setup, the performance curve after sorting should not display a downward trend. As shown in Fig.9, this result validates our argument. Thus, in this comparative experiment, the downward trend observed in the right image of Fig.3 is attributable to the imbalance in the pre-training data, further verifying the impact of parameter imbalance.

Weakness 3

Incomplete ablation studies for key components and lack of analysis on computational overhead and efficiency.

Re: Thank you for your advice. The key component of our method is the hyperparameter $M$. As shown in Tab.8, our experiments demonstrate that increasing $M$ improves performance. However, as shown in the following, larger values of $M$ also result in higher computational costs, measured in FLOPs. This trade-off underscores the importance of selecting an optimal $M$ to balance performance gains and computational efficiency.

Question 1

How do you quantify parameter imbalance without access to the pre-training data? What metrics are used?

Re: Although we cannot access the pre-training data directly, we can use a validation set to estimate the label frequency indirectly. Following GLA [a], we estimate the label frequency for different datasets. Therefore, we can also use the label frequency and imbalanced factor (IF) as the metric to evaluate parameter imbalance.

We sincerely appreciate your thoughtful feedback and constructive suggestions. To address your concerns comprehensively, we have carefully reviewed each point you raised. In the following, your weaknesses and questions are summarized in quotes, followed by our point-by-point responses.

Weakness 1

Although the introduced problem is important, the solution is kind of trivial. The simple combination of different models could partially resolve the problem, while each of the foundation models contains a large number of parameters, which is not applicable.

Re: Thanks for your advice. As shown below, larger values of $M$ allow for the discovery of more incomplete semantic factors, but they also lead to significantly higher computational expenses. This trade-off highlights the importance of selecting an optimal $M$ that balances performance gains and computational efficiency.

Our method is built on parameter-efficient fine-tuning and we need to store the parameters of multiple foundation models (CLIP, OpenCLIP, MetaCLIP). However, for different downstream tasks (ImageNet-LT, Places365-LT, iNaturalist2018), we only retain a single copy of the foundation model parameters and store additional task-specific parameters for different downstream tasks separately. Since the task-specific parameters require minimal storage, the additional costs introduced by our approach are relatively negligible. We have provided a detailed discussion of this limitation in the revised paper.

Question 1

In Fig. 3, how is the right picture plotted? How is the parameter imbalance computed for each sample?

Re: The x-axis in the left image of Fig.3 represents the class indices sorted according to the downstream label prior $P_S(Y)$ while the x-axis in the right image is sorted based on the estimated upstream label prior $\widehat{P}_P(Y)$. Since the pre-training data is inaccessible, we adopt the method proposed in [a] to estimate the upstream label prior $\widehat{P}_P(Y)$.

Question 2

In equation. 7, how is $P(Y=y|b, c)$ implemented in your experiments. Do you run each model once and generate a $P(Y=y|b, c)$ and how these models are jointly trained (or fine-tuned)?

Re: Yes, we run each model once and generate $P(Y=y|b, c)$. During the inference, we use our backdoor adjustment to ensemble different fine-tuned models (incomplete semantic factors) and make final predictions.

We also experiment with jointly training multiple foundation models to address the influence of incomplete semantic factors during the training phase. For details, we ensemble different fine-tuned models during the training phase. However, as shown in our results, this approach did not yield significant performance improvements. Joint training did not provide substantial additional benefits compared to our proposed method.

Question 3

In lines 184-195, it mentioned that " $P_P(Y)$ is inaccessible and we cannot use Eq. (1)". How is $P_P(Y)$ related to Eq. (1) where the symbol is not mentioned.

Re: We denote the $P_P(Y)$ as the label prior of the pre-training data to train the foundation model, and $\widehat{P}_P(Y)$ as the estimated label prior via GLA. Eq.1 shows how to bridge the gap between source distribution $S$ and target distribution $T$. Therefore, when we adapt this to the pre-trained distribution $P$, the equation should be changed as follows.

$P_{T}(Y=y \mid x) = \frac{P_{T}(x \mid Y=y) P_{T}(Y=y)}{P_{T}(x)} \propto P_{P}(x \mid Y=y)P_{T}(Y=y) \propto \frac{P_{T}(Y=y)}{P_{P}(Y=y)}P_{P}(Y=y \mid x)$

[a] Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. Advances in Neural Information Processing Systems, 36, 2024.