Continual Learning in the Presence of Spurious Correlations: Analyses and a Simple Baseline

We thank Reviewer C1FY for valuable feedback. The followings are our replies to the comments.

Weakness

W1. novelty.

Thank you for your comment, but we contend that while our problem setting involves the combination of the two topics, our key findings do not solely derive from the mere consideration of the two topics simultaneously. Specifically, most existing works on dataset bias only focus on offline learning settings. Conversely, most CL works overlook the issue of bias in CL. Consequently, there remain unpredictable findings in bias-aware CL scenarios, such as whether some factors may make dataset bias of a certain task in CL more severe or if applying existing CL and debiasing methods adequately addresses bias issues in CL.

Our paper conducts extensive experimental analysis to explore these considerations. We offer substantial experimental evidence in [Section 4,5] showing the bias transfer by typical CL methods. Moreover, the results for performance comparison in [Section 6] show the ineffectiveness of typical CL methods or naive combinations between CL and debiasing methods in bias-aware CL settings. At this point, our key findings provide a novel and significant perspective for many CL research communities.

Together with our empirical findings, we offer a standardized evaluation protocol as a tool for developing more advanced bias-aware CL methods. We emphasize that we propose BGS to serve as a baseline method when assessing a new bias-aware method using our protocol. Hence, for BGS, we argue that prioritizing simple implementation, performance and flexibility may be more crucial than pursuing novelty in method design.

W2-1. synthetic.

Skewing vision benchmark datasets (e.g., CIFAR, MNIST) with color is a conventional way employed in numerous debiasing studies [A1, A2]. Following these prior works, we constructed Split CIFAR-100S. While it seems to be quite synthetic, we argue that we can obtain valuable insights from experimental results on Split CIFAR-100S. Furthermore, to complement these synthetic settings, we additionally use CelebA, FMoW, and NLP datasets that contain a realistic gender bias (for results on the two latter datasets, refer to replies for Reviewer UUYC and 6w81, respectively). Hence, we assert that our synthetic color bias setting should not be perceived as a weakness, rather, it can diversify and enrich our experimental results.

W2-2. subjectiveness of bias

We skewed each task of Split-CIFAR100S with color, such that the training dataset of each task predominantly consists of gray images for the first five classes and color images for the remaining five classes. Namely, we create color bias, that is, spurious correlation between group and class labels. As a result, a biased model trained from the color-skewed training dataset would tend to predict gray images as one of the first five classes and vice versa. This shows our bias setting for Split-CIFAR100S mimics real-world spurious correlation that exists such as in face recognition or toxicity classification.

Question N/A

References [A1] Wang et al., Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation, CVPR, 2020. [A2] Nam et al., Learning from Failure: Training Debiased Classifier from Biased Classifier, NeurIPS, 2020.

We thank Reviewer A8St for the positive comments regarding the motivation of our research and various perspectives. We also thank you for addressing the limitations of our experimental settings and method. The followings are our replies to the comments.

Weakness

W1

We feel confident that we have presented a clear overview of the paper in both the abstract and the introduction. Would you kindly give us your feedback on any specific sentences that may be unclear, or more concrete ways for how we improve the clarity/engagement of our abstract and introduction further? We will be eager to reply to any concrete comments, but it is very hard for us to reply to the current comment.

W2

We appreciate your feedback on the points we have overlooked. In [Section 3.1], we clarified our use of DCA as a surrogate metric for BMR in cases where BMR is unavailable. To elaborate further, we define the degree of bias in a model as its sensitivity to group attributes in predictions of the model. Ideally, BMR serves as an optimal bias metric by comparing predictions between original and bias-flipped samples. However, if such counterfactual samples are unavailable, as in CelebA dataset, we indirectly gauge the degree of bias in the model by examining the accuracy difference between bias-aligned and bias-conflicted groups, i.e., DCA.

However, it is important to note that DCA can be somewhat misleading as a bias metric, especially in some extreme cases in which the overall accuracy is very low. For instance, a completely random classifier, with identical accuracy for each group (1 over the number of classes), results in a DCA of 0, but the model may indeed have bias so that the resulting BMR can be high. Therefore, it is preferable to use BMR as the primary bias metric.

The table below compares BMR and DCA for the results on Split CIFAR-100S, reported in [Table 1, Section 6]. We observe that DCA results mostly have a similar trend to BMR results, suggesting that DCA can serve as the surrogate metric of BMR. However, we note that Fine-tuning results in very low accuracy, hence, the DCA becomes unexpectedly low while the corresponding BMR is high. Moreover, DCA even worsens when BGS is applied due to the slight accuracy improvement, while BMR shows improvement. This result shows that DCA can sometimes be misleading as a bias metric. We will add this discussion in the Appendix of the final version.

W3

We have already reported the results of three additional methods, GDumb, MAS, and DER in [Appendix E], which are more recently developed CL methods. We would greatly appreciate your more concrete feedback on additional baselines to add.

W4

Our main contribution lies in highlighting the issue of dataset bias in CL, rather than introducing a new state-of-the-art bias-aware CL method, because many CL researchers may have inadvertently overlooked this critical issue. To that end, we thoroughly demonstrated that the bias transfer indeed exists and is significant in CL, emphasizing the importance of addressing the bias transfer issue in CL. We then offered a standardized evaluation protocol and BGS for foundational tools for developing new bias-aware CL methods. In this context, we believe that prioritizing simplicity, flexibility and efficiency of BGS is more essential than pursuing novelty, as the development of a highly innovative method is beyond the scope of our paper.

W5

We believe that we provided in-depth analytical results for the issue of bias in CL beyond naively exploring it. Specifically, we showed that in various bias-aware CL scenarios, bias transfer exists and is influenced by Normalized $\mathcal{F-I}$ values by analyzing the prediction and representation of CL models. We do not believe a paper is only worthy of publication if it surprises. While the intuition of spurious correlations affecting continual learning may have existed to some extent, we believe our paper shows substantial evidence of the idea via thorough and systematic experiments and analyses.

We thank Reviewer 6w81 for the supportive comments, e.g., “well-motivated and extensive studies for the bias transfer in CL and impactful results”. We also thank you for pointing out the limitations of our experimental settings and method. The followings are our replies to the comments on the weakness.

Weakness

We first note that [A1-A4] are all cited in the original manuscript, and we are referring them here again for clarity.

W1

We have already addressed the key differences between our paper and existing related works throughout the manuscript. As we outlined in [Section 1], [A1] empirically investigated the forward transfer of a bias when a model pre-trained on a biased dataset is fine-tuned with a downstream task. However, their findings do not cover the backward transfer of bias and the continual learning of the task sequences. Furthermore, as stated in [Section 2] and [Appendix D], [A2, A3] pose the issues of dataset bias in CL. Their analysis, however, only focused on the issue of local spurious features (LSF) in Domain-IL and lacked a comprehensive analysis of the transferability of bias.

We believe the above comparison of our work with others, which is already clearly written in the original manuscript, is quite explicit, and it underscores the novelty of our paper. If the reviewer lets us know further concrete questions regarding the novelty of our paper, we will be glad and committed to addressing them in further responses.

W2

We appreciate your comment on sample diversity. First of all, we would like to clarify a potential misunderstanding regarding our experimental settings in [Section 6]. Specifically, for examplar-based CL methods, including BGS, we used 1000 or 2000 examples, not 2000 or 4000 samples as the reviewer has mentioned. These sample quantities fall within the range commonly considered by many CL studies, e.g., [A4].

Meanwhile, to show that our findings of bias transfer are valid even for more realistic datasets, we conducted further experiments on two-task CL scenarios involving a sequence of two kinds of natural language datasets: MultiNLI and WaNLI. The task of both datasets is to determine if a given hypothesis is entailed by, neutral with, or contradicts a given premise. It is widely known that the contradiction class is spuriously correlated with the presence of negation words in hypothesis sentences. Thus, we set the group label as the presence of negation in the sentence. We then control the skewness between the class label and group label to investigate the bias transfer.

The results, as depicted in [Figure F.2, Appendix F] in the revised version, indicate that the bias transfer is evident even in more realistic CL scenarios based on NLP datasets. Additionally, it is clearly shown that the degree of bias transfer is influenced by Normalized $\mathcal{F}-\mathcal{I}$, in line with our findings on other datasets in the original manuscript.

W3

Our primary contribution lies in highlighting the bias issue in CL. In this context, we have allocated a relatively limited space for explaining the BGS method since our intention to introduce BGS is to offer a sensible and strong baseline for developing a more advanced bias-aware CL method. Furthermore, we believe that we have provided detailed descriptions of the algorithm, performance, and limitations of BGS in [Sections 6.2 and 6.3]. We would greatly appreciate the reviewer for letting us know any specific aspects that need further clarification, and we are fully committed to incorporating them in the further response and the final version.

W4

Throughout the paper, we believe we have already demonstrated the underlying mechanism of bias transfer --- specifically, we demonstrated that the degree to which a CL method focuses on stability (resp. plasticity) turns out to be the main factor in forward (resp. backward) transfer of bias. That is, if the previous tasks contain some dataset bias, the bias obtained in the previous task is preserved when Normalized $\mathcal{F}-\mathcal{I}$ is low (i.e., when the stability is more prioritized), and it negatively influences future tasks (i.e., the forward transfer of the bias occurs). The mechanism for the backward transfer of bias can be described similarly with the concept of plasticity.

References [A1] Salman et al., When does Bias Transfer in Transfer Learning?, arXiv preprint arXiv:2207.02842, 2022.

[A2] Lesort, Timothée., Spurious Features in Continual Learning. AAAI Bridge Program on Continual Causality, PMLR, 2023.

[A3] Jeon et al., Learning without Prejudices: Continual Unbiased Learning via Benign and Malignant Forgetting, ICLR, 2023.

[A4] Prabhu et al., GDumb: A Simple Approach that Questions Our Progress in Continual Learning, ECCV, 2020.

We thank Reviewer UUYC for the positive comments, e.g., “well-motivated and thorough studies for the bias transfer in CL and interesting results”. We also thank you for pointing out the limitations of our experimental settings and method. The followings are our replies to the comments.

Weakness

W1

Thank you for suggesting to add experiments on real-world datasets. Following the suggestion, we conducted further experiments on bias transfer for the two-task CL scenarios on the more realistic FMoW dataset. We set the class and group labels as the "building" category and "regions" of images, respectively. We construct two tasks depending on the year in which images were taken (please refer to [Appendix F] in the revised version for a detailed description of our two-task CL scenarios). Our observations from [Figure F.1 in Appendix F] again indicate the consistent trends of the bias transfer. Namely, both the forward and backward transfers of the bias exist and those transfers are affected by the relative focus on plasticity and stability. In the final version, we plan to include additional results for the longer sequences on the FMoW dataset.

W2

Thank you for your suggestion. Following your suggestion, we carried on additional experiments in two-task CL scenarios when using a pre-trained model. With the same setting as experiments for forward transfer in [Section 3], we compare the two types of CL scenarios; the bias level of the first task is 0 or 6, while the bias level of the second task is fixed as 0. The table below compares BMRs of T$_2$ on Split ImageNet-100C after learning the pre-trained model in the two CL scenarios. Due to the time limit, we reported the results for only two kinds of CL methods (ER and ER+LWF) with only two hyperparameters (the memory sizes corresponding to 10% and 30% of the T$_1$ training dataset). Although normalized $\mathcal{F}-\mathcal{I}$ is not perfectly aligned for each memory size, BMR gaps in the table imply that bias transfer occurs and is affected by normalized $\mathcal{F}-\mathcal{I}$ values even when using a pre-trained model.

W3

We sincerely appreciate your constructive feedback. We have comprehensively reviewed the papers you recommended but found that constructing a theoretical analysis on the bias transfer in continual learning seems very challenging, particularly for the short rebuttal period. The main reason for such a challenge is that we currently do not even have a rigorous theory on continual learning or knowledge transfer. Consequently, we decided to defer the theoretical analyses on our work to future work, but we will definitely pursue the direction after finalizing the current paper.

Question

Q1

Thank you for the detailed question. We note that even if the bias level is set to 0 (i.e., the skew-ratio of the dataset is 0.5), the BMR could still be non-zero --- in fact, we observed that BMR for a model trained on a Split CIFAR-100S dataset with bias level of 0 was 11%. This would be the reason for the bias transfer occurring in row 1 of [Figure 4, Section 4].

Q2

As we mentioned in [Section 4.2], for PackNet, the backward transfer does not occur since it freezes the parameters learned from previous tasks.

Q3

ER mitigates the forgetting of previous tasks by replaying samples stored in the examplar memory, while EWC and PackNet directly enforce preserving parameters of neural networks. Thus, since ER is indirect in transferring a model's knowledge to future tasks, the forward transfer of bias also occurs less for ER than EWC and PackNet, as seen in [Figure 5, Section 5]. However, we note that the lowest BMR for ER observed in [Figure 5, Section 5] does not necessarily imply superior performance in achieving debiased models. This is because the results are not acquired from the optimal hyperparameter in terms of BMR. For a more comprehensive performance comparison, refer to the results presented in [Table 1, Section 6].