CoSDA: Continual Source-Free Domain Adaptation

Thanks for your time and valuable reviews, here are our responses:

[W1] There setting of continual SFDA is meaningful for the following reasons:

1. Ease of massive data annotation. Note that in our setting, the label of data is only accessible in the first domain(i.e. source domain). The need for annotation on target domains is eliminated, which saves great effort.
2. Prevention of potential performance degradation. Due to the domain shift, haphazardly combining different datasets may result in subpar performance.
3. Spatio-temporal separation. During each adaptation process, we have only unlabeled data from current domain. Neither data nor weights from previous domains are stored. Therefore, each adaptation process can be performed at different time and location, which we refer to as spatio-temporal separation. With spatio-temporal separation, we can first deploy our model, and adapts it to unseen domains during pretraining upon encountering new data.

[W2] The novelty of our method lies in dual-speed optimization strategy. Specifically, the student model is updated every batch, while the teacher model is updated every epoch. To the best of our knowledge, we are the first to propose such strategy. The rationale for choosing Mixup as augmentation and integrating information maximization is discussed in section 3.1 , appendix A.1 and A.2.

[W3] Thanks for your advice. We will try to improve the presentation of results. Also thanks for pointing out typos.

[Q1] One scenario is autonomous driving, where the system may continually encounter different conditions and situations, which can be viewed as different domains. In such scenario, performing well in all encountered domains is of great significance.

[Q2] Multiple domains can come from diverse natural contiditions, such as different weather.

[Q3] To the best of our knowledge, foundation vision models are not as prevalent as LLM in NLP. There is not a clear way of describing all CV tasks in a natural way as natural language does in NLP. Furthermore, as visual data is inherently far smaller than language data in amount, foundation models probably not perform well in all downsteam, highly specific tasks. The need of adaptation may long stand even if with the existence of foundation models, as the environment is constantly evolving and new domains are ceaselessly appearing.

Thanks for your time and valuable reviews, here are our responses:

[W1] The novelty of our method lies in dual-speed optimization strategy. Specifically, the student model is updated every batch, while the teacher model is updated every epoch. To the best of our knowledge, we are the first to propose such strategy.

[W2] MI is surely effective. We also provide analysis both empirically(in ablation study) and theoretically(in appendix A.2)

[W3] CoSDA aims at achieving continual SFDA. With multiple target domains, simply using BN statistics from the target domain may results in severe catastrophic forgetting. Furthermore, the moving average mean and variance are not the mean and variance of the whole dataset. They can be viewed as weighted mean, instead. As the adaptation process goes on, the mean and variance should be more and more precise, so later batches should have higher weight.

[Q] Please refer to W1.

Thanks for your time and valuable reviews, here are our responses:

[W1] CoSDA is clearly different from CoTTA in three aspects:

1. The setting of continual test-time adaptation is not about catastrophic forgetting. It computes the average of results over multiple itrations. Whereas in our continual SFDA setting data from each domain is seen only once.
2. The dual-speed optimization strategy is the module designed for continual SFDA. While in CoTTA, both teacher model and student model are updated after each epoch.
3. CoTTA needs to inference $k$ times(typically $k=32$), while CoSDA only needs $k=1$. We discuss knowledge distillation-based methods in related works section, which is more relative than EMA.

[W2] We do discuss Continual DA methods in the last part of related works section.

[W3] The max number of domain is not 4, but 6. DomainNet has 6 domains, and the results are shown in Figure 2. The domains in the studied datasets have distinct covariate shift, while in continual test-time adaptation setting, all domains are augmented with 15 types of corruptions from the same images, therefore sharing great similarity. Furthermore, we implement CoSDA in continual test-time adaptation setting, and find CoSDA is superior even in this setting. The table below shows the classification error on CIFAR-10-to-CIFAR-10C:

[Q1] The rationale for choosing Mixup as augmentation is discussed in Section 3.1 and Appendix A.1. We have tried CutMix and the result is not as well.

[Q2] By default, all current SFDA works discussed in the paper train on the whole unlabeled dataset, so we follow the setting.

[Q3] The stochastic restore part is not removed, which can varified from our source code(in supplementary materials). We implement CoSDA in continual test-time adaptation setting, and find CoSDA is superior even in this setting. The table below shows the classification error on CIFAR-10-to-CIFAR-10C:

Thanks for your time and valuable reviews, here are our responses:

[W1] The technique contribution is somewhat limited. The proposed method can be easily derived from existing methods. For example, the idea of the teacher-student framework[1] and the mix-up strategy[2] for continual source-free domain adaptation is nothing new.

[W2] We implement CoSDA in continual test-time adaptation setting, and find CoSDA is superior even in this setting. The table below shows the classification error on CIFAR-10-to-CIFAR-10C:

[W3] The dual-speed optimization strategy is the module designed for continual SFDA. The student model captures short-term features and update to adapt to target domains, while the teacher model filters out long-term domain-invariant features. The EMA momentum does not have much impact, so we fix the parameter over all experiments. Specifically, we follow the settings in MoCo and BYOL and increase the momemtum from 0.9 to 0.99 using a cosine schedule, as stated at the end of Section 3. We will carry out experiments to demonstrate that afterwards. Thanks for your advice!

Thanks for your time and valuable reviews, here are our responses:

[W1] Both $\tilde{p}$ and $h_{\psi}(\tilde{x})$ are in the range [0, 1]. The hard-label from teacher is computed as $p := softmax(h_{\theta}(X)/\tau)$, while $h_{\psi}(\tilde{x})$ denotes logits normalized by softmax function.

[W2] As stated above, there is no distinct divergence in mathematical spaces. Also, ablation study is already provided as results (-) MI. You can find collapse in Table1 between columns "CoSDA" and "CoSDA (-) MI".

[W3] Yes, the moving average mean and variance are not the mean and variance of the whole dataset. They can be viewed as weighted mean, instead. As the adaptation process goes on, the mean and variance should be more and more precise, so later batches should have higher weight.

[W4] CoSDA outperforms all outher method either on adaptation accuracy(shown in green) or accuracy drop(shown in red), achieving best trade-off. For example, as reviewer nYkW metioned, comparing with EdgeMix in table1, in terms of the accuracy drop, our method is clearly at least 3x smaller than EdgeMix. Combined with other modules, our method lowers about 2x accuracy drop on source domain with neglectable decrease in adaptation accuracy.

[Minors] The core idea of our paper is that CoSDA achieves "continual SFDA", while in federated setting, the predictions from multiple clients are gathered and averaged without the concept of "continual". Thanks for pointing out our typos.

[Q1] During each adaptation process, we have only unlabeled data from current domain. Neither data nor weights from previous domains are stored. Therefore, each adaptation process can be performed at different time and location, which we refer to as spatio-temporal separation.

[Q2] $h_{\psi}(\tilde{x})$ refers to softmax probabilitie.

[Q3] CoSDA outperforms all outher method either on adaptation accuracy(shown in green) or accuracy drop(shown in red), achieving best trade-off. Ideally, adaptation accuracy should be as high as possible, while accuracy drop as small as possible.

Thank you for the response.

However, most of my concerns are not addressed properly. I will keep my score.

[W1 and. W2] then why you used $h$ for different mathmatical spaces? Clearly $h_{\theta}$ and $h_{phi}$ point to different mathematical spaces based on the author's response, so why no clarification is made in the manuscript? It is a standard for academic writing to use consistent notations for the variables from the same mathematical space.

[W3] The authors acknowledged my concern but did not make any revisions to their manuscript or give any plan to address my concern.

[W4] My concern is the clarity of the table presentation. Readers find it hard to gather the key information (mentioned by the authors) from the current manuscript. Again, the authors did not propose any revision to address this.

[Minor] Then, the literature review should be given to federated learning.

[Q1] The authors addressed this question very well.

[Q2] Again, the same notation $h$ should not be used to mention variables from different mathematical spaces.

[Q3] Again, my concern is the clarity of the tables, not the performance lift. The lack of clarity in the experiment section makes readers feel hard to find the technical contributions of the proposed work.

Thanks for your time and valuable reviews, here are our responses:

[W1] The novelty of our method lies in dual-speed optimization strategy. Specifically, the student model is updated every batch, while the teacher model is updated every epoch. To the best of our knowledge, we are the first to propose such strategy.

[W2] The dual-speed optimization strategy is the module designed for continual SFDA. The student model captures short-term features and update to adapt to target domains, while the teacher model filters out long-term domain-invariant features.

[W3] CoSDA outperforms all outher method either on adaptation accuracy(shown in green) or accuracy drop(shown in red), achieving best trade-off. In your example, in terms of the accuracy drop, our method is clearly at least 3x smaller than EdgeMix. Combined with other modules, our method lowers about 2x accuracy drop on source domain with neglectable decrease in adaptation accuracy.