FEATHER: Lifelong Test-Time Adaptation with Lightweight Adapters

Thanks for your detailed review and positive remarks about our work tackling an important and practical problem, approach to be easy and simple to implement, and experimental evaluation on various benchmarks to be robust.

We believe that there is some potential misunderstanding regarding some key aspects. We request the reviewer to take a look at our common response (and the revised version of the paper which incorporates the various suggestions). In addition, we also respond below to the reviewer’s individual comments. We hope our response will help address the questions/concerns and that the reviewer will reconsider the original assessment and consider raising the score.

Performance comparison of FEATHER vs. CoTTA: Table 3 clearly demonstrates that, for ImageNet-to-ImageNetC, FEATHER, with only 10.92% trainable adapter parameters, outperforms CoTTA (which adapts all (100%) of the network parameters) in terms of error rate. Further, Table 4 shows that FEATHER’s performance is comparable to CoTTA and other approaches, even with only 6.71% trainable adapters, while outperforming all these methods with as few as 50.32% trainable parameters (Table 5). FEATHER can be used with any desired number of trainable parameters depending on the computational budget. In Table 7 in Appendix of the old version (Now Table 4 in the main paper), FEATHER with CoTTA-based TTA objective outperforms EcoTTA for both CIFAR10-to-CIFAR10C and ImageNet-to-ImageNetC datasets. For ImageNet-to-ImageNetC datasets, FEATHER outperforms all the other approaches, including EcoTTA and CoTTA.

Efficiency comparison of FEATHER vs CoTTA: We believe the reviewer has missed a very important point here. The gradient calculation for conducting an update step of FEATHER and CoTTA is not similar. In FEATHER, the adapters are only present for some layers depending on the budget for the parameters. Thus, FEATHER does not introduce adapters after every layer. We have added the following line in section 3.1 to clarify this point: “In practice, we only insert the adapters in a few locations, depending on the computational and memory budget.”. Further, the backpropagation only requires the computation of gradients of the adapter parameters and not the entire network. (also see Appendix F for the architecture-specific adapter locations in detail.) Thus, FEATHER drastically improves the parameter efficiency over CoTTA. This is also reflected in the memory saving and wall-clock timing, which we have reported in the rebuttal (as requested by Reviewer CBjP) and in the updated version of the paper.

Comparison with baselines such as EATA, ECoTTA: In Appendix Table 7 in the previous version (now Table 4 in the main paper), we have reported the results for the experiments on CIFAR10-to-CIFAR10C and ImageNet-to-ImageNetC, and FEATHER performs better than EcoTTA, which uses EATA loss.

Access to source data: The initialization mechanism that we propose for FEATHER adapters circumvents the need for warm-up training of the adapters as in EcoTTA. Our approach does not inherently need source data for warm-up like EcoTTA. Our contribution is on the architectural aspect; however, if one wishes to use a learning objective like EATA, which depends on the source data for computing Fisher information, along with FEATHER, the source data would be utilized. However, this is a requirement of EATA and not that of FEATHER. We report the results of EATA + FEATHER in the updated version of the paper. (Table 6 and Table 7)

Efficiency (memory/run-time) comparisons: We have added the comparison in terms of runtime and memory usage in our common response. We provide detailed reasons for comparing the methods based on the number of trainable parameters. We believe there is some misunderstanding regarding the results reported for TENT (the entire network). We would like to highlight that the TENT (100% trainable) is an additional setting run by us for a transparent comparison. The original TENT method using only BN params performs poorly in the continual setting due to error accumulation and forgetting over time.

We thank the reviewer for the comments and suggestions. We request the reviewer to take a look at the common response and the updated version of the paper, which incorporates your and other reviewers’ suggestions. Below, we provide some clarifications regarding the results and novelty of the proposed approach. We hope that you will consider revising your assessment and raising your score.

Deciding where to add adapters: Our approach based on adding adapters in between the layers is generic. Based on the computational budget of the deployment environment, the adapters can be added to the main architecture. In the updated version of the paper, we report the locations of the added adapter layers in Appendix F, and the configs for all the architectures will be released with the code base. The location of adapter parameters is dependent on the TTA environment. For example, if the expected noise is corruption, the domain shifts are expected in the initial layers, whereas if the domain shifts occur in semantic space, the suitable adapter locations would be in the deep(er) layers (close to the output layer). In the TTA setting, all the information about the domain shifts is usually not available a priori. To be generic and comply with the TTA setting, this decision should be made based on the dataset and the architecture for which FEATHER is being used. In our experiments and lifelong TTA settings, it would be unfair to predefine/suggest/assume any details related to the domain shifts. Hence the location of the used adapters is generic. (As described in Appendix F)

Zero and Identity initialization vs residue architecture: The zero and Identity initialization preserves source knowledge since it acts as an identity operation to begin with during the start. Even though a residue architecture with zero initialization is simpler, it is not as effective as our proposed PWC and GCO-based adapters. In our earlier experiments, we found that, when adding a residual layer, the network collapses when the domain changes occur in the continual setting and adapter parameters are updated. We speculate the primary reason to be the flow of signal to pass through the added parameters and, after the update, the network starts to collapse. We also tried some other initialization schemes like near zero init; however, those do not work well with the continual TTA setup.

Difference with EcoTTA: As compared to EcoTTA, FEATHER employs more sophisticated adapters, which is supported by better performance in continual TTA, shown in Table 4 (Table 7 in the previous version), where we compare with EcoTTA as well. Please refer to general comments for more discussion and comparison with EcoTTA. Also, one may refer to section 4.2 of the EcoTTA paper for architecture-specific details, which are quite different from FEATHER.

Savings in terms of computational/memory requirements: Thanks for pointing this out and making this suggestion. We report the memory and compute savings in the general response and have updated the paper to also include these results. Our approach is more resource-efficient in terms of computational resources such as memory and FLOPs, which are usually required during the backward passes done during adaptation in test time. Further, we perform better than EcoTTA and other SOTA approaches in continual TTA, as highlighted in Table 4.

Improvements when combining FEATHER with other methods: We respectfully disagree that there is no improvement when FEATHER is combined with methods such as TENT and CoTTA. As compared to standard TENT and CoTTA, which require updating all the parameters (100%) during adaptation, when these methods are augmented with FEATHER, significantly fewer parameters need to be updated, as shown in Table 6. Note that (as we mention in our common response as well as reported by the additional experiments) a reduction in trainable parameters directly affects the memory budget requirement as well as inference/adaptation time (Table 7).

The objective function: Thank you for pointing this out. The objective function in the experiments of Tables 1,2,3,4,5 is the same as that of CoTTA. We briefly mention this in the second line of Section 5.2. In the updated version of the paper, we highlight this in the table captions as well.

Inconformity in the formulation of Lifelong/Continual TTA: Note that, in Section 2, we start by defining the standard (non-continual) TTA setting. However, in the second last paragraph, we define the continual/lifelong TTA setting. Since the primary focus of this work is the behavior of adapters in a continual setting, we follow the continual setup for the entire experiments and results section and report the observed findings.

Dear Reviewer M9Xj,

As the author-reviewer discussion period is about to get over (Nov 22, end-of-day AoE time), we request you to please take a look at our author response (and the updated version of the paper, which incorporates your suggestions and includes the experiments you had suggested) and let us know it your concerns are addressed, and if you have any follow-up questions.

Thanks.

We thank the reviewer for the comments and suggestions, and are glad that the reviewer finds our idea novel. Please refer to our common response as well as the updated version of the paper that has incorporated the various comments from all the reviewers. Below, we respond to your specific comments/concerns, and we hope that you will consider revising your opinion and consider increasing the score in light of our response.

Is FEATHER only applicable to CNNs? As we mention in the common response, although we demonstrate our idea for generic CNN-based architectures for computer vision, our idea is general and can be applied to any deep learning architecture (including transformers) that allows the insertion of adapters between layers. Some implementational aspects will need to be changed based on the underlying architecture, but the rest of the idea is more broadly applicable.

Improvement over CoTTA: Please refer to our common response.

Memory and computation savings: Please refer to our common response.

Why CoTTA “outperforms” our method? We respectfully disagree with this and would like to draw your attention to a few key points (also mentioned in our common response). Please note that, while CoTTA with all parameters trainable outperforms FEATHER marginally, CoTTA also has significantly more trainable parameters. In contrast, FEATHER only uses ~7.2% to 12.2% trainable parameters and yet performs comparably to CoTTA. Further, we demonstrate in Table 5 that with only 50.32% trainable parameters, FEATHER outperforms CoTTA, which highlights that even adapting just around half the number of parameters is enough.

Thank you for engaging in a discussion. We believe there is some potential misunderstanding about some of the key aspects, and we request you to look at our response below, which we hope will clarify the misunderstanding and we hope that you can do a fair assessment in light of these points.

1. For our experiments, for a fair comparison and to assess the improvement of performance over existing methods, we follow the architectures proposed in existing methods on lifelong TTA on the popularly used RobustBench platform, where the architectures are CNN-based. Since the numbers reported by other methods in the lifelong TTA setting use such architectures, it would be unfair to compare them against another architecture, like a transformer-based architecture.

2. We would reiterate that our aim is not to surpass the performance of the full fine-tuning approach but rather to perform at least comparably with a significantly smaller number of parameters (which also translates into memory and time savings, as our results show). Our experimental results show that FEATHER is flexible in terms of the available computational budget and even outperforms CoTTA if the computational budget is increased (roughly 50% of CoTTA budget; please refer to Table 5). Please note the primary aim here is to reduce the computational budget. Moreover, the results in Table 3 for the ImageNet-to-ImageNetC dataset show that FEATHER achieves an improvement with only ~10% of parameters being trained.

3. We respectfully disagree that a reduction in memory or computation cost does not align with the reduction in parameters (as shown in Table 7 and explained in the text around it, we get improvements on these metrics, too). We would also like to highlight that both the papers mentioned by you are about standard fine-tuning for transfer learning and not for TTA or lifelong TTA, whereas our work is about lifelong TTA. Lifelong TTA presents a very different set of challenges than standard fine-tuning, which is done during the training phase, whereas TTA or lifelong TTA must perform fine-tuning at inference time using the unlabeled test input(s).

We hope our response above clarifies the potential misunderstandings and that you reconsider your original assessment of our paper. We would be happy to answer any other concerns/comments you might have.

Dear Reviewer bPzr,

As the author-reviewer discussion period is about to get over (Nov 22, end-of-day AoE time), we request you to please take a look at our author response (and the updated version of the paper, which incorporates your suggestions and includes some additional experiments) and let us know it your concerns are addressed, and if you have any follow-up questions.

Thanks.

Thanks for going through our earlier response and the follow-up question. Our response to your question is given below, and we hope that it helps clarify the difference between adapter design and update strategies used in lifelong TTA and standard transfer learning.

The adapters used in our work are designed specifically, keeping in mind the challenges posed by TTA and lifelong TTA. Standard adapters used in transfer learning are not directly applicable in TTA (and especially lifelong TTA) because:

(1) In TTA and lifelong TTA, the adapter parameters are trained at inference time using an unsupervised loss (which can be CoTTA, TENT, EATA, etc.), whereas, in transfer learning, during fine-tuning on the target domain, the adapter parameters are trained using a supervised loss defined on the target domain's labeled examples. Because of the need for unsupervised adapter parameter updates, we leverage learning objectives that are based on entropy or cross-entropy (utilizing pseudo labels).

(2) Unlike standard transfer learning, in lifelong TTA, due to the need for unsupervised finetuning, there is a potential risk of error accumulation over longer horizons as the test domains keep drifting over time. Our framework FEATHER, because of the design strategies, such as identity initialization of the adapter, ensures that such error accumulation is mitigated effectively.

(3) In transfer learning using adapters, the adapter updates can leverage the labeled examples for multiple iterations, whereas, in our lifelong TTA setting, we have to operate in an online setting where we do not have the luxury to revisit the unlabeled test samples more than once. In this aspect, too, FEATHER’s adapter update strategy is different from the adapter update used by standard supervised fine-tuning-based transfer learning.

Thanks for your detailed and insightful comments. In addition to our common response to all the reviewers, we provide below our response to your specific concerns/questions and hope that you will reconsider your original assessment and revise your score as you deem appropriate.

Comparison with other SOTA methods that use source data: Please see our common response.

Memory usage and wall-clock time comparisons: In our common response, we have explained why we had shown results about savings in the number of parameters and how it results in reductions in memory usage and wall-clock time. We agree with your suggestion that it will be insightful to report these numbers as well, and have performed additional experiments and included these results in the updated version of the paper (Table 7). The table clearly shows that reducing the parameters (specifically updatable parameters) directly reduces the memory consumption by a significant margin, as well as the time required for the adaptation. Note that the reported wall-clock times are averaged over all 15 corruptions for all the methods.

Is the approach architecture-specific? Please see our common response.

Clarification about Table 7: We apologize for the misunderstanding. The experiments in Table 7 (of the original paper), as well as the rest of the paper, follow a lifelong/continual TTA setup (not standard TTA). We have updated the caption in Table 7 in the new version of the paper (Table 4 in the updated version of the paper).

Variant of CoTTA with only BN params updates: Thanks for your suggestion. In the updated version, we report results (Table 10) not just for CoTTA with BN params but also for EATA (BN params), EATA (all params), and EATA + FEATHER (adapter params). Our experiments highlight that, with a significantly smaller number of trainable parameters, similar performance can be achieved while keeping the source knowledge intact.

Dear Reviewer CBjP,

As the author-reviewer discussion period is about to get over (Nov 22, end-of-day AoE time), we request you to please take a look at our author response (and the updated version of the paper, which incorporates your suggestions and includes the experiments you had suggested) and let us know it your concerns are addressed, and if you have any follow-up questions.

Thanks.