Analytic Continual Test-Time Adaptation for Multi-Modality Corruption

W5: The motivation in the paper may benefit from further clarification.

We appreciate the reviewer’s acknowledgment of the challenges in MM-CTTA. MM-CTTA was first introduced in [2], focusing on enabling multimodal networks to adapt to dynamic target domains in an online manner within the context of 3D segmentation for autonomous driving. Building upon this foundation, our work emphasizes addressing the challenges of MM-CTTA in real-world applications.

For instance, we adopt a more challenging setting that combines the difficulties of MM-TTA and CTTA, where the model must handle continuous interleaved modality corruptions. Furthermore, we introduce even more extreme tasks, such as progressive single-modality corruption, where test data is processed in an online manner (i.e., each sample is seen only once). This setting closely aligns with practical applications requiring rapid adaptation, and our method demonstrates significantly faster adaptation compared to existing approaches.

[2] Cao H, Xu Y, Yang J, et al. Multi-modal continual test-time adaptation for 3d semantic segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 18809-18819.

W6: Does this imply that, if pseudo-labels were not used, error accumulation would not occur? If so, does this suggest that using pseudo-labels may be unnecessary?

Using pseudo-labels for model updates has been widely recognized as an effective approach for generating supervised signals in previous TTA works [3–7]. However, since TTA methods aim to adapt to test data without access to ground truth for supervision, error accumulation is an unavoidable challenge during the continual adaptation process.

In our work, we introduce DSM and SPS modules to minimize errors introduced by pseudo-labels. The effectiveness of these modules is demonstrated through comprehensive ablation studies.

Table 2: Ablation studies on different component combinations.

Q1: Another potential area for improvement in MDAA could be its robustness to label noise. the effectiveness of this strategy in highly noisy environments remains to be fully evaluated. Future research could focus on developing more advanced mechanisms for detecting and mitigating label noise across different modalities.

In MM-CTTA, noise primarily arises from corrupted data, which can lead to incorrect labels during model updates. To address this, we implement modality compensation through DSM, effectively mitigating the risks associated with noise.

For our experiments, we followed the corruption generation process outlined in [3], applying the highest corruption level (Level 5) for each test sample. This level is particularly challenging and has been used as a benchmark in other TTA works [3,5,7].

In future work, we plan to explore more robust filtering mechanisms to further enhance the practicality and resilience of our method in real-world applications.

[3] Yang M, Li Y, Zhang C, et al. Test-time Adaptation against Multi-modal Reliability Bias[C]//The Twelfth International Conference on Learning Representations. 2024.

[4] Wang D, Shelhamer E, Liu S, et al. Tent: Fully Test-Time Adaptation by Entropy Minimization[C]//International Conference on Learning Representations.

[5] Lei, Jixiang, and Franz Pernkopf. "Two-level test-time adaptation in multimodal learning." ICML 2024 Workshop on Foundation Models in the Wild. 2024.

[6] Niu S, Wu J, Zhang Y, et al. Towards Stable Test-time Adaptation in Dynamic Wild World[C]//The Eleventh International Conference on Learning Representations.

[7] Wang Q, Fink O, Van Gool L, et al. Continual test-time domain adaptation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 7201-7211.

In light of these clarifications, would you consider increasing your score for our paper? Otherwise, could you let us know any additional changes you would like to see in order for this work to be accepted?

We appreciate your advice and questions on the practical application of the model's capabilities., here are the responses:

W1: how can we ensure the model does not overly rely on a single modality when dealing with highly imbalanced modal data? Is there empirical evidence of MDAA’s effectiveness in extreme cases?

We are not entirely certain about the specific meaning of "overly rely on a single modality" and have considered two possible interpretations:

• Reliability Bias (as discussed in the article and illustrated in Figure 1(C)):

We have addressed this issue through dedicated experiments. In Tables 4 and 5, we evaluate various methods under interleaved modality corruption tasks, where the corrupted modality switches repeatedly over time, and the model does not know in advance which modality is corrupted.

While some comparison methods perform well during video-corruption phases, their performance degrades significantly when the corrupted modality changes. In contrast, The performance of our model is consistent with that at unimodal corruption.

• Modality Absence:

If "overly rely on a single modality" refers to situations where only one modality is available for prediction, our model is still effective. This is because our framework includes three independent classifiers. As long as data from at least one modality is available, its corresponding classifier will produce predictions and adapt itself accordingly.

W2: can the model effectively respond to previously unseen modality corruption in practical applications? Are there relevant test results? Additionally, how effective is MDAA in rapidly changing scenarios where modality shifts occur frequently?

We would like to clarify that the model is pre-trained on clean (i.e. non-corrupted) dataset, therefore all corruption types in the experiment results (e.g. Gauss., Traff, et al.) are all unseen to the model. Table 1-5 are all test results that can illustrate model’s effectiveness of continually adapting to unseen modality corruption.

The recursive solution of MDAA ensures a globally optimal solution that incorporates both the source domain and all previously encountered test domains. Unlike gradient-based methods, MDAA does not require a large number of samples or epochs to adapt to new domain shifts. Therefore, the performance of MDAA remains consistent across different modality shifting frequencies.

W3: The snow image in Figure 5 appears quite similar to the rain image. Could the authors check if there is a potential issue here?

We follow the method in [1] to implement corruption, with the corruption of this image specifically defined as snow in [1].

[1] Yang M, Li Y, Zhang C, et al. Test-time Adaptation against Multi-modal Reliability Bias[C]//The Twelfth International Conference on Learning Representations. 2024.

W4: Does the MDAA method incur significant computational overhead? For practical applications, what measures can help balance model complexity with real-time adaptability?

We have conducted additional experiments on runtime and VRAM usage. Since MDAA does not rely on backpropagation, it requires less computational time and consumes lower memory compared to most existing methods. While MDAA adapts to test data based on the analytic solution, the computational complexity mainly lies on the matrix inversion, which has a time complexity of $O(N^3)$. Here, N is the feature dimension, which is set as 8000 throughout our experiments. Note that the feature dimension does not increase during CTTA period, thus the complexity of each update keeps consistent.

Table 1: Comparison of methods in terms of runtime and memory usage.

W1: comparison of the source model performance when trained with cross-entropy loss versus least squares loss

Our learning goal is to map features to pseudo-labels using ridge regression rather than optimizing through back-propagation. Therefore, we are constrained to using a linear loss function, specifically the least squares loss, rather than cross-entropy loss.

W2: The formulation restricts adaptation only to the classification layer. It has been shown in [1] how adapting the encoders themselves can improve performance. While [1] requires knowledge of which modality is corrupted, it would be informative to know what the drop in performance is due to this compromise

The work [1] mentioned by the reviewer is an approach that adapts the encoders to achieve TTA. However, such an approach inevitably shift the feature space of entire model, leading to catastrophic forgetting. Consequently, [1] is not suitable for CTTA tasks as addressed in our work. Furthermore, [1] requires prior knowledge about which modality is corrupted, making it inapplicable to scenarios involving interleaved modality corruption.

In contrast, our method updates classifiers by solving the ridge regression problem using the least square error. This approach enables the classifiers to retain prior knowledge from both the source and test domains. To preserve the conditions of the ridge regression problem, parameter updates are restricted to the output layer (i.e., the classifier), while all preceding layers remain frozen.

In the future, we aim to explore improved strategies to enhance the encoder's feature extraction capabilities during the CTTA process.

[1] Lei, Jixiang, and Franz Pernkopf. "Two-level test-time adaptation in multimodal learning." ICML 2024 Workshop on Foundation Models in the Wild. 2024.

W3: The choice of leader network in DSM seems arbitrary in the absence of experiments which validate the choice. For example, how does choosing the lowest entropy network work? Furthermore, how what is the frequency of updates using DSM? These questions need to be clarified.

The selection process in DSM is not arbitrary. The prediction with the highest probability is chosen through DSM as the pseudo-label (i.e., the leader prediction) for subsequent updates, rather than using the lowest entropy. This approach is based on the observation that higher prediction probabilities generally reflect greater model confidence and, therefore, higher reliability [1–2]. This operation is repeated for each sample within a batch, hence the update frequency using DSM is once per batch. We will highlight corresponding context in the revised manuscript.

[1] Boudiaf M, Mueller R, Ben Ayed I, et al. Parameter-free online test-time adaptation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 8344-8353.

[2] Wang Z, Luo Y, Zheng L, et al. In search of lost online test-time adaptation: A survey[J]. International Journal of Computer Vision, 2024: 1-34.

Thanks a lot for the rebuttals for my reviews, However, only some of my concerns have been addressed, while there are still some questions that are not answered satisfactorily, thus, I am going to low my score to 5.

We sincerely appreciate all your attention and comments, here are our replies:

W1: The analysis of the differences between the proposed method and existing methods, as well as the limitations of the analysis method is missing.

Thank you for pointing this out. We have included the differences and the limitations as follows:

While most of existing methods are designed either to solve CTTA or MM-TTA, our method is capable of solving both challenges in CTTA and MM-TTA (i.e. error accumulation, catastrophic forgetting and reliability bias). Our method performs well when evaluated under the highly challenging setting of the superposition of both tasks (i.e., MM-CTTA), where model experiences continuous interleaved modality corruptions.

Specifically, our method is not a simple combination of existing MM-TTA and CTTA methods. To the best of our knowledge, our work is the first to introduce analytic learning to TTA to solve the continual domain shift issue. While most TTA methods update models through backpropagation, our method updates model parameters through a recursive closed-form solution, which is equivalent to the solution of global optimization on all encountered datasets, thereby solving the forgetting problem of continual domain shift.

W2: lacked experiments on ACs and SPS, as well as DSM and SPS.

Thank you for your comments on our ablation studies. We have conducted ablation experiments on ACs and SPS to further demonstrate the effectiveness of DSM. As shown in Table 1, incorporating DSM improved accuracy by over 2% compared to configurations without DSM across all experiments.

ACs serve as fundamental solutions for optimizing on both the source and all encountered test domains. Removing ACs would either result in a lack of update rules or violate the CTTA setting, where access to previous data is not permitted. We have theoretically proven that ACs’ solution is equivalent to the solution of joint learning of all encountered data with access to only data in current test stage. Therefore, we are unable to conduct the DSM and SPS experiments.

Table 1: Ablation studies on different component combinations.

W3: Need supplement the running time of existing methods and all proposed methods in different models.

Thank you for your suggestion. We have conducted additional experiments on runtime and VRAM usage. Since MDAA does not rely on backpropagation, it requires less computational time and consumes lower memory compared to most existing methods. While MDAA adapts to test data based on the analytic solution, the computational complexity mainly lies on the matrix inversion, which has a time complexity of $O(N^3)$. Here, N is the feature dimension, which is set as 8000 throughout our experiments. Note that the feature dimension does not increase during CTTA period, thus the complexity of each update keeps consistent.

Table 2: Comparison of methods in terms of runtime and memory usage.

Thanks for your review and valuable comments. Here we address your concerns individually as follows:

W1: The proposed test-time adaptation method is by simply optimizing the ridge regression problem on both source and target data, as per Eq. (12). The innovation is similar to adding a cross-entropy loss with pseudo labels on target data in regular classification tasks. It is unclear what are the main technical challenges in deploying test-time adaptation methods to multi-modality data

We would like to clarify that our method is not simply optimizing on source and target data through standard ridge regression problem. Under the continual test-time adaptation (CTTA) scenario, the model is required to continually adapt to incoming test data stream, which means that the model is forbidden to acquire any previous test data and source data in current test stage. Therefore, it is unachievable for the model to directly optimize on all encountered source and test data through standard ridge regression.

Apart from that, our method extends standard ridge regression to a recursive solution in Theorem 2. This extension addresses the aforementioned limitation, enabling optimization not only on newly arriving data but also retaining all prior knowledge from both the source and test domain without access to any previous data.

Theorem 2: $\hat{**W**} _ {\textup{T},t} = ({\tilde{**X**}} _ {\textup{exf},\textup{S}}^\top\tilde{**X**} _ {\textup{exf}, \textup{S}}**X** _ {\textup{exf},1:t}^\top **X** _ {\textup{exf},1:t} \gamma{**I**})^{-1} ({\tilde{**X**}}_{\textup{exf}, \textup{S}}^\top{\tilde{**Y**}} _ {\textup{S}}{**X**} _ {\textup{exf},1:t}^\top{\bar{**Y**}} _ {\textup{T},1:t}) =**P** _ {\textup{T},t}^{-1}**Q** _ {\textup{T},t}$

The primary challenge of TTA in multimodal data lies in reliability bias, which arises from the over-reliance of pre-trained models on specific modalities. For example, as shown in Figure 1(C), when the video modality is corrupted, the performance of both the multimodal and video-only models exhibits a similar downward trend. This occurs because the model disproportionately depends on video information while neglecting audio cues. To address this issue, we propose the Dynamic Selection Mechanism (DSM), which allows the model to dynamically identify and prioritize trustworthy modalities. DSM plays a vital role in selecting reliable samples for each modality-specific classifier, thereby ensuring a robust and effective model update process.

W2&Q1: How can the method address the challenge of continual domain shift

We answer these questions together as they are similar. We address the domain shift problem through our recursive solution in Theorem 1 and DSM module. While existing methods adapt models to current test domain (e.g., data corrupted by Gaussian noise) solely by training the model on these data, this results in overfitting to that specific domain. By contrast, our method absorbs useful knowledge from the current domain through DSM, while simultaneously preserves source and encountered domain knowledge during continual adaption via the recursive solution of ACs. Experiment results in Table 1-3 show that performance of our model does not drop during adapting to continual shifting domains compared to other methods. Here we give part of results from Table 1.

Table 1: Comparison with SOTA methods on audio progressive single-modality corruption task in terms of classification Top-1 accuracy (%), using dataset VGGSound-C in severity level 5. The best results for each domain are highlighted in bold.

Dear Reviewer,

As the deadline is approaching, we kindly request your feedback or response to our replies. If there are any additional questions or clarifications needed, please let us know, and we are willing to to clarify. Thank you for your time and consideration.

Regards,

The Authors