Response to Reviewer 5Yv6

Thanks for the constructive suggestions! We have addressed each point with careful consideration and revised our work accordingly. Below is our detailed response:

W1 [Claims of setting]

We agree that [1] has explored multi-modal shifts more broadly. However, our work explicitly identifies uni-modal shifts as a distinct and critical subclass of multi-modal shifts, which necessitates tailored solutions. Our conceptual contributions lie in:

• [1] studies shifts in any subset of modalities (e.g., partial or all), formulating to the form of $p_s(x)≠p_t(x)$ as shown in its "Sec. Problem Formulation". Whereas our work specifically models shifts occur in one modality $p_t(x^{(k)}) \neq p_s(x^{(k)})$ and $p_t(x^{(i)}) = p_s(x^{(i)}), \forall i \neq k$. We also discussed why [1] is inefficient in our setting in the related work.
• We further highlight ambiguities of terminology in existing literature, e.g., ref [1] and [2] both focus on "multi-modal shift". However, [2], different from [1], studies "shifts happens in all modalities". To resolve this inconsistency, we introduce a precise definition of uni-modal shifts to differentiate them from broader multi-modal shifts.
• Most importantly, we put our emphasis on "uni" is we believe its broad practical implication, as we stated in the introduction, many real-world corrupting factors impact only specific modalities.

We have reframed our conceptual contribution as identifying the unique challenges of uni-modal shift via theoretical and empirical analysis, while emphasizing its practical significance.

W2 [Novelty of methods]

Building on W1, our analysis reveals that uni-modal shifts undermine cross-modal fusion and induce negative transfer. While adapters and routing mechanisms are established concepts, our innovation lies in their specific integration to tackle the unique challenges of uni-modal distribution shift:

Residual Adapter Design

Unlike standard adapters, our architecture explicitly decouples shift-agnostic base features from shift-specific components through residual connections as in Eq. 8, enabling targeted adaptation without catastrophic forgetting.

Shift-Aware Router

The shift-aware router introduces a gating mechanism that dynamically selects adapters based on modality-specific shift detection, a crucial capability missing in prior routing approaches.

W3 [Experiments for autonomous driving tasks]

With due respect, we offer the following clarifications: While we use autonomous driving as an illustrative example, our experiments span diverse tasks (action recognition on Kinetics50, event recognition on VGGSound) with applications extending beyond autonomous driving.

We acknowledge the need for diverse multi-modal shift benchmarks (though most current multi-modal TTA research focuses on the two datasets, Kinetics50 and VGGSound, with various multi-modal shifts). However, reproducibility challenges with MM-TTA (unreleased code) prompted us to validate our approach on other datasets. We choose CMU-MOSI→CMU-MOSEI, real-world sentiment analysis datasets with possible modality-specific shifts: Both datasets include audio, text, and video data. The CMU-MOSEI paper highlights factors such as varying face detection methods that can contribute to visual distribution shifts. For instance, some clips may be more face-centered, indicating a video shift while other modalities remain largely unchanged. To support this claim, we calculate the Maximum Mean Discrepancy (MMD) between each modality to measure their discrepancies:

The significant vision shift (14× larger than audio/text) mirrors real-world scenarios where single modalities shift. Our experiments on this dataset aim to: 1) provide evidence that many real-world shifts resemble uni-modal shifts and 2) further validate the effectiveness of our method. We then simply apply our "selective adaptation" architecture on a recent work CASP (due to the difference between tasks, we choose to swiftly adopt our core idea to the existing codebase to present the performance in a short rebuttal period). The following are the results:

These results help demonstrate our method's effectiveness on more diverse tasks.

S1 [term router]

We apologize for the oversight. We have strengthened its description in Sec. 3.5 Selective Adaptation, where we detail its role in computing gating weights based on shift severity.

Q1

Please check our reply to Reviewer UDgQ W3 [Sensitivity of hyperparameter].

We thank the reviewer for the constructive feedbacks, which has strengthened our paper. We remain open to further revisions.

Response to Reviewer UDgQ

Thanks for the reviewer’s positive comments and constructive suggestions! We have carefully considered each point and revised our work accordingly. Below is our detailed response:

W1 [Novelty of method]

We emphasize that our design offers a straightforward solution to the novel challenge of uni-modal shift. While adapters and routing mechanisms are established concepts, our innovation lies in their specific integration to tackle uni-modal distribution shifts—a critical challenge we analyze theoretically and validate empirically. Our methodological contributions include:

Residual Adapter Design

Unlike standard adapters, our architecture explicitly decouples shift-agnostic base features from shift-specific components through residual connections as in Eq. 8, enabling targeted adaptation without catastrophic forgetting.

Shift-Aware Router

The shift-aware router introduces a gating mechanism that dynamically selects adapters based on modality-specific shift detection, a crucial capability missing in prior routing approaches.

This integration uniquely addresses scenarios where only one modality shifts, a common real-world issue under-explored in prior work.

W2 & Q3 [Experiments on multi-modal datasets with different modality combinations]:

First, we would like to bring to the reviewer’s attention that most current multi-modal TTA research focuses on Kinetics50 and VGGSound and with various multi-modal shifts. Nevertheless, we acknowledge the importance of more diverse multi-modal distribution shift benchmarks. Therefore, we further investigate a real-world distribution shift within multi-modal datasets by examining the transfer from CMU-MOSI to CMU-MOSEI, which includes audio, text, and video data for sentiment analysis. The CMU-MOSEI paper highlights factors such as varying face detection methods that can contribute to visual distribution shifts. Our experiments on this dataset aim to: 1). provide evidence that many real-world shifts resemble uni-modal shifts and 2). further validate the effectiveness of our method. For instance, some video clips may be more face-centered, indicating a video shift while other modalities remain largely unchanged. To support this claim, we calculate the Maximum Mean Discrepancy distance between each modality to measure their discrepancies:

The significant vision shift (14× larger than audio/text) mirrors real-world scenarios where single modalities shift. We then simply apply our "selective-adaptation" architecture on a recent work CASP (due to the difference between tasks, we choose to swiftly adopt our core idea to the existing codebase to present the performance in a short rebuttal period). The following is our results:

These results help demonstrate our method's effectiveness on broader ranges of modality combinations or more diverse multi-modal datasets.

W3 [Sensitivity of hyperparameter]

We thank the reviewer for this observation. We avoided heavily tuning the loss coefficient as it simply follows the self-training loss. Nevertheless, we do agree that a sensitivity analysis of α would provide technical insights. We will include this analysis in the revision to strengthen the empirical validation of our method. The results are as follows:

In conclusion, we observe robust performance for α∈[0,1]:

Q1 [Model collapse]

• Probabilistic sampling (Eq. 6) in Gumbel-Softmax introduces randomness to prevent deterministic routing.
• In each of our experiments, only one modality is shifted at a time, so the 'model collapse', i.e., consistently selecting one modality is not necessarily detrimental.

Q2 [Shifts of multiple modalities]

Good question! Although our setting aligns with many practical applications, real-world environments often involve more complex combinations of distribution shifts across modalities. This poses significant challenges and requires further research. For now:

• The selection of adapters is conducted in a soft way (Eq. 8), i.e., they may function simultaneously and allow partial contributions from multiple modalities.
• We have added a limitation section at the last of our manuscript with the discussion regarding our setting’s limitation of encountering more complex shifts within multiple modalities or continual shift environments. Please check our reply to Reviewer dkfb.

Suggestions [misuse of symbol]

We have corrected the issue. We thank your meticulous review!

We thanks the Reviewer's suggestions that help strengthen our paper’s demonstration of its applicability.

Response to All Reviewers, AC and SAC

Thanks for all Reviewers' valuable suggestions and the efforts of AC and SAC.

Reviewer dkfb, UDgQ and 5Yv6 all acknowledge the studied problem (we quote: "novel and practically relevant", "interesting real-world problem" and "challenging and practical scenario"). Reviewer hbmC, dkfb, and UDgQ all give positive comments on our methodology (we quote: "excellent idea", "effectively mitigates negative transfer" and "promising solution"). Furthermore, our theoretical and empirical results are appreciated by all Reviewers (we quote: "sufficient theoretical analysis", "mathematically sound", "trustworthy", “strong empirical validation” and “extensive experiments”). Lastly, all Reviewers find our paper well presented. We'll integrate the Reviewers' suggestions into our revision. Next, we respond to all of the questions.

Response to Reviewer hbmC

W1 [Clarity of Figure 3]

We appreciate this constructive feedback. To enhance clarity, we have expanded the figure caption to explicitly describe the router’s role and the architecture’s flow. The new caption reads:

Fig. 3: The architecture of our model. The original multi-modal data is processed through modality-specific encoders to extract feature representations. A router then dynamically decides whether to adapt each modality. For instance, in the case of video corruption, the router prioritizes video adaptation (red path) while preserving audio features (blue path). Finally, the extracted features are fused and passed to the prediction head.

W2 & Q1 [Study of router's adaptation weights]

We thank the reviewer for raising this insightful question. Indeed, selective adaptation is a critical aspect of our framework, and we have conducted additional analyses to address this point. To investigate how the model adapts to synthetic data shifts in specific modalities, we design experiments where we explicitly control shifts in either the video or audio data. Our key findings are as follows:

Modality-Specific Adaptation Routes:

When synthetic shifts are introduced to the audio data during inference (e.g., Gaussian or traffic noise on audio), the model prioritizes audio adaptation in 62–69% of cases, relying less on video adaptation (31–38%). Conversely, under video shifts (e.g., Gaussian or shot noise on video), the model adapts to video in 53–56% of cases, minimizing reliance on audio adaptation (44–46%). The results are:

Partial Adaptation Trends:

Notably, not all samples with a shifted modality follow the expected route. For example, even with video shifts, 44-47% of predictions still rely on the adapted audio data. We suspect this could be attributed to two reasons: 1) Modality imbalance, where certain modalities exert a greater influence on predictions in multi-modal tasks, makes the model tend to learn from the dominant modalities. 2) Convergence dynamics of the adapter and router. They may not receive sufficient adaptation on one iteration over the test set. It's important to note that making predictions with a partially adapted model is a characteristic of test-time adaptation. This observation highlights a potential area for further research.

In conclusion, these results highlight the model’s ability to dynamically prioritize modalities based on their reliability under distribution shifts. We will include these findings and detailed visualizations in the revised manuscript, along with a discussion of potential explanations.

Suggestion [Term "router"]

We acknowledge that the "router" could cause confusion and the adaptation selector could be more straightforward. We’ll make the change accordingly in the revision.

Thank the reviewer again for prompting the analysis. We believe it strengthens the paper’s contribution by demonstrating the adaptability and interpretability of our framework.

Response to Reviewer dkfb

Thanks for the reviewer’s positive comments and constructive suggestions! We have carefully considered each point and revised our work accordingly. Below is our detailed response:

W1 [Theoretical justification router’s effectiveness]:

We agree that the theoretical underpinning of our router mechanism could be further developed. Our current understanding is in line with modality-specific attention mechanisms: The router (followed by cross-attention fusion) essentially learns to enhance attention computation conditioned on shift detection statistics (Eq. 8). Besides, we notice that Li et al. [1] give a theoretical study on this direction very recently, showing how the MoE-like structure help in multi-tasks or multi-domain scenario. Drawing on the ideas presented in the article, we anticipate that theories related to 'shift-specific MoE' could serve as a starting point.

While time constraints prevent full theoretical development in this submission, we would welcome the reviewer’s suggestions on specific theoretical directions to prioritize in our revision or future work.

[1] Hongbo Li, et al. "Theory on Mixture-of-Experts in Continual Learning." The Thirteenth ICLR. 2025

Suggestion regarding literature review & W2 [Long-term adaptation]:

Thanks for bringing the related works to our attention. In terms of the Adabn, the reviewer might overlook our discussion in the related work where we discussed Adabn:

The Adabn (Li et al., 2017) recomputes the batch statistics for every test batch ....

As for the long-term adaptation stability, while our primary contribution remains in multi-modal shift rather than continual adaptation, we fully agree this is critical for real-world deployment. We now explicitly position our work as complementary to lifelong adaptation research: We have added a limitation section at the last of our manuscript as follows with the discussion regarding the lifelong/continual shift over time mentioned in the ref [A] along with our setting’s limitation of encounter more complex shifts within multiple modalities:

Section 6. Limitations
While our work provides a novel view for the unique uni-modal distribution shifts in multi-modal test-time adaptation, it is crucial to acknowledge several limitations. 

First, our current formulation primarily focuses on scenarios where a distribution shift occurs in only one modality at test time. Although this setting aligns with many practical applications, real-world environments often involve more complex combinations of distribution shifts across modalities. For instance, cases where two modalities experience concurrent shifts while the third remains unchanged, or shifts with varying magnitudes, pose significant challenges. Our theoretical and empirical analyses reveal that existing methods, including our proposed approach, struggle to disentangle conflicting signals from multiple shifted modalities, leading to suboptimal fusion and adaptation performance. This limitation underscores the need for more sophisticated mechanisms to diagnose and disentangle multi-modal shifts dynamically.

Besides, our method operates under the assumption of single mini-batch test-time adaptation, where adaptation occurs incrementally on small, temporally coherent batches. While this setup is practical for scenarios with transient shifts, it does not account for long-term or continual distribution shifts that evolve over extended periods [1-2]. For example, in real-world deployments, modalities may drift gradually or exhibit recurring shifts, requiring adaptation strategies that balance plasticity (to learn new patterns) with stability (to retain prior knowledge). Our method does not explicitly address catastrophic forgetting or the accumulation of adaptation errors over time, which remain critical open challenges.

These limitations, however, highlight promising directions for future research. The failure of existing methods in multi-shift scenarios motivates the development of adaptive routing mechanisms that can scale to higher-order modality combinations and dynamically prioritize shifts based on their severity. Similarly, extending our selective adaptation idea to continual or long-term settings could involve memory-augmented architectures or meta-adaptation strategies to stabilize performance over time. We hope our work inspires the community to explore these frontiers, ultimately advancing the robustness of multi-modal systems in increasingly complex real-world environments.

[1] Wang, Qin, et al. "Continual test-time domain adaptation." Proceedings of the IEEE/CVF Conference on CVPR. 2022.
[2] Brahma, Dhanajit, and Piyush Rai. "A probabilistic framework for lifelong test-time adaptation." Proceedings of the IEEE/CVF Conference on CVPR. 2023.

We appreciate the reviewer's suggestions, which have strengthened our paper's literature review and clarified its limitations.