Test-time Adaptation against Multi-modal Reliability Bias

Thanks for the detailed comments. In the following, we will answer your questions one by one.

Q1.1: The paper lacks some experimental details, such as clarifying the terms “Statical” and “Dynamic” in Table 1 and explaining the distinctions between the proposed Self-Adaptive Attention and traditional self-attention. It is unclear what these terms mean and how they affect the performance of different methods. The paper should provide more definitions and discussions on these terms. Moreover, the paper should elaborate on how the Self-Adaptive Attention differs from the conventional self-attention in terms of design, implementation, and advantages.*

A1.1: We apologize for any confusion regarding these terms. Allow us to elaborate:

"Statical" and "Dynamic". "Statical" denotes the state where the layer normalization (LN) and attention-based fusion (AF) layers remain frozen during the test-time phase. Conversely, "Dynamic" implies the updating of parameters within all normalization layers rooted in the source model as most existing TTA methods do in the test-time phase.

Self-Adaptive Attention v.s. traditional self-attention. The traditional self-attention mechanism usually keeps the parameters inherited from the source model and performs information integration across modalities. Apparently, the distribution shift between training and test-time data might hinder the similarity estimation between tokens. As a result, reliable fusion on biased modalities cannot be guaranteed. Instead, we hope that the model could focus more on the unbiased modalities and avoid the interoperation from the bias. To this end, we propose repurposing the cross-modal attention-based fusion layers in a self-adaptive way. As depicted in Figure 2 of the manuscript, our default approach in the RFRA framework involves updating only the parameters of the last Transformer layer (referred to as the AF layer) in a self-adaptive manner. We dub this paradigm as self-adaptive attention-based fusion, abbreviated as "SAF". SAF essentially repurposes the standard AF operation by modulating the parameters within the attention layer with the guidance of the proposed objective function.

Q2: The confidence threshold gamma work as the important parameter in the confidence-aware loss. However, the influence of the confidence threshold $\gamma$ on TTA performance is not explored, and it would be beneficial to understand its role more explicitly. The paper could conduct more experiments and analysis to show how different values of $\gamma$ affect the accuracy and robustness of TTA. The paper could provide provide some insights on how to choose an appropriate value of $\gamma$.

A2: Thanks for your comment. In response to your concern, we investigate the influence of the only hyper-parameter (i.e., threshold $\gamma$ in Eq. 6) in our approach. To this end, we vary $\gamma$ in the range of $[0.1, 0.2, 0.3, e^{-1}, 0.4, 0.5]$ and perform corresponding experiments on the Kinetics50-C benchmark with fog and traffic corruptions. Results are depicted in Fig. 10 within the revised manuscript. For your convenience, we attach the corresponding numerical results in the following table.

The results illustrate the stability of RFRA across varying threshold values of $\gamma$.

Q3: How does the attention value in Figure 4 calculated? Some explanation is needed. The paper should provide more details on how to compute the attention value for each modality pair in Figure 4.

A3: Thanks for your comment. In Figure 4, the attention values between modalities, denoted as "Attention X-Y" (X, Y $\in$ {A, V}), are calculated using Equation 4 from the manuscript:

$\mathbf{A}=\operatorname{Softmax}\left(\frac{\mathbf{Q} \mathbf{K}^T}{\sqrt{d}}\right)$.

For instance, "Attention X-X" represents the self-attention, computed by setting both the query ($\mathbf{Q}$) and key ($\mathbf{K}$) as the tokens of modality X. Meanwhile, "Attention X-Y" (X, Y $\in$ {A, V}) corresponds to the configuration where the query and key tokens come from modality X and modality Y, respectively.

Thanks for the insightful reviews. We will answer your questions one by one in the following.

Q1.1: The experiment results are mainly obtained under the setting of severity 5. To establish the method's generality, it is encouraged to expand the empirical results across a spectrum of scenarios, including different severity levels. A broader array of experiments, encompassing various severity levels, would not only fortify the method's reliability but also enhance the comprehensiveness of this study.

A1.1: Thanks for your valuable suggestions. In the submission, we have reported the comparison results on VGGSound-C and Kinetics50-C benchmarks under 15 visual corruption and 6 audio corruption with a severity level of 5.

In response to your constructive feedback, we conduct additional experiments to assess our RFRA's performance across different corruption types at severity level 3. We've summarized these results in Tables 10-12 within the revised manuscript. For your convenience, the corresponding results are provided in the following tables.

Q2.2: In the analysis of Figure 5, claims are made about the importance of maintenance between audio and video modalities, but the contrast with clean results is not evident. Additional clarification or supplementary results are needed to support these claims.

Thanks for your comment. We have supplemented more visualization results on different baselines for comparisons. Specifically, Fig. 11 in the Appendix of the manuscript shows the attention matrixes of the source model with vanilla attention-based fusion (AF), and the model adapted by Tent with dynamic LN.

Q3: There are a few typos and vague statements in the paper, such as "traffic noise in the audio modality" in the caption of Figure 1, "information bias" on Page 2, and inconsistent notations like "(Stat. LN) & AF" or "Stat. (LN & AF)" in Tables 1-3. These should be corrected for clarity and consistency.

We appreciate your feedback on the typos. In the updated version, we have carefully revised the typos.

Dear reviewer 9FmQ,

As the author/reviewer discussion will draw to a close soon, we would like to know if our response has addressed your concerns and questions. If you have any further concerns or suggestions for the paper or our rebuttal, please let us know. We would be happy to engage in further discussion and manuscript improvement.

Thank you again for the time and effort you dedicated to reviewing this work.

Thanks for your valuable reviews. We would like to address your concerns one by one in the following.

Q1.1: While the paper is well-written, it lacks essential details about CAV-MAE, such as the number of attention layers used for fusion. As many evaluations rely on the CAV-MAE framework, providing this information is crucial for the readers and reviewers to fully grasp the methodology.

A1.1: We apologize for the missing details on the backbone that confuses the reviewer. In the revised manuscript, we have supplemented the details of CAV-MAE ([A]) and more implementation details of our RFRA in Section C. For your convenience, we attach the added statement as follows.

In the implementation, we use the CAV-MAE model as the backbone. CAV-MAE adopts an encoder-decoder-like architecture that is pre-trained on large-scale video data with both the contrastive learning and mask image modeling paradigms. The CAV-MAE encoder consists of 11 Transformer layers dedicated to each modality for the modality-specific feature extraction, alongside one Transformer layer for cross-modal fusion. The input to the CAV-MAE encoder involves 10-second video clips containing both video and corresponding audio data. For the video stream, CAVMAE samples 10 frames within each video clip and randomly selects one frame feeding into the visual Transformer encoder. For the audio stream, each 10-second audio waveform is converted into one spectrogram and then inputted to the audio Transformer encoder.

During the fine-tuning phase, we maintain the visual and audio encoders of the pre-trained model and add one randomly initialized classification head to them. The fine-tuned model is regarded as the source model and denoted as “Source (Stat. (LN & AF))”. To investigate the robustness of different fusion manners, we design another variant of the source model that utilizes 12 Transformer layers for feature extraction and performs late fusion between the classification logit of each modality. The corresponding model variant is denoted as “Source ((Stat. LN) & LF)”. During the test-time adaption phase, unless otherwise specified, all baselines update the parameters of all normalization layers rooted in the source model, i.e., referred to as “Dyn. LN”. In contrast, as depicted in Fig. 2, our default approach in the RFRA framework involves updating only the parameters of the last Transformer layer to ensure reliable fusion, denoted as “SAF”.

Q1.2: Additionally, visualizing the corruption types on both video and audio modalities would enhance the paper's clarity and help readers better understand the benchmarks.

A1.2: Thanks for your valuable suggestion. In this work, to comprehensively evaluate modality bias, we introduce different distribution shifts on the video and audio modalities for the test sets of VGGSound ([B]) and Kinetics ([C]) datasets. For the video corruptions, we follow [D] to apply 15 kinds of corruptions into the video, and each corruption has 5 kinds of severity levels for extensive validations. Specifically, the corruptions on video modality include ''Gaussian Noise", ''Shot Noise", ''Impulse Noise", ''Defocus Blur", ''Glass Blur", ''Motion Blur", ''Zoom Blur", ''Snow", ''Frost", ''Fog", ''Brightness", ''Elastic", ''Pixelate", ''Contrast", and ''JPEG". Similar to the video modality, we add 6 kinds of common audio noise (https://freesound.org) with 5 kinds of severity levels captured in the wild. Specifically, the corruptions on audio modality include ''Gaussian Noise", ''Paris Traffic Noise", ''Crowd Noise", ''Rainy Noise", ''Thunder Noise" and ''Windy Noise".

In response to your valuable suggestion, we've incorporated visualizations of corruption types in both video and audio modalities within the updated manuscript. Fig. 6 showcases the visualization results of various visual corruption types observed within the constructed Kinetics-C benchmark. Additionally, Fig. 7 presents the Mel spectrogram visualizations, highlighting the raw audio and the corresponding audio corruption types. For a comprehensive understanding, we encourage you to refer to the updated manuscript for an in-depth exploration of these visualizations.

Thank you for upgrading your score! We appreciate the time and effort you dedicated to reviewing this work.

Q1.2: Furthermore, it is also common that the corruption types continually vary in the wild, resulting in the demand for continual test-time adaption (CTTA). The proposed approach would be more solid and universal if the proposed method could work in the MIXED DISTRIBUTION SHIFTS setting (i.e., continual TTA setting).

A1.2: Following the setting of continual TTA ([F, G]), we conduct additional experiments on the Kinetics50-C benchmark under the MIXED DISTRIBUTION SHIFTS setting as the reviewer suggested. In this setting, both baseline methods (Tent and SAR) alongside our RFRA continually adapt to evolving corruption types, and the averaged performance across all corruption types is reported. To ensure comprehensive evaluations, we vary the severity levels from 1 to 5. The results are summarized in Fig. 9 within the updated manuscript. For your convenience, we attach the corresponding numerical result (regarding accuracy) in the following tables.

Notably, the performance of the vanilla TTA method (Tent) significantly degrades, especially under high-level mixed distribution shifts (e.g., 38.3 at Severity 5). In contrast, our RFRA showcases relative robustness against these shifts across different severity levels. It's worth highlighting that while our RFRA was primarily designed to address the challenge of multi-modal reliability bias, it demonstrates consistent performance superiority compared to SAR, an approach explicitly tailored for handling mixed distribution shifts in the continual TTA setting. This observation underscores the adaptability and resilience of RFRA beyond its initial design scope.

Q2.1: Some details are missing. It is not clear how many layers of the attention module are used and repurposed during test time adaption in the proposed approach. And how many parameters do these self-attention layers account for?

A2.1: We apologize for the initial oversight in providing comprehensive details. In the revised manuscript, we have supplemented the details of CAV-MAE ([H]) and implementation details of our RFRA. For your convenience, we attach the added statement as follows.

In the implementation, we use the CAV-MAE model as the backbone. CAV-MAE adopts an encoder-decoder-like architecture that is pre-trained on large-scale video data with both the contrastive learning and mask image modeling paradigms. The CAV-MAE encoder consists of 11 Transformer layers dedicated to each modality for the modality-specific feature extraction, alongside one Transformer layer for cross-modal fusion. The input to the CAV-MAE encoder involves 10-second video clips containing both video and corresponding audio data. For the video stream, CAVMAE samples 10 frames within each video clip and randomly selects one frame feeding into the visual Transformer encoder. For the audio stream, each 10-second audio waveform is converted into one spectrogram and then inputted to the audio Transformer encoder.

Thanks for your constructive reviews and suggestions. In the following, we will answer your questions one by one.

Q1.1: I think some challenging TTA settings that are orthogonal to the cross-modal reliability bias might help to improve the practicalness and impact of this work. First, in the practical multi-modal scenarios, the severity of distribution shifts might dynamically vary. It would make the work more practical if the authors could additionally investigate the robustness of the proposed method under the setting MIXED SEVERITIES.

A1.1: In this paper, we reveal the multi-modal reliability challenge for the TTA community and propose a reliable fusion and robust adaption (RFRA) approach to tackle the issue. Following the widely-used TTA evaluation protocol ([A, B, C, D]), we introduce different corruption types to either the video or audio modality. In the main paper, we reported the performance under different corruption types with a severity level of 5.

In response to your insightful suggestion, we conduct additional experiments on the Kinetics50-C benchmark, investigating the robustness of the proposed method under the setting of MIXED SEVERITY. To this end, we create test pairs for each corruption type by blending severity levels 1 through 5, resulting in 5N test pairs, where N represents the original size of the test data. After that, we shuffle the obtained test pairs and randomly choose N pairs for each corruption type. To verify the effectiveness of RFRA under the MIXED SEVERITY setting, we choose the typical TTA method Tent ([A]) and the SOTA TTA method SAR ([E]) as baselines for comparisons. The results are depicted in Fig. 8 within the revised manuscript. For your convenience, we attach the corresponding numerical result (regarding accuracy) in the following tables.

The results indicate the effectiveness of RFRA in addressing cross-modal reliability bias across various corruption types exhibiting mixed severity levels.