Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noises

Thank you for your detailed review and your recognition of our work. We appreciate your insightful comments and suggestions. Here are our responses.

Q1: Method clarification.

Thank you for your feedback. For ranking process, during the method design phase, we have considered several metrics such as magnitude (Euclidean Norm) and specific dimension comparison. However, there are some drawbacks of these metrics. For example, large components in the vectors will disproportionately affect the magnitude. Specific dimension comparison ignores other dimensions which may be important and does not represent the overall vector well. Therefore, to combine multiple dimensions, we calculate the min and max of all the $h$ element by element to obtain $h_{min}$ and $h_{max}$. Then, we obtain the Q1 and Q3 through linear interpolation. Specifically, Q1 is the 25th percentile and Q3 is the 75th percentile. Consequently, from the experiments, we find that we can achieve better and more robust results than using magnitude or specific dimension comparison.

In our sample identification process, $h$ is a vector and the lower bound $Q_1-\frac{3}{2}IQR$ and the upper bound $Q_3+\frac{3}{2}IQR$ are also vectors which have the same dimension as $h$. We compare these two vectors element by element. As we illustrate in the paper, we select $h$ for adaptation if $\beta+\frac{(1-\beta)t}{iter}$ percent of the values in $h$ fall in the range [$Q_1-\frac{3}{2}IQR$, $Q_3+\frac{3}{2}IQR$] for stability. And we also explore the $\beta$ in our ablation experiments.

In case of the definition of IQR, we have given the full definition and calculation of IQR in Definition 1. For clarity, we have revised the manuscript to present a better understanding of calculating the IQR and quartiles, including the ranking criteria in the appendix in detail. For the fig3, we have revised the manuscript and cited it in our introduction section. Thank you for your suggestions.

Q2: Relationship between three strategies.

We would like to clarify that the three strategies we proposed are interconnected and not independent of each other. First, IQR is used to smooth the adaptation process to avoid abrupt huge distribution gap. IQR smoothing can help unimodal assistance identify the rich multimodal data more accurately. In the first few iterations, without IQR smoothing, it is hard for unimodal assistance to identify high-confident samples with rich information because the batch contains many strong OOD samples where the source model cannot handle well. Therefore, based on IQR smoothing, the function of unimodal assistance will be boosted, especially in mixed OOD samples. Meanwhile, IQR smoothing and unimodal assistance deal with the shift from two perspectives. IQR smoothes the adaptation process and unimodal assistance selects the data which are good for the entropy minimization. Additionally, mutual information sharing is introduced to reduce the discrepancies across different modalities and enhance the information utilization of different modalities. This is beneficial to unimodal assistance because mutual information sharing can help the unimodal branch produce more confident unimodal predictions, thus enhancing the unimodal assistance strategy. Therefore, each component is designed to complement the others, enhancing the overall effectiveness of our approach. And we can observe from the ablation experiments that the strategies combined can further improve the performance of SuMi.

Thank you for your detailed response. I'll keep my score.

Q3: Effectiveness of the method across multiple modalities.

Thank you for your suggestions. To evaluate the performance of SuMi on additional datasets, we select MOSI as the dataset. We choose CMU-MOSI for two primary reasons. First, it includes a text modality, allowing us to work with datasets that extend beyond just video and audio. Second, MOSI encompasses three modalities—text, image, and audio—enabling us to evaluate the performance of SuMi across a dataset with more than two modalities.

Here, we introduce four types of corruptions for text modality. Specifically, we introduce random deletion of word or character (RD), random insertion of word or character (RI), word shuffling (WS) and sentence permutation (SP). We have added detailed explanations for these corruptions to the appendix of our manuscript. For backbone, we use the stacked transformer blocks trained on MOSI dataset. Then, we fine-tune the model on corrupted MOSI. Here are the results.

For strong OOD, we use Corrn to denote that n modalities are corrupted, miss n to denote n modalities are missing and corr+miss to denote both missing modalities and corruption modalities are present. From the table, we can observe that in dataset with more modalities, SuMi can also outperform existing method, demonstrating its effectiveness. We have added these results in our revised manuscript.

We have made revisions to our paper in response to your suggestions. Specific details can be found in the updated manuscript. If you have further questions, please feel free to reach out.

Thank you for your detailed review and your recognition of our work. We appreciate your insightful comments and suggestions. Here are our responses.

Q1: Figure 2 needs revision; it’s currently unclear what is being input to the fusion layers.

Sorry for the confusion. In figure 2, we input the filtered representations by IQR smoothing into the fusion layers. Specifically, after the encoders, we will obtain representations of each modality. Then, we select the representations using IQR smoothing and input these selected features into the fusion layers. We input the features into the same fusion layer three times to get multimodal prediction and unimodal predictions. Here we present the pseudo-code for a better illustration:

# afeat and vfeat are the features of audio and video modalities selelcted by IQR smoothing
aoutput = fusion(x=afeat)
voutput = fusion(x=vfeat)
feats = cat[afeat, vfeat]
outputs = fusion(x=feats)

We have revised the figure in our paper for a better presentation.

Q2: Figure 3(c) lacks clear explanations for the X and Y axes.

We would like to clarify that the labels for the X and Y axes are included in the figure caption. However, we understand that they may not have been sufficiently clear. We have revised the figure and its caption to enhance clarity and ensure that the axes are easily interpretable for all readers.

Q3: The settings (weak/strong OOD) should be labeled in Tables 1 and 3.

Thank you for your feedback. Table 1 and 3 present the results of data with corrupted video modality (i.e. only weak OOD data of video modality). Table 2 and 4 include the strong OOD (multiple modalities corrupted) and have been labeled in the tables.

Q4: $\mu$ is not in Equation 4.

We would like to clarify that $\mu$ is indeed included in equation 4. We've double-checked the equation and can confirm that $\mu$ is correctly included. It is a trade-off between the two modalities.

Q5: How to obtain the predictions when on modality is missing?

In our approach, we follow established practices from previous research by substituting any missing modalities with zero vectors. This allows us to maintain the input dimensions required by the network while enabling it to process the available data effectively. We have clarified this methodology in the revised manuscript to ensure that it is clearly articulated.

We have made revisions to our paper in response to your suggestions. Specific details can be found in the updated manuscript. If you have further questions, please feel free to reach out.

Q3: Explanation of IQR and exploration of smoothing process.

IQR stands for interquartile range. We choose this specific quantile range because it effectively captures the central tendency and variability of the data while being less sensitive to outliers compared to other ranges. Specifically, there are three advantages:

• Robustness: The IQR is less sensitive to extreme values compared to other quantile ranges, making it a more reliable measure of central tendency and variability in datasets that may contain outliers. Unlike the range, which can be affected by extreme values, the IQR focuses on the central values, making it a robust measure of variability.

• Focus on Central Data: By concentrating on the middle 50% of the data, the IQR provides a clearer picture of the distribution and helps to mitigate the impact of skewed data.

• Simplicity and Interpretability: The IQR is straightforward to calculate and interpret, making it an effective choice.

Due to these advantages, interquartile range becomes a popular method in the statistical learning.

In case of the smoothing process, we have indeed explored alternative functions for smoothing, including logarithmic and exponential functions. These alternatives are tested to assess their effectiveness in enhancing the performance of our model. In our paper, we opted for the simple linear process for its clarity. Here are the results using different smoothing functions on mixed distribution shifts of Kinetics50-C. Specifically, we use $f(t)=\log (\frac{(e-1)t}{iter}+1)$ as log function and as $f(t)=\exp{(\frac{t \ln 2}{iter})}-1$ exp function, which ensures that the value range of the function is between 0 and 1. Then the selected samples of iteration t are in [Q1-$\frac{3}{2}f(t)$IQR, Q3+$\frac{3}{2}f(t)$IQR].

From the table, we can observe that using $f(t)=\exp{(\frac{t \ln 2}{iter})}-1$ function can improve the performance of the model slightly. From the properties of the exp function, it can be seen that the function grows slowly when the variable t is small and quickly when the variable is large. For log function, it grows quickly when the variable t is small and slowly when the variable is large. This indicates that slowing down the smoothing process in the initial phase helps the model's performance. Additionally, we can observe that the function will not affect the performance drastically, indicating the effectiveness of smoothing process itself. Therefore, we opted for the simple linear process in our paper for its clarity and simplicity.

Q4: Hyperparameters selection.

Thank you for your feedback. In the ablation experiments, we explore the impact of $\mu$ on the performance of the model. Although we find that adding weight to the weak modality will help to utilize the multimodal features, the performances with different $\mu$ are stable, indicating the stability of unimodal assistance strategy. For instance, the accuracy improves only around 0.1 to 0.5 when increasing $\mu$ on Kinetics50, which is small compared to adding the strategy itself. Therefore, without much hyperparameter selection, SuMi can also achieve good results. This also indicates that unimodal assistance can select high-confident samples with rich multimodal information.

We have made revisions to our paper in response to your suggestions. Specific details can be found in the updated manuscript. If you have further questions, please feel free to reach out.

Thank you for your detailed and constructive comments on our paper. We appreciate your insightful comments and suggestions. Here are our responses.

Q1: Discussing the practical applications of the method in detail.

We would like to clarify that the scenarios we address are indeed common in real-world applications.

Let us first illustrate weak OOD and strong OOD samples in multimodal data. Weak OOD samples are those with only one modality corrupted. Strong OOD samples are those with multiple samples corrupted or missing modality. Strong OOD samples are quite common in real-world applications. For instance, consider a vehicle operating during a thunderstorm: the camera may capture visual information affected by raindrops, leading to noisy visual data, while the audio sensors are also impacted by the sounds of thunder and rain, resulting in noisy audio signals. Additionally, situations where a sensor fails due to weather conditions or external factors can lead to one modality being missing while another is corrupted. These strong out-of-distribution (OOD) conditions, where multiple modalities are affected by environmental factors, are quite prevalent. This illustrates the practical relevance of our task and underscores the importance of addressing both missing modalities and sensor disturbances in a cohesive framework, especially at test time. Furthermore, our intent is to handle distribution shifts in real-world applications. Missing modality is a common issue in multimodal noises. Meanwhile, different from existing methods to address missing modalities at train time with the labels, our method address it at test time without labels.

Q2: Effectiveness of the method across multiple modalities.

Thank you for your suggestions. To evaluate the performance of SuMi on additional datasets, we select MOSI as the dataset. We choose CMU-MOSI for two primary reasons. First, it includes a text modality, allowing us to work with datasets that extend beyond just video and audio. Second, MOSI encompasses three modalities—text, image, and audio—enabling us to evaluate the performance of SuMi across a dataset with more than two modalities.

Here, we introduce four types of corruptions for text modality. Specifically, we introduce random deletion of word or character (RD), random insertion of word or character (RI), word shuffling (WS) and sentence permutation (SP). We have added detailed explanations for these corruptions to the appendix of our manuscript. For backbone, we use the stacked transformer blocks trained on MOSI dataset. Then, we fine-tune the model on corrupted MOSI. Here are the results.

For strong OOD, we use Corrn to denote that n modalities are corrupted, miss n to denote n modalities are missing and corr+miss to denote both missing modalities and corruption modalities are present. From the table, we can observe that in dataset with more modalities, SuMi can also outperform existing method, demonstrating its effectiveness. We have added these results in our revised manuscript.

Q3: Limited Scope in Addressing Diverse Distribution Shifts.

To evaluate the robustness of SuMi in addressing real-world distribution shift, we conduct experiments on two datasets (MOSI and CH-SIMS). Specifically, MOSI and CH-SIMS are multimodal sentiment analysis datasets which include three modalities. They contain different topics of conversations, different speakers, and different recording environments which can all be seen as real-world distribution shifts. We use stacked Transformer blocks as the backbone and pre-train the model on MOSI and CH-SIMS as the source model for the setting MOSI$\rightarrow$ CH-SIMS and CH-SIMS$\rightarrow$ MOSI, respectively. Here are the results.

As shown in the table, we can observe that in real-world distribution shifts, SuMi can still outperform existing methods, showing its robustness.

Q4: Fragmented Methodological Motivation.

We would like to clarify that the three strategies we proposed are interconnected and not independent of each other. First, IQR is used to smooth the adaptation process to avoid abrupt huge distribution gap. IQR smoothing can help unimodal assistance identify the rich multimodal data more accurately. In the first few iterations, without IQR smoothing, it is hard for unimodal assistance to identify high-confident samples with rich information because the batch contains many strong OOD samples where the source model cannot handle well. Therefore, based on IQR smoothing, the function of unimodal assistance will be boosted, especially in mixed OOD samples. Meanwhile, IQR smoothing and unimodal assistance deal with the shift from two perspectives. IQR smoothes the adaptation process and unimodal assistance selects the data which are good for the entropy minimization. Additionally, mutual information sharing is introduced to reduce the discrepancies across different modalities and enhance the information utilization of different modalities. This is beneficial to unimodal assistance because mutual information sharing can help the unimodal branch produce more confident unimodal predictions, thus enhancing the unimodal assistance strategy. Therefore, each component is designed to complement the others, enhancing the overall effectiveness of our approach. And we can observe from the ablation experiments that the strategies combined can further improve the performance of SuMi.

We have made revisions to our paper in response to your suggestions. Specific details can be found in the updated manuscript. If you have further questions, please feel free to reach out.

Thank you for your detailed and constructive comments on our paper. We appreciate your insightful comments and suggestions. Here are our responses.

Q1: Multimodal Data Diversity.

Thank you for your suggestions. To evaluate the performance of SuMi on extra datasets, we select MOSI dataset. We choose CMU-MOSI for two primary reasons. First, it includes a text modality, allowing us to work with datasets that extend beyond just video and audio. Second, MOSI encompasses three modalities—text, image, and audio—enabling us to evaluate the performance of SuMi across a dataset with more than two modalities.

Here, we introduce four types of corruptions for text modality. Specifically, we introduce random deletion of word or character (RD), random insertion of word or character (RI), word shuffling (WS) and sentence permutation (SP). We have added detailed explanations for these corruptions to the appendix of our manuscript. For backbone, we use the stacked transformer blocks trained on MOSI dataset. Then, we fine-tune the model on corrupted MOSI. Here are the results.

For strong OOD, we use Corrn to denote that n modalities are corrupted, miss n to denote n modalities are missing and corr+miss to denote both missing modalities and corruption modalities are present. From the table, we can observe that in dataset with more modalities, SuMi can also outperform existing methods, demonstrating its effectiveness. We have added these results in our revised manuscript.

Q2: Insufficient Comparative Analysis with Sample Selection-Based TTA Methods.

To provide clearer insights into the distinct benefits of SuMi, we combine our strategies into other methods and present the results here.

DeYO proposes pseudo-label probability difference for sample identification which is parallel to our selection strategy (IQR+UA). EATA also proposes its own sample selection method. Therefore, we replace their sample identification methods with ours (IQR+UA). For their combinations with MIS, we just add MIS loss function to their objectives. From the table, we can observe that our strategy can enhance their performances. This indicates the effectiveness and distinct benefits of SuMi's components.