Partition-Then-Adapt: Combating Prediction Bias for Reliable Multi-Modal Test-Time Adaptation

Response to weaknesses

$**W1**$: Applicability to different batch sizes.

$**A1**$: Thank you for your insightful comment. We agree this is an important question as online data is often insufficient, and at times, only a limited amount is available. And addressing this problem would firmly validate the applicability of our method dealing with real-world scenarios. In fact, we provide results for PTA’s applicability to different batch sizes in Fig. 8, Appendix D, Page 14, where small batch size has been also discussed. To handle extremely small batch sizes, we implement a memory queue to store historical predictions for computing Eq. (1) and (2), effectively adapting the original computation method for offline use. Therefore, PTA still works well and consistently outperform SOTAs in such cases.

$**W2**$: The meaning of the third term in Eq. (4).

$**A2**$: Thank you for your helpful review. We acknowledge the typo in Line 174 — "penalizes" should in fact be "promotes". To avoid any misunderstanding, we will reframe Lines 174-175 as ''The third term regularizes the model by guiding $\mathcal{A}^-$ with $\mathcal{A}^+$. Specifically, it aligns the distributions of $\mathcal{A}^-$ with those of $\mathcal{A}^+$.''

$**W3**$: Definition of the confidence term $\mathcal{K}$.

$**A3**$: Thank you for your valuable feedback. We confirm that the confidence term ( $\mathcal{K}$ ) represents the maximum softmax probabilities of the current classifier for a given batch data. We will clarify this definition in Lines 133-134 as ''Let the prediction confidence of $\mathcal{X}$ be denoted as $\mathcal{K}(\mathcal{X}) = \max(\text{softmax}(\mathbf p(\mathcal{X})))$, which corresponds to the highest softmax probability produced by the current classifier for the input batch $\mathcal{X}$.''

Response to weaknesses and questions

$**WQ1**$: Supporting evidence.

$**A1**$: Thank you for the valuable suggestion. The intuition behind our use of prediction frequency stems from the well-documented effect of entropy minimization under domain shifts: it tends to amplify spurious confident predictions, often leading to a collapsed solution where only a few classes dominate [20]. This is especially problematic in multi-modal settings, where noisy modalities can further reinforce such bias. To address this, we quantify prediction bias using the frequency of predicted labels within each batch. Samples assigned to dominant classes, i.e., those predicted more frequently than the batch average, are likely influenced by such bias. Fig. 3 (c) and (d) demonstrate that our design effectively mitigates prediction bias in multi-modal domain shifts, significantly reducing false positive rates for the two most biased classes (24.2 $\rightarrow$ 5.7 and 24.0 $\rightarrow$ 6.7). Moreover, our empirical results in Fig. 5 (e) confirm that using batch-average frequency as the bias threshold outperforms static baselines. To the best of our knowledge, we are the first to propose prediction frequency as a bias indicator for partitioning reliable/unreliable samples in the MMTTA setting. We will add this discussion and cite relevant works in the final version.

Response to weaknesses

$**W1**$: Evidence of some claims.

$**A1**$: Thanks for your valuable feedback. The empirical results in Tables 1-3 and 6-7 provide direct evidence that existing methods perform less effectively under multi-modal domain shifts compared to single-modal domain shifts. For example, compared to the pre-trained model, READ achieves average performance gains of 2.7% across two independent single-modal shift scenarios and 3.6% on multi-modal domain shifts. In contrast, our method demonstrates superior gains of 3.1% (single-modal) and 9.7% (multi-modal) respectively, showing significantly stronger robustness. Moreover, we have provided a deeper analysis in Fig. 3 (a), which illustrates that existing reweighting strategies, such as UA+EW and LE+RA, are less effective under multi-modal domain shifts compared to our method. To strengthen the logical connection of the context, we will reframe Line 116 as "However, as we demonstrate in the experimentation section (Tables 1-3 and 6-7), the effectiveness of existing reweighting strategies become less effective under multi-modal domain shifts compared to single-modal domain shifts."

$**W2**$: Comparison with DeYO.

$**A2**$: Thank you for pointing this out. We have already included a comparison and discussion of representative reweighting methods in Fig. 3 (a) and Lines 239–265. Specifically, while DeYO [1] introduces a selective reweighting strategy, it operates on a strict subset of low-entropy samples with high PLPD scores (see Eqs. 8–10 in [1] and its official implementation). This design overlooks the accumulation of prediction bias that is especially critical in multi-modal domain shifts, where confident but biased predictions are common. In contrast, our PDR module first quantifies batch-level prediction bias (Eq. (1)) to partition the data into potentially reliable and unreliable subsets, and then reweights them accordingly (Eq. (2)). This allows PDR to explicitly suppress false positives and promote reliable adaptation beyond confidence alone. Moreover, we have conducted a new experiment under multi-modal domain shift on Kinetics50 using the same learning rate and update modules. Our method achieves 41.5% accuracy, significantly outperforming DeYO’s 28.5%. We will incorporate these results and discussions into Section 4 and add DeYO to Fig. 3 (a) and (b) for a more comprehensive comparison.

$**W3**$: More citations.

$**A3**$: Thank you for the helpful suggestion. We acknowledge that TTL [2] also employs weighted entropy loss. However, the reweighting strategies differ: TTL focuses on vision-language tasks by reweighting original and augmented views within the same sample, while our method operates on multi-modal domain shifts by partitioning the batch based on prediction bias and confidence. Unlike TTL and ETA, which primarily rely on entropy, our PDR module explicitly identifies and suppresses biased predictions, a key issue under multi-modal shifts. We will cite TTL and include this distinction in the related work section to clarify our contribution.

$**W4**$: Experimental details.

We sincerely value your detailed comments.

$**W4.1**$: The ''Source model'' denotes the CAV-MAE pre-trained model (see Lines 198-199) for experiments on synthetic multi-modal domain shifts, and denotes a distinct transformer-based model [10] (see Lines 201-202) for the experiments on real-world domain shifts. The ''Source'' represents the performance of the pre-trained model for direct inference without any adaptation. We will add the meaning of ''Source'' in Line 196 and Appendix C.

$**W4.2**$: We adopt CAV-MAE as the backbone architecture following [9, 31, 38]. Specifically, CAV-MAE employs an encoder-decoder architecture pre-trained on large-scale video datasets using both contrastive learning and masked image modeling. Its encoders both comprise 11 modality-specific Transformer layers for feature extraction, followed by an additional layer for cross-modal fusion. For more details, please kindly refer to [8, 31]. We will incorporate the details of CAV-MAE in Appendix B.

$**W4.3**$: As we declare in Lines 204-205, ''Following [31, 38], we update query/key/value transformation matrices of the attention layer in the fusion block''. To be more specific, the total parameters are 1.8M (1% of total), which remain the same as [31, 38].

$**W4.4**$: To eliminate the confusion, we will change Kinetics50 with multi-modal domain shifts as Kinetics50-MC (multi-modal corruption) and remain Kinetics50 with single-modal domain shifts as Kinetics50-C following your advice.

$**W4.5**$: In Tables 1-2, each number denotes the average performance over six audio corruptions under a specific video corruption. We will add the meaning of each number to the captions of Tables 1-2. We also present detailed results regarding the performance under a specific audio corruption paired with a specific video corruption in Table 10 of the Appendix.

$**W5**$: Reproduction of comparison methods.

$**A5**$: Thank you for the observation. To ensure fair comparison, we reimplemented all methods using their official code (if available), under the same adaptation protocol [31] using their default hyper-parameter settings. We report the average over 5 random seeds. Minor discrepancies with the original papers may arise due to differences in GPUs and random seeds. Even when compared to SuMi’s original results, our method outperforms it under video-modal shifts (64.7 vs. 63.9) and achieves comparable performance under audio-modal shifts (71.4 vs. 71.9). We guarantee that all methods are implemented under fair conditions. We will clarify these implementation details and hyperparameter settings in Appendix C to ensure transparency and reproducibility.

$**W6**$: Hyper-parameters.

$**A6**$: Thank you for your insightful review. We use the baseline READ's default learning rate of 1e-4. As Fig. 5 (c) and (d) already show our method's performance with varying learning rates and updating modules, we assume you seek a comparison with other methods under identical settings. For clarity, we provide the following tables:

We have the following observations: (1) our method consistently outperforms READ under the same learning rates and updating modules; (2) our method is memory-efficient, requiring significantly less GPU memory compared to READ.

$**W7**$: Add experiments on CIFAR-C and ImageNet-C.

$**A7**$: Thank you for your constructive and thoughtful feedback. Follow your advice, we run experiments on single-modal datasets, such as CIFAR10/100-C and ImgeNet-C using CNN-based backbones. We report the average accuracy over 15 types of corruptions for these benchmarks in the following table:

We observe that our method outperforms READ on single-modal datasets and matches specialized single-modal methods (surpassing on ImageNet-C). Crucially, it excels over all SOTAs in challenging multi-modal domain shifts (see Tables 1-2).

We also run experiments on CLIP (ViT-B-16). To simulate multi-modal domain shifts, we use ImageNet-C for the image branch and apply Gaussian Noise to the text branch. For your convenience, we attach the performance below:

In summary, these additional experiments further demonstrate a broader applicability of our method.

Response to weaknesses

$**W1**$: Applicability to different batch sizes.

$**A1**$: Thank you for raising this important point. We agree that small batch sizes are common in real-world, low-latency scenarios and must be addressed. To this end, we evaluate our method under various batch sizes in Fig. 8 (Appendix D, Page 14). Results show that PTA remains robust and consistently outperforms SOTAs even with small batches. To mitigate bias estimation instability, we introduce a memory queue that stores historical predictions for computing Eq. (1) and (2). This allows more stable frequency estimation without requiring large batch sizes. This design ensures that PTA remains practical and effective under constrained settings.

$**W2**$: Clarification of some assumptions.

$**A2**$: Fig. 3 (c) presents an example illustrating that high-confidence predictions can become unreliable under multi-modal domain shifts. We adopt CAV-MAE in Fig. 3 (c) to ensure a fair comparison, as it is also used by previous methods [9, 31, 38]. The results clearly show that the positive rate of high-confidence predictions drops significantly under such domain shifts. Moreover, the empirical results reveals that addressing this prediction bias issue substantially improves adaptation performance (see Tables 1-3). For results obtained with other model architectures, please kindly refer to Q/A5. Additional figures validating this observation with other model architectures will be included in the Appendix.

Response to questions

$**Q1**$: In-depth analysis of SuMi.

$**A1**$: Thank you for your constructive feedback. For fair comparison, we reimplement all comparison methods using their official code (if any) and report the average accuracy across 5 independent seeds under both single- and multi-modal shifts. SuMi outperforms the pre-trained model (Source) under single-modal domain shifts (61.01 vs. 60.58 in Table 6, and 69.88 vs. 69.32 in Table 7), but fails under multi-modal domain shifts (see Tables 1–2). The reason might originates from SuMi’s design. Specifically, the key components of SuMi are (1) uni-modal assistance (UA) with entropy-based reweighing (EW) and (2) KL-based mutual information sharing.

First, as we demonstrate in the empirical results (Lines 239-248) and the analysis (Lines 249-260 and Fig. 3 (b)), UA+EW assigns high weights to the selected candidates, regardless of whether they are false positives or not, which may poses error accumulation. Second, the KL-based loss function may not provide meaningful mutual information under severe multi-modal domain shifts, as significantly performance drop of the pre-trained model (Fig. 1 (a)) occurs. As a result, the above properties could be the reason why SuMi achieves unsatisfactory performance for MMTTA. We will add the above discussion in Section 4.

$**Q2**$: Degradation of the pre-trained model.

$**A2**$: Thank you for your insightful suggestion. We agree with your hypothesis that multi-modal domain shifts degrade model calibration more severely than single-modal shifts, likely due to conflicting noise patterns or misaligned feature distributions across modalities. Multi-modal domain shifts degrade model calibration primarily because they induce complex interactions between modalities in the fusion block, leading to degraded discrimination of fused features (Fig. 1 (a)). These skewed feature representations bias the classifier toward a few dominant classes (Fig. 1 (b)). Consequently, the normalized entropy, computed from these biased predictions, significantly impairs model calibration (Fig. 1 (c)). We will improve the presentation in Lines 31-34 with the above discussion.

$**Q3**$: Explain the effectiveness of PTA under single-domain shfits.

$**A3**$: Thank you for your thoughtful comment. We agree that PTA effectively addresses prediction bias in single-modal domain shifts, though the issue is less pronounced than in multi-modal shifts. First, our experiments show that the pre-trained model’s performance degrades under both single- and multi-modal domain shifts (Fig. 1 (a)), leading to increasingly noisy (incorrect) predictions. Entropy minimization exacerbates this by sharpening low-entropy predictions, regardless of their reliability, which propagates noise and risks a collapsed trivial solution. Empirical results from Tent (Tables 1–2 and Table 6) confirm that excessive entropy minimization with noisy predictions can degrade performance compared to the pre-trained model (Source). Second, following prior works [9, 31, 38], we use randomly sampled batch data, ensuring no intentional label imbalance. This indicates that prediction bias primarily arises from noisy predictions, further amplified by entropy minimization, rather than imbalanced data.

$**Q4**$: Add experiments on CIFAR10/100-C and ImageNet-C.

$**A4**$: Thanks for your suggestion. We provide more experiments on CIFAR-10/100-C and ImageNet-C and report the average accuracy across 15 types of corruptions (severity level 5).

We observe that our method outperforms READ on single-modal datasets and matches specialized single-modal methods (surpassing on IN-C). Crucially, it excels over all SOTAs in challenging multi-modal domain shifts (see Tables 1–2). We will add the comparison table on these datasets to help illustrate the broader applicability of our method following your advice.

$**Q5**$: Performance on other multi-modal classification models.

$**A5**$: Thank you for your valuable feedback. First, we validate our method’s effectiveness on synthetic multi-modal domain shifts (see Tables 1-2 and Tables 4-7) using CAV-MAE [8, 31] as the backbone. Second, we demonstrate our method's generalization to real-world domain shifts using a Transformer-based multi-modal classification model [10] as the backbone (see Table 3). Third, we prove our method's broader applicability using CLIP (ViT-B-16) as the backbone. To simulate multi-modal domain shifts, we use ImageNet-C for the CLIP's image branch and apply Gaussian Noise to its text branch. We report the comparison table below:

In summary, out method demonstrate its robustness on multiple multi-modal classification models.

Response to limitations

$**L1**$: Improve Fig. 6 by including the efficiency of the Source.

$**A1**$: We appreciate your feedback. The number of samples processed per second (samples/sec) by Source is 92.4. We will revise Fig. 6 by including the Source in the comparison to improve clarity, following your advice.

$**L2**$: Improve Fig. 6, and Tables 6-7.

$**A2**$: We appreciate your valuable suggestions. The unit for the number of trainable parameters in Fig. 6 is indeed millions (M). We will revise Fig. 6 by adding the unit to the y-axis for the number of trainable parameters and improve Tables 6-7 by highlighting the best-performence of comparison methods, following your advice.

Response to weaknesses

$**W1**$: Theoretical analysis.

$**A1**$：Thanks for your suggestion. To strengthen the theoretical foundation of PDR, we provide a brief analysis of why prediction bias arises and how our design mitigates it.

Suppose the output logits are $\mathbf{z}\_i = [z_{i1}, \dots, z_{iK}]$, we have the softmax probability $p\_{ik} = \frac{e^{z\_{ik}}}{\sum\_{j} e^{z\_{ij}}}$ and the entropy loss $\mathcal{L}\_{\mathrm{ent}} = -\frac{1}{N} \sum\_{i=1}^N \sum\_{k=1}^K p_{ik} \log p\_{ik}$.

We first derive the gradient of the softmax function with respect to the logits: $\frac{\partial p_{i j}}{\partial z_{i k}}=p_{i j}\left(\delta_{j k}-p_{i k}\right)$, as well as the gradient of the entropy loss with respect to the softmax outputs: $\frac{\partial \mathcal{L}\_{\mathrm{ent}}}{\partial p\_{i j}}=-\log p\_{i j}-1$. By applying the chain rule, we obtain the gradient of the entropy loss with respect to the logits as follows: $\frac{\partial \mathcal{L}\_{\text {ent}}}{\partial z\_{i k}}=\sum\_{j=1}^K \frac{\partial \mathcal{L}\_{\text {ent}}}{\partial p\_{i j}} \cdot \frac{\partial p\_{i j}}{\partial z\_{i k}}=\sum_{j=1}^K\left(-\log p\_{i j}-1\right) \cdot p\_{i j}\left(\delta\_{j k}-p\_{i k}\right)$, where the Kronecker delta: $\delta_{j k} = \{ 1 \text{ if } j = k, \text{ else } 0 \}$.

This expression shows that entropy minimization encourages the model to produce increasingly (over) confident (i.e., low-entropy) predictions by amplifying the largest logit and suppressing the others, which is also discussed in [10]. To address prediction bias in multi-modal domain shifts, PDR partitions each data batch into biased and unbiased subsets using a prediction bias indicator (Eq. (1)), which leverages batch-average prediction frequency to identify overconfident predictions, often false positives due to entropy minimization. These biased samples are regularized via entropy maximization (the second term in Eq. (2)), which counteracts noise propagation by encouraging uniform probability distributions, as supported by theoretical insights from [a, b, 31] on entropy regularization under noisy conditions.

Additionally, to balance bias and confidence levels and prevent issues like unreliable gradient from low-confidence predictions [19], we propose a quantile ranking strategy (Alg. 1). This strategy reweights unbiased samples by assigning higher weights to confident predictions and lower weights to uncertain ones, ensuring stable and reliable test-time adaptation. The theoretical reasonability of this approach stems from its alignment with robust optimization principles, where reweighting mitigates the impact of noisy outliers, as discussed in [19]. We demonstrate PDR’s consistent stability and superior performance across challenging multi-modal domain shift scenarios, including continual settings (Table 8, Appendix D) and changing environments (Section 4.4).

We will add the above theoretical analysis and discussion in the Appendix.

[a] J. Liang, D. Hu, Y. Wang, R. He and J. Feng, "Source Data-Absent Unsupervised Domain Adaptation Through Hypothesis Transfer and Labeling Transfer," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 8602-8617, 1 Nov. 2022, doi: 10.1109/TPAMI.2021.3103390.

[b] J. Li, Z. Yu, Z. Du, L. Zhu and H. T. Shen, "A Comprehensive Survey on Source-Free Domain Adaptation," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 8, pp. 5743-5762, Aug. 2024, doi: 10.1109/TPAMI.2024.3370978.

$**W2**$: The meaning of $Z$ and $\mathcal{K}$.

$**A2**$: $Z$ represents the frequency of predicted class for $x$, which is a scalar. $\bar{Z}$ is also a scalar since it denotes the average frequency of the predicted labels within a batch. $\mathcal{Z}$ is a vector, consting of all instances' bias level computed by Eq. (1). And $\mathcal{K}$ is also a vector, representing the softmax confidence. We will clarify the meaning of $Z$ and $\mathcal{K}$, and bold the vectors to improve the presentation for better clarity, following your advice.