Test-time Adaptation for Cross-modal Retrieval with Query Shift

Thanks for the detailed comments. In the following, we will answer your questions one by one.

Q1: The TCR method proposed in the paper performs model adaptation at test time, which may increase additional computational costs. It is recommended that the authors analyze the computational complexity of the model and the additional cost incurred.

A1: Thanks for your comments. In response to your insightful suggestion, we conduct additional experiments to analyze the efficiency of TCR. To this end, we choose the pre-trained model BLIP as the source model and perform zero-shot retrieval on the COCO dataset. We measure the GPU time during the test-time adaptation phase. Results are summarized in Table 19 within the revised manuscript. For your convenience, we attach the corresponding results in the following table.

Note that the learnable parameters of all the methods are the same for a fair comparison. The results underscore that TCR achieves adaptation more efficiently than the augmentation-based method DeYO[A]. Compared to the vanilla Tent[B] and EATA[C] (only low-entropy samples are employed for optimization), TCR requires only a negligible additional time cost, primarily due to the nearest neighbor selection in the query prediction refinement module.

Q2: Are the COCO-C and Flickr-C datasets constructed by the authors themselves? It seems that the paper does not explain whether the results of the baseline methods for comparison were obtained by the authors' own experiments or cited from their respective articles. If they were obtained through their own experiments, it should be clarified whether such comparisons are fair (whether they were trained on the new baselines), which is quite confusing for readers.

A2: We apologize for the confusion arising from the initial presentation. Following the setting in [D], we have introduced 31 types of corruptions to establish the Query Shift benchmarks (i.e., COCO-C and Flickr-C datasets). The details of the benchmarks are provided in Appendix B.1 and the data processing code is available at https://github.com/Jielin-Qiu/MM_Robustness. Specifically, for image corruptions, we employ image_perturbation/perturb_COCO_IP.py to construct image corruptions for the COCO dataset and image_perturbation/perturb_Flickr30K_IP.py for the Flickr dataset. For text corruptions, we utilize text_perturbation/perturb_COCO_TP.py to construct text corruptions for the COCO dataset and text_perturbation/perturb_Flickr30K_TP.py for the Flickr dataset.

In the paper, we aim to achieve online adaptation for cross-modal retrieval models under query shift, which is a novel challenge. Unfortunately, existing test-time adaptation (TTA) methods overlook the query shift in cross-modal settings and do not address this challenge. Moreover, most existing TTA methods are specifically designed for the recognition task and cannot be directly employed for the cross-modal retrieval task. To solve the problem, we propose a simple baseline so that the recognition-oriented TTA methods could be employed for the cross-modal retrieval task. For a fair comparison, the optimizer, learning rate, and training parameters across all the methods are the same. Besides, we carefully set the temperature for baselines on various datasets, as detailed in Appendix B.2. For your convenience, we attach the corresponding content as follows.

To guarantee the performance of the baselines, we select the optimal temperature (Eq. 1) for the TTA baselines upon each dataset. According to Fig. 4(a), the temperature is fixed as $0.01$ for COCO-C, Flickr-C, COCO, Flickr, and Nocaps datatsets, $0.001$ for Fashion-Gen dataset, and $0.0001$ for CUHK-PEDE and ICFG-PEDES datasets.

The code will be released upon acceptance of the paper.

How the number of selected candidates impacts the results. We conduct more experiments to investigate how the number of selected candidates affects performance. To this end, we directly perform zero-shot retrieval experiment on the COCO dataset with pre-trained BLIP as the source model. In the paper, we retrieve the most similar sample from the gallery set for each query, thus the number of selected candidates is equal to the batch size $B$. In the additional experiment, we vary the number of selected candidates at different values, i.e., $[0.2B, 0.5B, B, 2B, 5B, 10B, 50B]$. Specifically, assume that the number of selected candidates is $\lambda B$, where $\lambda$ is an integer. When $\lambda < 1$, we randomly select $\lambda B$ candidates from the original $B$ selected candidates. When $\lambda \geq 1$, we retrieve the most similar $\lambda$ candidates from the gallery set for each query, forming a new set of $\lambda B$ selected candidates. The results are depicted in Table 18 within the revised manuscript. For your convenience, we attach the corresponding numerical results in the following tables.

From the results, one could observe that enlarging the number of selected candidates would significantly degrade the performance. Such a phenomenon indicates that an excessively large number of candidates may lead to the underfitting issue, which highlights the necessity and effectiveness of the query refinement module.

Q4: In Section 3.2.2, given the shift between the source and target domains, it is unclear why source-domain-like data can be directly selected based on centers. Could the authors provide further analysis and explanation on this approach?

A4: We appreciate your feedback. As illustrated in Fig. 1(c), we observe that distribution shift would diminish the modality uniformity, defined as the average distance between all samples and the modality center (Eq. 14 in the manuscript). For your convenience, we have attached the corresponding equation below.

$$\text{Uniformity}=\frac{1}{N^{Q}}\sum_{i=1}^{N^{Q}}\|\mathbf{z}_{i}^{Q}-\overline{\mathbf{Z}}^{Q}\|.$$

In other words, the distribution shift would narrow the distance between samples and their modality centers. Therefore, we conclude that data from the source domain should exhibit higher modality uniformity. Based on the conclusion, we select samples farther from their modality centers as the source-domain-like data, since these samples enjoy higher modality uniformity.

Q5: In section 3.5，the definition of $S(x_{i}^Q)$ in Equation (11) lacks corresponding theoretical analysis.

A5: We apologize for the initial oversight regarding the theoretical analysis in Section 3.5. The proposed noise-robust adaptation loss (Eq. 11) aims to achieve robustness against heavy noise by excluding high-entropy query predictions from adaptation and assigning higher weights to query predictions with lower uncertainties. Specifically, for a given query sample $x_{i}^{Q}$, let its entropy be denoted as $E(x_{i}^Q)$.

When $E(x_i^Q) \geq E_m$, the weight $S(x_i^Q)$ is defined as

$$S(x_i^Q) = 0.$$

In this case, $x_{i}^{Q}$ is excluded from optimization. Such a design prevents high-entropy (i.e., noisy) query predictions from degrading performance, as their gradients produced by entropy loss might be biased and unreliable.

When $0 \leq E(x_i^Q) < E_m$, the weight $S(x_i^Q)$ is positive and defined as

$$S(x_i^Q) = 1 - \frac{E(x_i^Q)}{E_m}.$$

The weight is inversely proportional to entropy, i.e., the weight decreases as entropy increases. Formally,

$$S(x_i^Q) \propto \frac{1}{E(x_i^Q)}.$$

The adaptive weighting strategy enjoys two advantages. On the one hand, query predictions with lower entropy (i.e., reliable predictions) are assigned with higher weights, thus guiding the optimization. On the other hand, query predictions with higher entropy (i.e., uncertain predictions) are assigned with lower weights, thereby preventing overfitting on noisy query predictions.

Thanks for the insightful reviews. We will answer your questions one by one in the following.

Q1: This research setting is limited by the assumption that each query batch contains i.i.d. samples. However, in real scenarios, query shift may occur unpredictably, introducing non-i.i.d. data within the same batch. This raises concerns about the method’s applicability under such conditions.

A1: In order to address your concerns, we conduct more experiments on the COCO-C benchmark, investigating the robustness of the proposed TCR under non-i.i.d. settings (i.e., Mixed Severity Levels and Mixed Corruption Types). Specifically,

• Mixed Severity Levels: For each corruption, we create the test pairs by selecting $1/m$ of the data from each severity level, resulting in a total of $N$ test pairs, where $m$ is the number of severity levels and m=5 / 7 / 2 for the image / character-level / word-level corruptions.

• Mixed Corruption Types: For the text retrieval, we construct the test pairs by selecting 1/16 of the data from each image corruption (1 through 16), resulting in a total of $N$ test pairs. For the image retrieval, we create test pairs by selecting 1/15 of the data from each text corruption (1 through 15), resulting in a total of $N$ test pairs.

To verify the effectiveness of TCR under the Mixed Severity and Mixed Corruption Types settings, we choose the typical TTA method Tent ([A]) and the SOTA TTA methods EATA ([B]), DeYO ([C]) as baselines for comparisons. In the experiment, we carry out the experiments on the COCO-C benchmark, and the corresponding results are depicted in Tables 13-15 within the revised manuscript. For your convenience, we attach the corresponding numerical results (regarding Recall@1) in the following tables.

Note that for the Mixed Corruption Types setting, there are five levels of the mixed corruptions in text retrieval, corresponding to the image corruptions with five severity levels. For image retrieval, the severity levels of character-level / word-level / sentence-level text corruptions are 7 / 2 / 1. Thus, we select the two highest severity levels for character-level and word-level corruptions, and combine them with sentence-level corruptions, resulting in two levels of the mixed corruptions.

Thanks for your valuable reviews. We would like to address your concerns one by one in the following.

Q1: In Section 4.2, it is said that “We compare TCR with five SOTA TTA methods (Tent (Wang et al., 2021), EATA(Niu et al.,2022), SAR(Niu et al.,2023), READ(Yang et al.,2024), and DeYO...”. These methods should be introduced in Section 2.2 of the related work part.

A1: Thanks for your valuable suggestions. We apologize for the missing details on the baselines. In response to your constructive feedback, we provide more detailed introdution of the baselines, which can be found in Appendix B.3 due to space limitations. For your convenience, we attach the added statement as follows.

Test-time Adaptation (TTA) aims to reconcile the distribution shifts in an online manner. Towards achieving this goal, fully TTA ([A]) has been proposed, which fine-tunes the BatchNorm layers by minimizing entropy during the test phase. EATA ([B]) employs a Fisher regularizer to limit excessive model parameter changes and filter out high-entropy samples via the selection strategy. SAR [C] removes high-gradient samples and promotes flat minimum weights, enhancing robustness against more challenging TTA scenarios such as mixed domain shifts, single-sample adaptation, and imbalanced label shifts. READ ([D]) proposes a noise-robust adaptation loss and reliable fusion module to tackle the reliability bias challenge in the multi-modal setting. DeYO ([E]) reveals the unreliability of treating entropy as the confidence metric and establishes a novel metric by measuring the difference between predictions before and after applying an object-destructive transformation.

Q2: Line 212, ,where Q and G denotes as query modality and gallery modality for clarity in the following. Change to “denote”

A2: Thanks for your careful reading. We apologize for the typos and have revised them in the updated manuscript.

Q3: Tables 1 and 2 appear too early. They should not be on Page 7 but on the page where they are referred for the first time.

A3: Thanks for your valuable suggestions. We apologize for the misplacement of Table 1 and Table 2 in the submission and have revised them in the updated manuscript.

Reference:

[A] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.

[B] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In ICLR, 2023.

[C] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In ICML, 2022.

[D] Mouxing Yang, Yunfan Li, Changqing Zhang, Peng Hu, and Xi Peng. Test-time Adaption against Multi-modal Reliability Bias. In ICLR, 2024.

[E] Jonghyun Lee, Dahuin Jung, Saehyung Lee, Junsung Park, Juhyeon Shin, Uiwon Hwang, and Sungroh Yoon. Entropy is not enough for test-time adaptation. In ICLR, 2024.

Q.3: This paper introduces several hyperparameters, such as the temperature parameter (τ) for controlling the trade-off between smoothness and sharpness, and others for balancing the different loss terms. How sensitive is TCR to these hyperparameters, and how easy is it to tune them for new domains? Some more results from these ablations studies will be very beneficial.

A.3: Thanks for your comments. There are two hyperparameters in the paper, i.e., the temperature $\tau$ for controlling the trade-off between smoothness and sharpness, the trade-off parameter $t$ for controlling the intra-modality uniformity. For parameter sensitivity analysis, we have conducted ablation studies about $\tau$ and $t$ in Fig. 4(a) and Fig. 4(b), respectively. For your convenience, we attach the corresponding numerical results of Fig. 4(a) in the following tables.

The results denote that TCR demonstrates stable performance within the range of [0.001,0.05] and achieves the best performance when $\tau=0.02$.

In response to your concern, we conduct more ablations studies on the trade-off parameter $t$ under "Flickr-C", "ICFG2CUHK", "Base2Fashion" settings. The results are summarized in Fig. 4(b) within the updated manuscript. For your convenience, we attach the corresponding numerical results (regarding Recall@1) in the following tables.

The results indicate that TCR is not sensitive to the choice of the parameter $t$.

Besides, we apologize for the omission of the sensitivity analysis for these hyperparameters. In the revised manuscript, we have supplemented the detailed parameter analysis. For your convenience, we attach the added statement as follows.

As shown in Fig. 4(a), we observe that: i) selecting an appropriate temperature for the existing TTA approach across various datasets is challenging; ii) even a low temperature (e.g., $1e-4$) is a better setting across all datasets, the performance degrades as a low temperature tends to make model overfitting on noisy query prediction. In contrast, the query prediction refinement module not only stabilizes the temperature setting for all the datasets but also prevents the model from either underfitting or overfitting by excluding some irrelevant samples in the gallery. Besides, TCR demonstrates stable performance within the range of [0.001,0.05] and achieves the best performance when $\tau=0.02$. As depicted in Fig. 4(b), one could observe that TCR is not sensitive to the choice of $t$.

Thanks for your constructive reviews and suggestions. In the following, we will answer your questions one by one.

Q1: Given the variety of potential shifts in real-world data (e.g., subtle cultural variations, extreme distortions, rare domain-specific content), how does TCR perform across these different types of shifts?

A1: Thanks for your constructive comments. In the submission, we have reported the experiment results on a variety of real-world query shift, including noise, blur, and weather (e.g., frost, snow, and fog) distortions. Besides, we have evaluated TCR on the rare domains, such as ReID domain, e-commerce domain and natural image domain. Notably, we have carried out the experiments to verify that TCR could handle the maximum severity level of the corruptions (Tables 1-2 and Tables 7-8). In other words, TCR is able to address the most extreme distortions, which are highlighted and widely acknowledged in previous works ([A] [B] [C]).

In response to your insightful suggestion, we conduct additional experiments in the even rarer remote sensing domain. Specifically, we choose the BLIP as the source model and perform zero-shot retrieval on the remote sensing datasets RSICD ([D]) and RSITMD ([E]). To verify the effectiveness of TCR, we choose the typical TTA method Tent ([A]), the SOTA TTA methods EATA ([F]) and DeYO ([G]) as the baselines for comparisons. Results are summarized in Table 20 within the revised manuscript. For your convenience, we attach the corresponding results in the following tables.

The results indicate that TCR could also achieve the best performance in even rarer remote sensing domain.

Q.2: Could the model's performance degrade if it encounters shifts it was not explicitly evaluated against? A thorough breakdown of the model’s robustness to a diverse set of query shifts would strengthen the understanding of its general applicability.

A.2: Thanks for your comments. In this paper, we have evaluated TCR on the Flickr and COCO datasets with image and text corruptions, which simulate real-world query shift. It is worth noting that we have introduced a total of 130 perturbations across various severity levels, comprising 80 for the image modality and 50 for the text modality. Specifically, the image modality includes 16 types of corruption, each with 5 levels of severity, while the text modality comprises 15 corruption types with 7/2/1 severity levels for character-level/word-level/sentence-level corruptions. Besides, we conduct experiments on the datasets with real-world query shift, including the ReID, e-commerce, and natural image domains.

In Appendix D.3 of the manuscript, we have reported the experiment results on the 130 perturbations in Fig. 7 and Fig. 8. Specifically, we carry out the experiments on the COCO-C benchmark and report the average performance across various severity levels for each corruption. For your convenience, we have included a summary of these results (regarding Recall@1) in the following tables.