Multimodal Unsupervised Domain Generalization by Retrieving Across the Modality Gap

We thank you for your positive and encouraging review of our submission! Please find our responses to the weaknesses below:

Response to weakness 1: We repositioned the figures as requested.

Response to weakness 2: Thank you for the suggestion! We added Figure 18 to page 27 of the revised manuscript. This is a t-SNE visualization corresponding to Figure 3 in the introduction. We randomly select 100 ImageNet class names and compute the text features corresponding to the "a photo of a {}" prompt (referred to as the base classification problem). Additionally, we use a bad augmentation selected from Table 11 with a loss of 16, and a good augmentation with a loss of 0. The features are aggregated and visualized in this figure. The augmentation with higher loss does not preserve the variance of the classifier prototypes, likely reducing the ZS accuracy and diminishing its effectiveness.

Response to weakness 3: We appreciate your suggestion to include statistical testing for the significance of the ablation results. Unfortunately, due to resource limitations, we were unable to perform enough runs of each ablation experiment to conduct meaningful statistical tests. In Tables 4 and 5 of the submission, we reported the average of three random trials. We did not include the standard deviation, as the number of trials was insufficient to provide meaningful error bars.

Thank you for the very thorough review of our work! We have made numerous revisions and hope that you can re-evaluate based on the revisions and our responses below.

Response to weakness 1:

We thoroughly revised the introduction and figures according to your suggestions. We broke Figure 1 into three figures, each with revised captions. We then repositioned each figure to be closer to the relevant text. We improved the quality of Figures 7 and 8 (formerly Figures 4 and 5).

Response to weakness 2:

We added Figure 1 to create a more intuitive visualization of the task setting. We then added a paragraph beginning at line 80 detailing a specific application scenario.

To elaborate further on the use case described in the revised introduction, traditional domain generalization methods, including unsupervised ones, rely on training datasets tailored to a specific classification task. In contrast, our MUDG paradigm removes the requirement for training data to be task-specific. This flexibility allows the same source dataset to support multiple related classification problems without the need for manual dataset re-engineering, enabling broader applicability and reducing the overhead associated with task-specific data preparation.

Consider e-commerce applications where products often require classification in multiple directions. For instance, a product like shoes might need to be categorized based on material (e.g., leather vs. synthetic), style (e.g., casual vs. formal), or usage (e.g., indoor vs. outdoor). Traditionally, this would involve manually curating datasets tailored to each specific classification task. Our approach automates this process by enabling a single, task-agnostic dataset to support various classification tasks.

Response to weakness 3:

From our understanding of the literature, a two-level data structure is generally used with both coarse and fine quantization components. The coarse quantization decreases the query time by limiting the amount of samples that must be searched through. The main goal of fine quantization is to compress the embeddings, although it does also affect recall and latency. We were primarily interested in studying the coarse quantization component, because it is clearly impacted by the modality gap issue.

In our experiments, we chose not to explore sophisticated fine quantization techniques (such as product quantization) because of the following technical issue. In FAISS, fine quantization operates on the residuals of the coarse quantization (i.e. the difference vector between the sample embedding and the closest centroid). Unfortunately, our algorithm repositions the centroids away from the sample embeddings. This increases the length of the residual vectors and decreases the accuracy of the fine quantization. A simple fix would be to store centroids in both the image space and the query space, and calculate residual vectors with respect to image centroids. We believed that including this would detract too much from the primary focus of our work.

Response to weakness 4:

We acknowledge that the proposed method does not demonstrate superior performance across all datasets in Table 2. This outcome can be attributed to several underlying factors. We primarily tuned our method on ImageNet, so the results are better on datasets with similar characteristics, such as SUN and DomainNet. Herding is a challenging coreset selection baseline that is good at selecting samples which best approximate the PDF of the underlying data distribution. Consequently, the herding method might outperform in scenarios where the retrieved dataset is similar in distribution to the target dataset. This is the case for target datasets assembled by querying a search engine, with minimal post-processing. For datasets (such as Office Home, DomainNet, and the different versions of ImageNet in Table 3) where the curators attempted to gather a wide range of target domains, our method is likely to outperform the coreset selection baselines. We emphasize that our method still achieves the best overall target score compared to these challenging baselines.

Response to weakness 5:

The finetuning parameters in Table 9 (such as learning rate, which layers to freeze, and EMA constant) were determined by training on a random 16-shot subset of ImageNet and validating on another random 16-shot subset. These parameters are not specific to our problem setting. The other parameters, listed in Table 10, are dataset-dependent and were determined using a validation set.

We performed an analysis of the sensitivity of important hyperparameters in Figure 15. We observed that increasing either $n_{neighbors}$, $m$ or $k_1$ improves the target accuracy by retrieving a larger training set, but the accuracy saturates at a certain value. Generally, increasing $n_{probe}$ also improves the target accuracy.

We thank you for your critical and encouraging review of our submission! We are happy that you found our experiments to be expensive and the visualizations to be helpful. Please find our responses to the weaknesses below:

Response to weakness 1:

In reference to Tables 2 and 3, we agree with the reviewer that the MUDG setting is less stringent than either the ZS or SFDG setting. However, we do also have many MUDG baselines, and the comparisons with the MUDG baselines are fair. We clearly label which method is under which setting, so as not to misrepresent the results. However, some readers familiar with the literature will expect to see comparisons to zero-shot methods to establish a lower bound. This presentation of results is inline with prior work, including PromptStyler. We also include an upper bound in Table 2.

We emphasize that the goal of including different ZS baselines is to present a well-calibrated lower bound, not to misrepresent results.

Response to weakness 2:

We agree. The MUDG setting is more computationally expensive than either SFDG or ZS. This is an expected tradeoff.

As a quantitative example, for ImageNet-1K, we augment each of the 1000 class names 16 times, for a total of 16,000 queries. The retrieval step took 30 seconds in total. For each query, we retrieve the 64 nearest neighbors, but this is not any slower than retrieving only one nearest neighbor when using FAISS [3] for approximate nearest neighbors search. Upon retrieval, the 64 nearest neighbors are already ranked by similarity to the query. The implementation of step 2 only compares the rank of each image relative to the queries that retrieved it. This finished in 55 seconds for the ImageNet-1K target task. Clustering (step 3) then took 9 minutes and 20 seconds on one CPU, but could be easily sped up using a GPU implementation of k-means. Finally, downloading the 96,000 selected images took 158 seconds.

Please note that computational efficiency of the retrieval process was not a goal of our work.

[3] FAISS

Response to weakness 4:

We think that there might be a misunderstanding of the ablation experiment in Table 5. This experiment intends to show that removing any of the three components (paired k-means, adaptive augmentation, or diversity preserving loss) from our method decreases the overall target accuracy. Removing the diversity-preserving loss decreases the average target accuracy by 0.3% and 0.4% on DomainNet and OfficeHome, respectively. Removing the paired k-means retrieval decreases the average target accuracy by 0.2% and 1.4% on DomainNet and OfficeHome, respectively. Removing the adaptive augmentation decreases the average target accuracy by 0.3% and 0.2% on DomainNet and OfficeHome, respectively.

Response to Question:

The reviewer suggests to include MUDG baselines in Table 3. We are unable to reproduce the PromptStyler results because they did not release code. However, we do agree with the reviewer that MUDG baselines are needed in Table 3. We ran the additional experiments; the revised rows for the ViT-B/16 model size is as follows:

"IN" stands for ImageNet. "OH" stands for Office Home.

Thank you for the great suggestions! We appreciate your highlighting of our submission's "rigorous theoretical analysis and empirical validation" and "strong practical value". Here are our responses to the weaknesses:

Response to weakness 1:

We have added Figure 1 comparing DG, UDG, and MUDG as suggested. Additionally, we included a paragraph beginning at line 80 detailing a specific application scenario to illustrate the practical relevance of MUDG.

Traditional domain generalization methods, including unsupervised ones, rely on training datasets tailored to a specific classification task. In contrast, our MUDG paradigm removes the requirement for training data to be task-specific. This flexibility allows the same source dataset to support multiple related classification problems without the need for manual dataset re-engineering, enabling broader applicability and reducing the overhead associated with task-specific data preparation.

To elaborate further on the use case described in the revised introduction, consider e-commerce applications where products often require classification in multiple directions. For instance, a product like shoes might need to be categorized based on material (e.g., leather vs. synthetic), style (e.g., casual vs. formal), or usage (e.g., indoor vs. outdoor). Traditionally, this would involve manually curating datasets tailored to each specific classification task. Our approach automates this process by enabling a single, task-agnostic dataset to support various classification tasks.

Response to weakness 2:

We would like to reiterate that our approach is only applicable to scenarios where the target visual concepts are present in the source dataset. This assumption is reasonable across all the benchmarks explored in our experiments. The LAION dataset, which we used as our source, represents a broad subset of images available on the web. For traditional computer vision benchmarks like ImageNet, retrieving relevant images from this dataset poses no challenge.

For more fine-grained target datasets, such as airplane classification, relevant images still exist within the source dataset. However, the suboptimal alignment between the labels and images in the embedding space could reduce the effectiveness of our framework.

In the e-commerce use case described earlier, the validity of our assumptions is clear. A company like Amazon, for example, likely has images for nearly every product imaginable. While these images may not be labeled for a specific classification task, their availability in a task-agnostic dataset supports our framework's applicability.

Intuition: The intuition behind retrieving relevant images and fine-tuning the model on pseudo-labeled data aligns closely with the principles of semi-supervised learning. Even though the constructed dataset lacks ground-truth labels, the retrieved images represent high-confidence samples for each class. Training on these samples enables the model to achieve high accuracy, with the caveat of inheriting biases present in the model used for retrieval.

Response to weakness 3:

We agree that the list of six contributions can be confusing to the reader. In our revision, we reduced this to two bullet points. The core challenge of our work is how to effectively retrieve images from the source dataset for the downstream task. To address this challenge, we presented a method for more accurate cross-modal search (the first contribution) and a procedure for diversified retrieval (the second contribution). The corresponding paragraphs in the introduction are clearly labeled. We hope that this improves the organization and readability of the introduction.

Thank you for the revisions and for taking the time to address my concerns. The added comparison and clearer explanation helped me better understand the importance of MUDG. I’ve decided to raise my score to 8. Appreciate your hard work!