We thank the reviewer rwX7 for their thoughtful feedback. We are happy to read that the reviewer found our benchmark comprehensive with practical insights for real-life applications and for acknowledging that our work is open-source, reproducible and easily extendable.

We address the reviewer's comments point by point below.

• Outdated methods: We thank the reviewer for their feedback and refer them the detailed general response about deep DA methods that are completed in the reponse by 4 new methods with one from NeurIPS 2023.

• Weak correlation: We believe that the weak correlation between modalities is a major result of this paper, highlighting the need for the ML community to go beyond Computer Vision applications and investigate other, more challenging modalities. DA has clearly not been solved outside of CV, and this benchmark aims to demonstrate that. There is for the moment no silver bullet in DA, and we believe it is an important contribution to clearly show this through an open, diverse and reproducible benchmark.

• No new theory: By design, our work provides a benchmark which is an empirical assessment of the field of DA. This is in line with the Benchmark and datasets topic from the ICLR call for paper. We do not see which new theories could be provided in this scope. Note that we added more theoretical insights to analyse the observed performance and to better link them to the literature in Section 4.1 of the revised paper (see first answer to Reviewer PVS6).

We thank the reviewer qzy2 for their positive feedback and for acknowledging our work's clarity, our comprehensive benchmark and our extensive experiments. We address the reviewer's concerns point by point below.

1. Figure 1 Clarity:

We thank the reviewer for their valuable feedback. Figure 1 illustrates the different data shifts and assumptions typically studied in the DA literature. The focus is on a binary classification task for visualization purposes but the shifts and assumptions are not limited to it (see [3] for a formalization of the shifts). We have updated the caption of Figure 1 in the revised paper to make it clearer. If the figure is still unclear, we encourage the reviewer to let us know.

[3] Quinonero-Candela, Dataset Shift in Machine Learning. MIT Press 2009

2. Lack of recent deep methods:

We refer the reviewer to the general comment on this point. Regarding the work mentioned :

• [1] is a paper for multi-source DA which is not applicable to our pairwise DA benchmark approach. Note that in [1], the best and relatively close competitor is DeepCoral, which is present in our current benchmark.
• [2] is a task specific DA approaches for time-series based on frequency approach. Our benchmark proposes only one time-serie dataset (BCI) for which frequency representation have limited performance. This is why we used spatial covariance representation and Riemanian geometry for representation, which actually leads to best performances with Linear OT over deep DA (maybe also due to relativeley small training datasets).

[1] Quantitatively measuring and contrastively exploring heterogeneity for domain generalization." In SIGKDD, pp. 2189-2200. 2023.

[2] Domain adaptation for time series under feature and label shifts. In ICML, pp. 12746-12774. 2023.

3. Score on simulated datasets:

We thank the reviewer for this interesting question. We believe that deep learning methods are unlikely to perform well on the simulated datasets (except for the subspace shift), since they are unstructured and low dimensional (only 2 features) but providing such results would be interesting. Given the time constraints, we focused on adding new Deep DA methods for the rebutals but we will run the deep DA methods with nonlinear MLP on the simulated datasets for the final version.

4. Using supervised scorers:

In realistic application of unsupervised DA, we do not have access to target labels. This is why unsupervised scorers are necessary. While our benchmark includes target labels to measure generalization performance, the unsupervised scorers, which are the ones that can be used in practice, do not rely on these labels. One of the main contributions of this benchmark is its study of unsupervised scorers. This provides practitioners with a sense of how much they can trust these scorers and their methods.

We thank the reviewer for their detailed and encouraging feedback. We appreciate that the reviewer found our contribution to UDA excellent with our open-source, reproducible and useful benchmark for researchers and practicioners. Below, we address the reviewer's concerns point by point.

1. Lack of Clarity in Figure 1:

We thank the reviewer for their comment. Figure 1 is provided to give an illustration of the theoretical shifts tested with the simulated datasets. The color of the samples indeed represent the class and we selected an example where one of the class has multiple modes (what you refer to as clusters) because that is something that occurs often in ML (for instance a class dog contains very different groups of dogs). We have modified the caption in the revised paper to describe more precisely Figure 1. The subspace shift example cannot be cast as a conditional shift because each classes are moved diferently but the discriminant subspace assumption is exact. We are open to suggestions from the reviewer to make it better since illustrating this assumption in 2D turned out to be challenging (maybe in 3D with an additional noise feature?).

2. Subspace Shift Classification:

We would like to clarify that the term "subspace-shift" refers to a specific type of domain shift, characterized by the assumption that an alignment of source and target conditional and marginal distributions exists within a specific subspace. Based on the helpful feedback from the reviewer, we realized that the term "subspace-shift" could be misunderstood as referring to the methodology used to address this type of shift. To prevent such confusion, we have explicitly clarified that it denotes a shift occurring in the orthogonal complement of the invariant subspace $\mathcal{Z}$ and illustrate the subspace in the Figure.

3. Limited Analysis of Dataset and Shift Impact:

In real world applications, the traditional theoretical shifts are often not representative of the data complexity. The shifts are usually a combination of several of them, and characterizing which shift is present in a real dataset is not easy. This is a wide open fundamental question in unsupervised DA and out of scope for our benchmark. Yet, we agree that this is an interesting question. To adress the reviewer's concern, we have added a paragraph discussing the shift and how well they can be compensated in Sections 4.1 and 4.2.

4. Impact on Method Performance:

We added in Section 4.1 of the revised paper a discussion on the matter as well as important take-aways for DA researchers and practicioners. We note that, as explained above, for real data such analysis is more complex because there is no unique shift type clearly present in the data but rather a mix of several shifts.

5. Missing Methods:

We agree with the reviewer that pseudo-labeling methods are important. We included DeepJDOT, which can be seen as a pseudo labeling approach since it uses the prediction from a classifier inside the loss. The suggested MIC approach is restricted to CV applications, and we believe it would not fit the scope of our benchmark, as described in the general answer. Note that we included more recent ones like SPA [1].

6. Lack of Clarity on Scorer Variability within the Same Method:

We thank the reviewer for the suggestion. We added in the revised paper a visualization of the average rank of the scorers in Figure 4 in Appendix D.4 and a spider plot reporting the average rank for each methods (scorer performance depends on the methods) when comparing different scorers in Figure 5.

We thank the reviewer PVS6 for their valuable comments. We are happy to hear that the reviewer found our benchmark comprehensive and that our realistic model selection adressed challenges in the DA literature. We address the reviewer's concerns point by point below.

1. Lack of theoretical insights:

We thank the reviewer for pointing out that the theoretical insights were not clear enough. We have added a more detailed discussion on the benchmark's results in Section 4.1. The main points are as follows: - On simulated data with known shifts, DA methods tend to show a significant gain on the shift they were designed for. For instance, mapping methods perform well under conditional shift but struggle under target shift, which is a known issue for this kind of methods (Redko et al. 2019). - The performance of reweighting methods is often close to the performance of the Train Src baseline on real datasets. This can be explained by the fact that supports of the source and target distributions significantly differ. In this case, reweighting methods are known to underperform (Segovia-Martín et al., 2023). - The performance of mapping methods is dataset-dependent, potentially due to the number of classes and presence of target shift. - Very regularized transformations are the best in average accross all modalities. The four methods, LinOT, CORAL, JPCA, and SA are within the top 5 methods in terms of average ranking. This highlights that regularized adaptation methods are robust across datasets and modalities, offering effective alignment with minimal risk of negative DA, making them reliable and safe default options.

2. Missing recent deep DA methods:

We invite the reviewer to refer to the general comment regarding the missing deep DA methods.

3. Diverse metrics:

We thank the reviewer for their suggestion. We fully agree with this statement and this is why in the current code design, we already log various metrics. This was not visible in the submitted manuscript, therefore, we have added a table similar to Table 2 but constructed with the F1-score in the appendix in Table 22. We find that the conclusions remain the same as the ones with accuracy score showing the robustness of our benchmark.

We thank the reviewer for providing these extra references and we will consider incorporating them in our benchmark in the future.

During the rebuttal, we added SPA [1], a more recent method (2023), to our benchmark to show that our methodology allows evaluating advances in UDA. Note that adding the proposed methods highly tailored for CV tasks to our multimodal benchmark is not straightforward. It requires significant efforts and methodological adaptation to make sure they are fairly evaluated.

[1] Xiao et al. SPA: A Graph Spectral Alignment Perspective for Domain Adaptation. NeurIPS 2023

Dear Reviewers,

If you have not already done so, please engage in discussion with the authors to clarify any remaining issues as the discussion period is coming to a close in less than a day (2nd Dec AoE for reviewer responses).

Thanks for your service to ICLR 2025.

Best, AC