Diffusion Transportation Cost for Domain Adaptation

This is comment 2 out of 2.

4. Comparison to additional methods and datasets:

   In this paper, we specifically focused on the use of OT for unsupervised DA tasks, and as such, our comparison includes only DA methods. However, since there are only a few papers proposing new transport costs, we briefly discussed methods that are outside the DA context in the related work section.

   The method presented in [a] is a semi-supervised approach that assumes the availability of a set of source-target pairs, referred to as key-points. While this method operates under different settings and is therefore not directly comparable, it introduces an interesting approach for computing the transport cost. We have included it in the revised version (see paragraph 3 in the related work section and paragraph 3 in Appendix A). In fact, the idea shares similarities with the approach by Duque et al. (2023), which is included in the related work section and briefly explained in Appendix A.

   In [b], the method is not a DA method, nor does it propose a new cost function for the OT problem, making it outside the scope of this paper.

   In [c], the authors propose a neural network-based framework for learning task-specific cost functionals. However, the two cost functions presented are semi-supervised or supervised, which makes the setting different from ours. Thus, the method will not be included in the comparison tables, but we have added it to the related work section.

   We appreciate bringing these interesting works to our attention, which have helped enrich our literature review.

   As per your suggestion, we plan to add an additional dataset, such as DomainNet, to the experiments section. While obtaining results on a new dataset within the discussion period may not be feasible, we will make every effort to include them in the camera-ready version.

This is comment 1 out of 2.

We thank the reviewer for their insightful questions and for taking the time to carefully review our paper.

1. Key contributions of Diffusion-OT:

   The key difference between Diffusion-OT and existing methods is that Diffusion-OT modifies the transportation cost itself, whereas methods like JDOT, POT, and UOT focus on proposing new problem formulations. While a few methods, such as ETD and RWOT, also introduce new transport costs, Diffusion-OT captures both cross-domain and intra-domain relationships through a label-guided diffusion process, which is composed of the three stochastic matrices you pointed out. Additionally, in contrast to competing methods, Diffusion-OT is a data-driven approach, meaning its cost can be easily integrated into most OT-based methods, including JDOT, POT, and UOT, as demonstrated in the experiments section. This approach allows OT-based methods to account for data geometries, effectively leverage source label information, and select the problem formulation that best suits the specific DA task, according to the properties you mentioned.

2. The validity of OT using Diffusion-OT:

   Your observation is correct - the Diffusion-OT cost is not a metric. However, in OT, the cost function does not necessarily need to be a metric for the problem to be valid. While the Wasserstein distance requires the cost function to be a metric in order to define a valid distance, OT as a broader framework encompasses more than just the Wasserstein distance. In our case, the proposed cost is not a metric, and the solution to the OT problem does not represent a distance, nor do we require it to be, as we only use the resulting transportation plan for all the DA tasks presented.

3. The optimization process:

   Prop. 1 suggests that the diffusion process accounts for both cross-domain discrepancies and intra-domain geometries. The OT problem seeks to find a transportation plan that minimizes the given cost. In this context, the goal is to minimize domain discrepancies while preserving intra-domain structure. When source labels are available, the diffusion process can be further guided in a task-specific manner.

   In response to your question, we have included an additional illustration in Figure 4 of the supplementary materials, which highlights how the resulting transportation plan is directly influenced by the selected cost. For further details, please refer to Appendix D.1 in the revised version.

   Regarding your fourth question, in the main formulation, we apply the operator to the original space $\mathcal{X}$. However, in the deep experiments, the kernels are computed using the learned features, i.e., in the representation space $\mathcal{Z}$. In this case, the representations are learned using a deep model, and indeed, the model parameters are variables in the optimization process, which follows an alternating optimization framework. In short, we first train a deep model to learn the latent space $\mathcal{Z}$, then fix the model, using the learned features to compute the Diffusion-OT cost and obtain the transportation plan. Subsequently, we incorporate the plan into the loss function and proceed with the next gradient descent step on the deep model, continuing the learning process for the latent space. For further details, please refer to Appendix D.2, which provides an in-depth description of the DeepJDOT framework.

This is comment 1 out of 2.

We thank the reviewer for the thorough review and the valuable feedback.

1. Motivation and advantages of Diffusion-OT:

   Thank you for raising this concern. The statement aims to highlight that while OT is widely explored in DA, most existing works focus on the model architecture or problem formulation, leaving the choice of transport cost relatively overlooked despite its potential for improvement. For instance, the commonly used squared Euclidean distance assumes that samples lie in a Euclidean space, which is often not the case in practice. While frameworks like DeepJDOT address this by learning a latent space, we believe this approach is not always sufficient. Our proposed cost tackles this by employing the manifold assumption, using the Euclidean distance only locally. Furthermore, our method considers intra-domain geometries and integrates source label information directly into the cost. The advantages of our approach, outlined in the final paragraph of the introduction in detail, are supported by extensive experimental results. We briefly discussed the limitations of competing methods like ETD and RWOT, which also propose new transport costs, in the related work section. As for other OT-based methods, we do not necessarily view them as competing approaches, as most of these methods can be applied using our proposed cost.

2. Performance in the MNIST-USPS experiment:

   Since the code for RWOT is not available, we are unable to conduct a detailed analysis. However, we observe that all baseline methods implemented using the DeepJDOT framework (DeepJDOT, JUMBOT, m-POT) achieved relatively low accuracy in this scenario, indicating that the DeepJDOT architecture may not be optimal for the MNIST-USPS experiment compared to the architecture used in RWOT. Additionally, we note that the proposed method outperformed all baselines in this scenario as well.

3. The hyperparameter $\epsilon$:

   There is no single optimal methodology for selecting the Gaussian kernel parameter. However, a common practice is to set $\epsilon$ as the median of the pairwise distances in the data. This approach was utilized in all our deep experiments.

This is comment 2 out of 2.

4. Comparison to state-of-the-art methods: In our comparison, we included competing approaches that introduce new transportation costs for the OT problem and focus on the DA task — specifically, ETD and RWOT. Both methods define the cost as a weighted Euclidean distance, with the weights learned using different strategies. For our method, we chose a simple framework — DeepJDOT — based on the ResNet50 model. We included comparisons with baselines that use the same framework but rely on the traditional OT cost, specifically the squared Euclidean distance.

   We would like to point out that the research on DA is broad and ongoing. Since it is not feasible to compare with all the latest DA methods, we have chosen to focus exclusively on OT-based approaches. For this reason, we have opted not to include papers [2] and [5] in our comparison tables. Regarding the recent method SPA [2], while it is not OT-based, it is a graph-based approach that also considers intra-domain structures, which makes it understandable why you find it relevant for comparison. However, since SPA relies on an adversarial training framework and includes additional objectives beyond the proposed graph spectral alignment term (such as adversarial loss and pseudo-labels loss), comparing it to our cost would not provide a fair evaluation. Nonetheless, we acknowledge your valuable point. While our current results using the DeepJDOT framework demonstrate how Diffusion-OT improves upon baselines and outperforms competing methods like ETD and RWOT, we agree that including additional OT-based methods, even those that do not propose a transportation cost, in the literature review and comparison could further strengthen our evaluation.

   It is worth highlighting that our cost is derived directly from the data and can be efficiently computed, and therefore it can be integrated into most OT-based methods. The proposed method in [1], termed PPOT, is one such example; it introduces a regularization term in the optimization problem to encourage the learning of a representation space with better-separated classes. Although the method uses the squared Euclidean distance as the transport cost in the paper, it is not restricted to this choice, and our proposed cost can be easily incorporated into the framework. Unfortunately, the code for PPOT has not been published. We have reached out to the authors to request access and are awaiting their response. We hope to provide results of Diffusion-OT within the PPOT framework by the end of the discussion period.

   In [3], the squared Euclidean cost is used between clustering representations rather than between individual samples, making it incompatible with our proposed cost. The paper presents two sets of results for each dataset: one using a DANN model with ResNet50 as the backbone for Office-Home and ResNet101 for VisDA, and the other using CDTrans, a SOTA transformer-based method. When conducting a fair comparison against the ResNet-based model, Diffusion-OT achieves better results for Office-Home by an average of $0.7$% and outperforms in 8 out of the 12 scenarios. While their method achieves $81.3$% with the CDTrans framework, the original CDTrans paper reports $80.5$% for this dataset, indicating that the improvement is attributable to the architecture rather than the method itself. For VisDA, our method used ResNet50, whereas they used ResNet101, making a direct comparison unfair even with their lower result. In [4], the authors propose an attention-based architecture for UDA and highlight its strong connection to the OT problem. While we achieve significantly better results on the Office-Home dataset, the comparison is not fair due to the architectural differences and the use of different backbones for VisDA.

   We acknowledge that our choice of architecture may limit the fairness of comparisons with SOTA methods. To address this, we plan to evaluate our method with additional backbones and integrate it into a SOTA framework, such as CDTrans. While obtaining results on a new architecture within the discussion period may not be feasible, we will make every effort to include them in the camera-ready version.

We thank the reviewer for their review and the helpful feedback.

1. Missing results for competing methods:

   For ETD and RWOT, we reported the results as stated in their original papers. ETD did not provide results for the VisDA dataset. Although RWOT presented results for this dataset, the reported accuracy for DeepJDOT ($77.4$%) indicates that the ResNet model was modified, as it differs from both our reproduction ($69.85$%) and the original DeepJDOT paper ($66.9$%). Since the authors did not release the code or provide details of these modifications, we cannot determine what changes were made. Consequently, to ensure fairness, we chose not to include their results.

2. Improvement on the Digits dataset:

   The improvement on the Digits dataset is indeed small. As mentioned in the paper, this dataset is considered an easier DA task, and the feature extractor appears to learn well-separated features for the classes, even when using the standard cost. Therefore, the minor improvement observed is expected.

3. Generalization to Diffusion-OT:

   We thank you for the question and agree with your general observation. While this paper focuses on Unsupervised DA for coherence, we believe Diffusion-OT has the potential to be extended to other applications, which we consider as directions for future work. In the meantime, one of the key advantages of our proposed cost is its adaptability to a broad range of OT-based methods, including those applicable to certain Universal Domain Adaptation tasks.

We thank the reviewer for the thorough review and the valuable feedback.

1. Computational complexity:

   Thank you for raising this point. In response, we added a discussion on the method's computational complexity in Appendix C.2, and we provide it here as well. The most commonly used transport cost, the pairwise squared Euclidean distances, has a computational complexity of $\mathcal{O}(n^2)$, assuming balanced datasets. In contrast, our proposed cost involves the following steps:

   1. Computing pairwise distances for three matrices: $\mathcal{O}(n^2)$.
   2. Applying the exponential function to the matrices: $\mathcal{O}(n^2)$.
   3. Normalizing the matrices to be row-stochastic: $\mathcal{O}(n^2)$.
   4. Multiplying the three probability matrices: $\mathcal{O}(n^3)$.
   5. Applying the logarithm to the final diffusion operator: $\mathcal{O}(n^2)$.

   As noted in Appendix D, we often use doubly-stochastic normalization instead of row-stochastic normalization, employing the Sinkhorn algorithm. Since the complexity of Sinkhorn is also $\mathcal{O}(n^2)$, this does not increase the overall complexity. In total, the computational complexity of the proposed cost is $\mathcal{O}(n^3)$. While higher than the traditional cost, our method avoids the need for a regularization term, which can often add to the optimization complexity.

   Additionally, we note that up to the fourth step, the computation involves three $n \times n$ matrices rather than one, resulting in higher memory requirements.

2. Theoretical results:

   We agree that such theoretical results could strengthen the theoretical foundations. Unfortunately, we do not have these results at present. We would like to note that none of the competing methods offer theoretical results. In contrast, while our analysis focuses on the asymptotic behavior of the proposed diffusion operator, we provide a detailed analysis that motivates our approach. Furthermore, to establish confidence in the superiority of our method over traditional approaches, we conducted extensive experiments across two distinct frameworks and multiple datasets.

3. Failure cases:

   Thank you for the comment. First, we would like to emphasize that our assumption that both domains reside in the same space does not present a barrier when the underlying structures differ. When using the squared Euclidean distance as cost, whether explicitly stated or not, it is assumed that the source and target domains are embedded in $\mathbb{R}^d$. The manifold assumption extends this concept, allowing for the possibility that the samples may not lie within a Euclidean space. Notably, the manifold assumption does not compromise performance for the UDA tasks presented. However, our method is currently limited to scenarios where the source and target domains share the same feature space (specifically, due to the second diffusion step involving inter-domain distances). All competing methods have this limitation as well. We have revised the beginning of Section 4 to provide a clearer explanation of this assumption.

   In our empirical experiments on real-world data, as well as in our simulations, we did not observe any clear failure cases. However, it is important to note that our proposed cost function contains a hyperparameter, and the method's performance may vary depending on its choice.

This is comment 2 out of 2.

4. Comparison to state-of-the-art methods:

   In the experiments section, we compared our method with competing approaches that introduce new transportation costs for the OT problem and focus on the DA task — specifically, ETD and RWOT. Both methods define the cost as a weighted Euclidean distance, with the weights learned using different strategies. For our method, we chose a simple framework — DeepJDOT — based on the ResNet50 model. We included comparisons with baselines that use the same framework but rely on the traditional OT cost, specifically the squared Euclidean distance. Nonetheless, we acknowledge your valuable point. While our current results using the DeepJDOT framework demonstrate how Diffusion-OT improves upon baselines and outperforms competing methods like ETD and RWOT, we agree that using a SOTA architecture and comparing against SOTA OT-based methods, even those that do not propose a transportation cost, could further strengthen our evaluation.

   It is worth highlighting that our cost is derived directly from the data and can be efficiently computed, and therefore it can be integrated into most OT-based methods. The proposed method in [c], termed PPOT, is one such example; it introduces a regularization term in the optimization problem to encourage the learning of a representation space with better-separated classes. Although the method uses the squared Euclidean distance as the transport cost in the paper, it is not restricted to this choice, and our proposed cost can be easily incorporated into the framework. Unfortunately, the code for PPOT has not been published. We have reached out to the authors to request access and are awaiting their response. We hope to provide results of Diffusion-OT within the PPOT framework by the end of the discussion period.

   In [d], the squared Euclidean cost is used between clustering representations rather than between individual samples, making it incompatible with our proposed cost. The paper presents two sets of results for each dataset: one using a DANN model with ResNet50 as the backbone for Office-Home and ResNet101 for VisDA, and the other using CDTrans, a SOTA transformer-based method. When conducting a fair comparison against the ResNet-based model, Diffusion-OT achieves better results for Office-Home by an average of $0.7$% and outperforms in 8 out of the 12 scenarios. While their method achieves $81.3$% with the CDTrans framework, the original CDTrans paper reports $80.5$% for this dataset, indicating that the improvement is attributable to the architecture rather than the method itself. For VisDA, our method used ResNet50, whereas they used ResNet101, making a direct comparison unfair even with their lower result.

   We acknowledge that our choice of architecture may limit the fairness of comparisons with SOTA methods. To address this, we plan to evaluate our method with additional backbones and integrate it into a SOTA framework, such as CDTrans. While obtaining results on a new architecture within the discussion period may not be feasible, we will make every effort to include them in the camera-ready version.

5. The hyperparameter $\epsilon$:

   There is no single optimal methodology for selecting the Gaussian kernel parameter. However, a common practice is to set $\epsilon$ as the median of the pairwise distances in the data. This approach was utilized in all our deep experiments.

This is comment 1 out of 2.

We thank the reviewer for the thorough review and the valuable feedback.

1. Intra-domain geometry using the Gromov-Wasserstein discrepancy:

   Typically, methods based on the Gromov-Wasserstein (GW) metric incorporate intra-domain geometry but do not account for inter-domain relationships.

   In [a], the authors propose a method that learns node embeddings for both sets within the same metric space and computes the transport plan by solving an optimization problem that combines the GW discrepancy with cross-domain distances. While this method does not specifically address DA applications, we recognize its potential applicability to certain DA tasks.

   In [b], the authors present a method for Heterogeneous DA. The method includes adding a regularization term, $\Omega_l$, which contains inter-domain relationships. However, the cross-domain distances are computed only after transportation, in the target domain, and require target labels, making this a semi-supervised approach.

   We appreciate bringing these works to our attention and have included them in our paper (see paragraph 3 in the related work section, and paragraph 3 in Appendix A).

2. Advantages of pre-defined transport costs over learned strategies:

   Although learning the cost is often an effective strategy, it is not always feasible. Our approach has the distinct advantage that the cost is not learned, making the method inherently compatible with both classical and deep learning frameworks. This flexibility ensures adaptability to a wide range of OT-based methods, including various problem formulations, efficient solvers, and architectures. As demonstrated in the experiments section with the DeepJDOT framework, our method achieves superior performance in most scenarios compared to ETD and RWOT, particularly for real-world data.

3. The manifold assumption:

   The assumption that both domains are supported on the same hidden manifold is central to our method, but it does not restrict the method's applicability to different domain shifts. When using the squared Euclidean distance as the transport cost, whether explicitly stated or not, it is assumed that the source and target domains are embedded in $\mathbb{R}^d$. The manifold assumption extends this concept, allowing for the possibility that the samples may not lie within a Euclidean space. This assumption strengthens our approach to focus on the local relationships within and between domains, which we achieve using Gaussian kernels and the diffusion process. The main limitation of this assumption is that both domains are required to share the same feature space, and it is common to most DA methods. We have revised the beginning of Section 4 to provide a clearer explanation of this assumption.