Improving Distribution Matching via Score-Based Priors and Structural Regularization

We sincerely appreciate the reviewer's detailed feedback and thoughtful insights. Your comments have been invaluable in helping us refine and improve the clarity of our manuscript. Below, we respectfully address the key concerns and provide clarifications regarding the benefits and rationale behind our proposed methods.

Reviewer Concerns and Clarifications

1. The the importance of flexible prior model in the settings of distribution matching.

   • Why It's Beneficial: Under the settings of AUB [1]/VAUB, a more flexible prior provides the opportunity to achieve a tighter bound, leading to a more accurate approximation of the Jensen-Shannon divergence between embedded distributions. Additionally, in the context of structural-preserving loss, a flexible prior contributes to better feature separation, particularly in scenarios with a limited number of data samples.
   • Concrete Evidence: We kindly refer the reviewer to the updated Section 5.1, where we have included a detailed comparison between different prior distributions to illustrate these benefits.

2. Gromov-Wasserstein (GW) Distance for Structure-Preservation

   • Why GW Distance?: The Gromov-Wasserstein (GW) distance is especially well-suited for preserving geometric relationships because it operates on relative distances between points rather than their absolute positions. This property ensures invariance to transformations such as rotations and translations, which hierarchical generative models often find challenging unless explicitly regularized.
   • Why Not Hierarchical Models?: While hierarchical generative models offer significant flexibility, they generally lack guarantees for preserving pairwise or cluster-based relationships in latent space. By contrast, GW constraints directly enforce such preservation, making them a powerful tool for tasks requiring structural consistency.

3. Some claims in the note

   • (Lines 33-35): "Simply collecting more data or building bigger models..."

     • Clarification: We agree that while scaling laws demonstrate significant improvements in performance with larger models and datasets, they do not inherently address critical issues such as fairness, robustness, or causality. For instance, even a perfectly trained predictive model can result in discriminatory decisions in real-world applications, unfairly impacting individuals or groups. Such issues may arise from historical biases in the training data, imbalanced base rates, or other systemic factors. To address this, we will revise our statement to acknowledge the advantages of scaling while also emphasizing scenarios where additional constraints, such as mitigating biases in training data, are essential.

   • (Line 43): "One of the most used distribution matching methods is adversarial models..."

     • Clarification: We acknowledge that this claim may oversimplify the diversity of distribution matching (DM) methods. Our intention is to focus on specific settings of distribution matching where both distributions are represented solely by samples or data. For example, this includes tasks such as matching distributions between sketches of cats and their corresponding photos or personalized data distributions between males and females. We will revise this statement to provide a more precise definition of the distribution matching tasks considered in our work.

• (Line 65): "Domain adaptation tasks require not only distribution alignment..."
  - Clarification: Perfect alignment in theory only guarantees the latent space is invariant between domain information, i.e. a trivial solution to distribution alignment tasks is to have both distributions transformed into a normal Gaussian. Therefore, in order to preserve information other than domain information during transformation into the latent space, distribution alignment goals are not enough.

  • (Line 75): "Unlike Gaussian priors, score-based models do not bias..."

    • Clarification: While score-based models reduce prior biases, we acknowledge the risk of overfitting. This will be discussed in the limitations section, and additional regularization methods will be explored in the future.

  • (Line 85): "Our framework integrates semantic information from pretrained models..."

    • Evidence: We acknowledge the lack of evidence here. The intuition here follows from prior works typically uses Euclidean distances to calculate the GW-inspired structural loss. Such Euclidean distances is computed pixel-wise between images samples, which implies that the only if the pixel-wise distance between two images has high correlations between semantic difference between images that the model want to extract and classifies on, such structural loss could be useful. However, in the case of digital images, pixel-wise distance often does not naturally correlate with the semantic meaning of an image. Additionally, for high-dimensional image data, the curse of dimensionality diminishes the effectiveness of Euclidean distance, making it less suitable as a metric for capturing meaningful relationships. we need to use a different distance metric other than Euclidean distance. CLIP loss is a valid candidate for this distance metric considering that it create a text-aligned embeddings on images that shows higher correlations to semantic meaning of the images. We will add this discussions, as well as, an extra qualitative graph in the methodology section to support our idea for CLIP losses. Please see the revised section 3.3.

  • (Lines 11-12): "Distribution matching can be applied to multiple tasks including fair classification..."

    • Fair Classification vs. Representation Learning: Fair classification is a task which focuses on directly achieving equitable outcomes, while fair representation learning aims to create representations that inherently mitigate bias. Therefore, we try to use distribution matching in the framework of fair representation learning method to tackle on the fair classification tasks in our experiment which is fair classification tasks in the Adult dataset.

Overall, we sincerely thank the reviewer for their thoughtful feedback and insightful questions. Your comments have been instrumental in guiding us to significantly improve our manuscript. We deeply appreciate the opportunity to address these points and refine both the clarity and contributions of our work.

None author response found. I will keep my score.

We sincerely appreciate the reviewer’s thoughtful feedback and are glad to hear that they found our paper interesting.

Weaknesses

1. Although using score-based priors is interesting, however, score-based priors are inherently more computationally intensive...

The reviewer raises a very valid point that our method is inherently more computationally expensive than using a simple well-defined prior. However, we would like to emphasize that our method does provide significant computational savings, particularly by eliminating the need to backpropagate through the UNET when updating the encoder and decoder, which is required in the LSGM setting.

Unfortunately, due to time constraints and the need to focus on training and retraining other experiments, we were unable to include a detailed analysis of training time in this submission. However, we plan to include this important analysis in our camera-ready version, should our paper be accepted.

2. From Equation (13) to Equation (14), it seems like noisy versions of the latent samples are introduced to develop a...

We thank the reviewer for their valuable feedback. We would like to clarify that the reviewer is correct in noting that variational inference from a Gaussian distribution is typically needed for sampling images. While this is indeed a plausible extension for our model and could be an interesting avenue for future work, it is not necessary for tasks such as domain adaptation, domain translation, or fair classification in our current setup.

For domain adaptation or fair classification, the score model is not required for inference. Specifically, we only need the encoders, as we train a classifier directly on the encoded latent representations, which does not involve the score-based model. Similarly, for domain translation, the score-based model is not required. Instead, we use the encoders and decoders: we obtain an image from domain 1 by encoding it with the domain 1 encoder and decoding it with the domain 2 decoder.

As the reviewer correctly points out, if we were to generate new samples for domain translation, we would indeed need to perform inference from a Gaussian distribution, which would involve the score-based model.

Questions

1. Could the authors elaborate more on why other optimal transport methods... We sincerely appreciate the reviewer’s insightful question. To elaborate further, traditional optimal transport methods, such as the Wasserstein distance, are typically designed to compare distributions between spaces of the same dimensionality because they rely on directly comparing points in one space with corresponding points in another space. These methods work by calculating pairwise distances between points from each space and then finding an optimal transport plan that minimizes the total cost of moving mass between them. When the spaces have different dimensions, a direct point-to-point comparison becomes problematic because there is no natural correspondence between points in the two spaces. Additionally, computing pairwise distances requires points to be comparable in a geometric sense, which is not possible when the dimensionalities differ [2].

In contrast, the approach we propose with the Gromov-Wasserstein (GW) metric allows for the comparison of spaces with different dimensions by focusing on the preservation of relative distances between points, rather than requiring direct correspondences [1]. This makes the GW metric more flexible, as it can compare structures in spaces of differing dimensionalities by capturing the geometric relationships between points rather than their absolute positions. By comparing the relative distances between points in each space, GW can still measure structural similarity despite differences in dimensionality, making it suitable for more general applications across diverse datasets.

We hope this explanation clarifies why traditional optimal transport methods are constrained to spaces of the same dimensionality and how our method overcomes this limitation.

2. Equation (14) shows the final derivation of the proposed method, which, to me, appears similar to denoising score matching...

This is a very insightful question from the reviewer, and we are excited to clarify that our SFS trick is actually a distilled version of the LSGM objective. We demonstrate this by showing that applying the Sticking-the-Landing principle [3] to the LSGM objective results in the SFS trick. Essentially, this means that our model should be more stable during optimization, as it eliminates the Jacobian term of the score model. This allows the score model to be bypassed in backpropagation when updating the encoder and decoder, contributing to both optimization stability and significant computational memory savings.

Additional details can be found in the updated Section 3.2, and the full proof is provided in Appendix E.1.

[1] Zhang, Z., Goldfeld, Z., Mroueh, Y., and Sriperumbudur, B. K. (2023). Gromov-Wasserstein Distances: Entropic Regularization, Duality, and Sample Complexity. arXiv preprint, arXiv:2212.12848. Retrieved from https://arxiv.org/abs/2212.12848.

[2] McCann, R. J., and Pass, B. (2019). Optimal Transportation Between Unequal Dimensions. arXiv preprint, arXiv:1805.11187. Retrieved from https://arxiv.org/abs/1805.11187.

[3] Roeder, G., Wu, Y., and Duvenaud, D. (2017). Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference. arXiv preprint, arXiv:1703.09194. Retrieved from https://arxiv.org/abs/1703.09194.

[4] Vahdat, A., Kreis, K., and Kautz, J. (2021). Score-based Generative Modeling in Latent Space. arXiv preprint, arXiv:2106.05931. Retrieved from https://arxiv.org/abs/2106.05931.

W7. If I understand correctly, the usefulness of the GW regularizer is demonstrated only for the domain adaptation task...

To address the reviewer’s feedback, we have rerun all experiments under three configurations: without GW, with GW in the pixel/Euclidean space, and with GW in the semantic space, for all tasks where the semantic space is relevant (excluding the fairness experiment). For detailed results and discussion, please refer to Section 5. We demonstrate that as higher-dimensional image datasets are used, the less useful and even detrimental GW in the pixel space can become (please refer to Section 5.4 and Appendix F

W8. Given that the SFS modification approximates the objective in LSGM [1]...

Please refer to the Section 3.2 of our updated paper where we demonstrate the robustness of our method compared to LSGM.

[1] Roeder, G., Wu, Y., and Duvenaud, D. (2017). Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference. arXiv preprint, arXiv:1703.09194. Retrieved from https://arxiv.org/abs/1703.09194.

[2] Vahdat, A., Kreis, K., and Kautz, J. (2021). Score-based Generative Modeling in Latent Space. arXiv preprint, arXiv:2106.05931. Retrieved from https://arxiv.org/abs/2106.05931.

[3] Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. (2022). DreamFusion: Text-to-3D using 2D Diffusion. arXiv preprint, arXiv:2209.14988. Retrieved from https://arxiv.org/abs/2209.14988.

[4] Uscidda, T., Eyring, L., Roth, K., Theis, F., Akata, Z., and Cuturi, M. (2024). Disentangled Representation Learning with the Gromov-Monge Gap. arXiv preprint, arXiv:2407.07829. Retrieved from https://arxiv.org/abs/2407.07829.

[5] Nakagawa, N., Togo, R., Ogawa, T., and Haseyama, M. (2023). Gromov-Wasserstein Autoencoders. arXiv preprint, arXiv:2209.07007. Retrieved from https://arxiv.org/abs/2209.07007.

[6] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. arXiv preprint, arXiv:2103.00020. Retrieved from https://arxiv.org/abs/2103.00020.

[7] Kladny, K.-R., von Kügelgen, J., Schölkopf, B., and Muehlebach, M. (2024). Deep Backtracking Counterfactuals for Causally Compliant Explanations. arXiv preprint, arXiv:2310.07665. Retrieved from https://arxiv.org/abs/2310.07665.

[8] Pan, Y., and Bareinboim, E. (2024). Counterfactual Image Editing. arXiv preprint, arXiv:2403.09683. Retrieved from https://arxiv.org/abs/2403.09683.

[9] Komanduri, A., Zhao, C., Chen, F., and Wu, X. (2024). Causal Diffusion Autoencoders: Toward Counterfactual Generation via Diffusion Probabilistic Models. arXiv preprint, arXiv:2404.17735. Retrieved from https://arxiv.org/abs/2404.17735.

[10] Sauer, A., and Geiger, A. (2021). Counterfactual Generative Networks. arXiv preprint, arXiv:2101.06046. Retrieved from https://arxiv.org/abs/2101.06046.

W4. The experimental setups lack important details, making it difficult to understand how the proposed method is exactly applied to various tasks. For example, what are the model inputs and targets for the source and target domains across all tasks?

We sincerely apologize for the lack of clarity in our description of the experimental setup, and we appreciate the reviewer for bringing this to our attention. To address this, we have thoroughly revised the manuscript to provide a detailed explanation of how the proposed method is applied to various tasks. Specifically, we have clarified the model inputs and targets for the source and target domains across all tasks to ensure transparency and reproducibility. We would also like to mention that our method does not explicitly require the source or target distribution but we use this label correspond to other prior works. The source and target domain distribution can be switched as we do not use the source label but rather the GW-SP loss for both the source and target domain.

W5. I do not think that domain adaptation and translation between USPS and MNIST...

We thank the reviewer for raising an important point. We agree that the MNIST to USPS experiment may be too simplistic to effectively demonstrate the advantages of using CLIP embeddings and domain adaptation. Unfortunately, we did not have enough time to run experiments such as MNIST-to-MNIST-M or SVHN due to running other experiments for the rebuttal. We apologize and would like to add these in the camera-ready submission if our paper is accepted. We would, however, like to say that we have ran other experiments that show as the reviewer has mentioned shows that directly applying simple datasets such as MNIST and USPS, there is not much improvement in performance when running on the CLIP embedding or the pixel spacw, bur is significantly better than not using it at all. Please refer to Appendix F that shows separation distance between inter class and different class on MNIST and show that both GW on the pixel and semantic space can be useful.

We would also like to respectfully refer the reviewer to Appendix C, where we demonstrate that applying the SP loss to CLIP embeddings results in greater separation between class representations while promoting better alignment compared to SAUB without the GW-SP loss on CLIP embeddings. This is illustrated in Figure 6, where the UMAP visualizations show a more distinct separation between classes when SP loss is applied. These visualizations highlight the clear benefit of incorporating the GW-SP loss into the framework.

In addition to the improved separation, as reflected in the results presented in Table 1, we observe that this strong separation allows us to tackle interesting experimental settings. For instance, even with limited source-labeled dataset (0.2%), our approach still achieves accurate predictions for the target domain classes that closely match performance achieved when having access to the full source-labeled dataset. This is discussed further in Appendix D.

W6. I can hardly agree that the domain translation results are informative...

We thank the reviewer for their thoughtful feedback. We acknowledge that the FairFace dataset used for gender translation may not provide visually striking examples to effectively demonstrate the success of our translation method. To address this concern, we have conducted additional experiments using the CelebA dataset, specifically focusing on domain translations between "black hair females to blonde females," which offer clearer visual distinctions. These results can be seen in Section 5.4 and more random samples can be seen in Appendix L.

Additionally, we would like to draw the reviewer’s attention to Appendix G, where we present multi-domain translation experiments on unpaired, rotated MNIST data (angles: 0°, 30°, and 60°). While MNIST is a relatively simple dataset, translating between rotations while preserving stylistic and semantic information is a non-trivial challenge, as explored in prior works on causal-based image generation models [7, 8, 9, 10]. We believe our results in this setting are compelling, as they demonstrate the ability to successfully translate rotations while retaining nearly all semantic information.

W1. It would be beneficial to provide more clarification and motivation regarding the VAUB formulation and the associated challenges...

To address the reviewer’s feedback, we have made substantial improvements to our storyline to better highlight the novelty of our approach. We kindly refer the reviewer to the Introduction Section and Methodology Section.

W2. The Score Function Substitution (SFS) appears very similar, if not identical, to score distillation sampling (SDS) ...

We sincerely thank the reviewer for their insightful and valuable suggestion regarding the connection between the Score Function Substitution (SFS) trick and the Score Distillation Sampling (SDS) loss as applied to Latent Score-Based Generative Models (LSGM). We apologize for not including a more detailed comparison in the original manuscript and deeply appreciate the opportunity to address this in our revised version. We kindly direct your attention to Section 3.2, as well as the detailed explanation and proof provided in Appendix E. Through this exploration, we have made the important discovery that our SFS trick can be interpreted as a distilled version of the LSGM loss, aligning with the insightful intuition you suggested.

We would also like to highlight that, unlike SDS loss [3], where the application of the Sticking-the-Landing principle [1] is relatively straightforward due to the cancellation of gradient summations within the same expectation, our derivation introduces additional complexity. Specifically, our approach addresses a multiplicative gradient, which we believe represents a non-trivial contribution with the potential to inspire future research in latent score-based and diffusion models. This added complexity enhances the value of the SFS trick, making it a more robust and impactful objective function.

Furthermore, we have empirically demonstrated that our method exhibits greater stability compared to LSGM through stability analysis experiments. We kindly refer you to Section 3.2.1 for a detailed discussion and results. We sincerely hope this explanation clarifies the distinction and further reinforces the value of our proposed approach.

W3. The use of the GW metric for VAE does not seem novel either [9,10]...

We sincerely thank the reviewer for their insightful comment. In response, we have revised the introduction and methodology sections to address the Gromov-Wasserstein (GW) metrics first introduced in GWAE [5] and to clarify that our approach incorporates this regularization into the semantic space (CLIP embedding [6]).

While recent works, such as [4], have successfully utilized GW-based metrics for preserving geometry in generative models, a key limitation of these approaches is the tradeoff they impose between preserving geometry and aligning to a simple prior or employing expressive learnable priors, which can be memory-intensive or unstable. Our approach addresses this tradeoff by combining GW regularization in the semantic space with a stable, memory-efficient flexible prior, providing a significant advantage. This combination enables robust geometry preservation while aligning effectively with the prior. Moreover, by leveraging the semantic space for GW regularization, our method introduces a more meaningful and structured alignment, overcoming the limitations of prior works that operate in the pixel space, which can become less effective at higher dimensions. This integration highlights the novelty and practical contributions of our approach.

We sincerely appreciate the reviewers’ detailed and thoughtful feedback, which has been invaluable in improving the quality and clarity of our work. Below, we respectfully address the key concerns raised, highlighting the changes we have made to the manuscript in response to your suggestions.

Reviewer Concerns and Clarifications

1. Missing Discussion & Contextualization of Related Work

• We deeply apologize for the oversight in not properly citing and contextualizing the relevant prior work. We are extremely grateful that you brought this to our attention and have updated the manuscript to address your concerns:
  • Clarification of Our Contribution
    We have carefully refined our motivation to emphasize that our work extends the Gromov-Wasserstein framework by incorporating a semantic metric space when computing the GW cost. In contrast, prior works have primarily utilized the Euclidean space. This distinction helps highlight the novelty and importance of our approach.
  • Empirical Evidence
    In Experiments 5.3 and 5.4, we now provide empirical evidence demonstrating that using the semantic metric space leads to improved performance compared to prior works relying on the Euclidean space. These results validate the effectiveness of our proposed method and clarify its advantages.
  • Additional Intuition
    To further aid understanding, we have added additional intuitive explanations in Appendix F to illustrate why the semantic space offers advantages over the Euclidean space in this context. We hope this addition addresses your concerns and enhances the clarity of our work.

2. Experimental Section

• Clarification of the Structure-Preserving Loss Usage:
  We sincerely apologize for the lack of clarity in Sections 5.1 and 5.2 regarding the use of the structure-preserving loss. To resolve this, we have explicitly updated the experimental descriptions to indicate whether and which GW loss was employed in each experiment. We hope this revision provides the clarity needed to better understand the experimental setups.

• Comparison to Competing Methods with Flexible Priors:
  Thank you for the insightful suggestion to compare our score-based prior with other trainable priors. In response, we have included comparisons with methods leveraging flexible priors, such as Mixture of Gaussians (MoG) and VampPrior, in Section 5.1. Additionally, we now include comparisons with Latent Score-Based Generative Models (LSGM), demonstrating that our score-based models, combined with the SFS techniques, generally outperform these methods. We hope these results provide a more comprehensive understanding of the performance and advantages of our proposed approach.

• Quantitative Results for Section 5.4:
  We greatly appreciate your suggestion to include quantitative results for Section 5.4. In response, we have revisited the domain translation experiments on the CelebA dataset and added quantitative results to complement the qualitative findings. Specifically, we now report metrics quantifying the degree of semantic preservation during hair attribute translation. These results validate our claims with robust quantitative evidence and further demonstrate the effectiveness of our method.

We are deeply grateful for the reviewers’ feedback, which has been instrumental in guiding us toward meaningful improvements. Your suggestions have significantly strengthened the clarity, rigor, and completeness of our manuscript. We respectfully request that you review our updated work, as we believe it now better addresses your concerns and more effectively communicates our contributions.

Thank you once again for your time, effort, and invaluable insights. We truly appreciate your dedication to helping us refine and improve our work.

[1] Vahdat, A., Kreis, K., and Kautz, J. (2021). Score-based Generative Modeling in Latent Space. arXiv preprint, arXiv:2106.05931. Retrieved from https://arxiv.org/abs/2106.05931.

[2] Tomczak, J. M., and Welling, M. (2018). VAE with a VampPrior. arXiv preprint, arXiv:1705.07120. Retrieved from https://arxiv.org/abs/1705.07120.

We sincerely thank the reviewer for their thoughtful and thorough evaluation of our work, as well as their valuable feedback on the revised manuscript. We deeply appreciate the time and effort the reviewer has dedicated to carefully assessing our revisions and providing constructive suggestions for further improvement.

If the reviewer feels that the revisions we have made have resulted in meaningful improvements to the manuscript, we kindly ask if they would consider reflecting this in their assessment score. However, we fully understand if the reviewer believes this is unwarranted at this stage. Regardless, we are truly grateful for the reviewer’s detailed insights.

We appreciate the reviewer's detailed feedback and believe that the points below address the key concerns.

Reviewer Concerns and Clarifications

1. Clarification of the contribution. We acknowledge that the introduction might appear to lack focus. We start with the broad applicability of distribution matching (DM) methods, including fairness, robustness, causality, and explainability. We then proposed a new DM method inspired by previous DM works, and compared the performances between previous DM methods in the scope narrowed down to domain adaptation, translation, and fairness in the experiment section. To improve clarity, we streamlined the introduction in the revised version to state the goal of the experiment section more clearly. Please be so kind to reread the introduction section and methodolgy section of are new manuscript.

2. Response to Q1:

   We use the term "flexibility" to indicate the ability of the prior model to learn more complex distributions. We first mention that prior methods lack flexibility in their prior distribution or have expressive priors that are computational expensive and then propose to create a more flexible prior by using the score-based prior. We then define such a flexible model in section 3.2 and provide specific synthetic dataset experiments to show the necessity of a flexible prior model in section 5.1. We also make a connection to the term 'structure preserving' (SP) with the Gromov-Wasserstein (GW) distance in section 3.1. We explicitly show the benefits of introducing SP in the domain adaptation tasks in Table 1 as well. However, the original submission lacked SP comparisons in the experiments on translation. To give stronger evidence for SP, we explicitly added SP comparisons in the domain translation and domain adaptation tasks in the revised version.

3. Response to Q2:

   We thank the reviewer for their insightful question. Prior works have shown that preserving the geometric properties of data is crucial for learning meaningful and robust representations [3, 4, 5]. Recent studies [1, 2] have demonstrated that enforcing geometry preservation constraints can induce disentanglement in the latent space. We believe these traits are particularly important for domain adaptation, as previous research has shown that disentangled representations lead to better performance in domain adaptation and translation tasks [6, 7, 8, 9]. However, past approaches face significant practical challenges. The simultaneous goals of preserving data geometry while matching a simple prior often lead to distortions within the latent space. To address this, [2] advocates for the use of more expressive priors, such as meta-priors, Gaussian mixtures, and neural priors, which provide greater flexibility in capturing complex data distributions while preserving geometric consistency. Despite their advantages, these learnable priors often have practical limitations, including poor scalability to high-dimensional spaces, significant computational expense, and instability during training. Our method addresses these challenges by learning an expressive prior through a score-based model, leveraging the expressivity of diffusion models while ensuring stable performance. This stability is achieved by avoiding the computation of the UNET Jacobian term during updates to the encoder and decoder, significantly improving both scalability and training stability. We have also updated our paper to reflect this additional information.

4. Response to Q3:

   We appreciate the opportunity to clarify how we define and compute our semantic-preserving loss. We define our semantic-preserving loss as being calculated between the data dimension and the embedding dimension. For instance, in a domain adaptation setup with two domains—source and target—there will be one structural loss for each domain, namely $L_{SP}^{src}$ and $L_{SP}^{tgt}$. Specifically, $L_{SP}^{src}$ is computed between $x{src}$ and $z_{src} \sim q_{\theta}(z|x_{src})$, and similarly for $L_{SP}^{tgt}$. On the other hand, Papers 1–5 that you kindly reminded us on, focus on formulating the loss in the embedding space between domains. Using the same domain adaptation example, there would be only one loss defined, $L_{OT/GW} = OT/GW(z_{src}, z_{tgt})$. Therefore, it is not meaningful to make a direct comparison between our structural loss and the losses proposed in these papers, as they are formulated differently and operate at distinct levels of abstraction.

5. Response to Q4:

   In response to the reviewer’s suggestion, and after careful consideration, we agree with the proposed change. As such, we have decided to replace our FairFace dataset experiment with a CelebA benchmark. Specifically, we now focus on domain translation tasks between females with black hair and females with blonde hair. These results can be observed in Section 5.4.

6. Response to Q5:

   We apologize for not providing sufficient motivation and clarity regarding the use of Gromov-Wasserstein (GW) regularization. To address this, we emphasize that studies [1, 2] have shown that enforcing geometry preservation constraints, such as GW distance, induces disentanglement in the latent space, which is critical for effective domain adaptation. In our revised Section 3.3, we further clarify that computing GW in a semantic space, rather than solely in Euclidean space as proposed by [1, 2], leads to more meaningful latent representations, as intuitively shown in Appendix F. Additionally, in the updated Section 5.3, we provide an empirical comparison of three configurations: no GW constraint, GW in Euclidean space (GW-EP), and GW in semantic space (GW-SP). These results demonstrate that GW-SP consistently outperforms the alternatives, reinforcing its effectiveness.

[1] Uscidda, T., Eyring, L., Roth, K., Theis, F., Akata, Z., and Cuturi, M. (2024). Disentangled Representation Learning with the Gromov-Monge Gap. arXiv preprint, arXiv:2407.07829. Retrieved from https://arxiv.org/abs/2407.07829.

[2] Nakagawa, N., Togo, R., Ogawa, T., and Haseyama, M. (2023). Gromov-Wasserstein Autoencoders. arXiv preprint, arXiv:2209.07007. Retrieved from https://arxiv.org/abs/2209.07007.

[3] Chen, N., Klushyn, A., Ferroni, F., Bayer, J., and van der Smagt, P. (2020). Learning Flat Latent Manifolds with VAEs. arXiv preprint, arXiv:2002.04881. Retrieved from https://arxiv.org/abs/2002.04881.

[4] Uscidda, T., Eyring, L., Roth, K., Theis, F., Akata, Z., and Cuturi, M. (2024). Disentangled Representation Learning with the Gromov-Monge Gap. arXiv preprint, arXiv:2407.07829. Retrieved from https://arxiv.org/abs/2407.07829.

[5] Hahm, J., Lee, J., Kim, S., and Lee, J. (2024). Isometric Representation Learning for Disentangled Latent Space of Diffusion Models. arXiv preprint, arXiv:2407.11451. Retrieved from https://arxiv.org/abs/2407.11451.

[6] Cai, R., Li, Z., Wei, P., Qiao, J., Zhang, K., and Hao, Z. (2019). Learning Disentangled Semantic Representation for Domain Adaptation. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-2019), 2060–2066. Retrieved from http://dx.doi.org/10.24963/ijcai.2019/285.

[7] Lee, S., Cho, S., and Im, S. (2021). DRANet: Disentangling Representation and Adaptation Networks for Unsupervised Cross-Domain Adaptation. arXiv preprint, arXiv:2103.13447. Retrieved from https://arxiv.org/abs/2103.13447.

[8] Liu, Y.-C., Yeh, Y.-Y., Fu, T.-C., Wang, S.-D., Chiu, W.-C., and Wang, Y.-C. F. (2018). Detach and Adapt: Learning Cross-Domain Disentangled Deep Representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.

[9] Bousmalis, K., Trigeorgis, G., Silberman, N., Krishnan, D., and Erhan, D. (2016). Domain Separation Networks. CoRR, arXiv:1608.06019. Retrieved from http://arxiv.org/abs/1608.06019.