Rethinking Joint Maximum Mean Discrepancy for Visual Domain Adaptation

Weaknesses 1 (Paper Writing): According to your suggestions, we will polish the revised manuscript to enhance the readability.

Weaknesses 2 (More Discussions on the Novelty of this Submission): Thanks for your recognition of our work. Following your suggestions, we will enhance the revised manuscript by including more discussion on our innovative contributions, such as the limitations of the proposed loss function and future research directions (as addressed in our responses to your raised Weaknesses 4 and Limitation).

Weaknesses 3 (More Experimental Results, e.g., Combination with More Methods): Following your suggestion, we have supplemented more experimental results: in response to the raised Weakness 1 by #Reviewer 1, we include runtime evaluations for computing the proposed loss function and optimizing its corresponding objective; to address the raised Weakness 3 by #Reviewer 1, we perform ablation studies by individually using JMMD and HSIC; in response to the raised Weakness 2 by #Reviewer 2, we conduct ablation studies by using different JMMD kernels for the proposed JMMD-HSIC. Furthermore, in Section 6 of the supplementary materials, we provide: (1) robustness experiments of the proposed method with different classifiers (Table 1 in Lines 117-118), (2) comparative experimental results on the Office10-Caltech10 dataset (Table 2 in Lines 117-118), and (3) comparative experimental results on the ImageCLEF-DA dataset (Table 3 in Lines 135-136).

Moreover, we have incorporated the proposed loss function into both a shallow subspace-learning-based domain adaptation method (CRLP-DA21 [3]) and three end-to-end deep adaptation frameworks (JAN [1], DSAN [2], and DLP-NNM [4]), as shown in the following two tables. Although the primary objective of this work is to address the challenges of applying JMMD in shallow subspace-learning methods, our final loss function demonstrates successful integration with end-to-end deep adaptation approaches. Notably, JAN is the pioneering work published in ICML'17 that first apply JMMD into end-to-end deep adaptation frameworks, but their original paper identified a critical issue. The following sentence is originated from the ICML paper: · · · while in conventional shallow domain adaptation methods the joint distributions are not easy to manipulate and match. This issue serves as a key motivation for our work. DSAN represents another significant contribution that specifically addresses how to define class conditional distribution discrepancy in deep adaptation frameworks. CRLP-DA21 and DLP-NNM offer representative solutions that mitigate the impact of pseudo-labeling issues on loss functions requiring target domain pseudo-labels through innovative classifier designs. Our experimental results show that the proposed loss function achieves varying degrees of performance improvement when applied to all four of these methods, which can demonstrate the effectiveness of our proposed loss function.

Weaknesses 4 and Limitation (Limitations of the Proposed Method and Future Work): The limitations of our proposed loss function are as follows: (a) Domain Adaptation Setting. The proposed JMMD-HSIC can only address the closed-set domain adaptation setting, where the label spaces of the source domain and target domain are identical. However, we do not explore its application in more complex domain adaptation scenarios, such as open-set domain adaptation [5], partial domain adaptation [6], weakly-supervised domain adaptation [7], etc. (b) Task. This submission only investigates the application of JMMD-HSCI in cross-domain classification tasks, while its potential in other more complex tasks, such as semantic segmentation and object detection, remains under-explored. (c) Classifier. Although this study has verified that JMMD can improve performance across different classifiers, how to design JMMD kernels for different classifiers has not been sufficiently explored. For future work, we plan to explore: (1) designing an advanced label kernel to handle more diverse domain adaptation scenarios; (2) extending the framework to more complex visual tasks; (3) designing novel label kernels by systematically analyzing how different classifiers vary in their sensitivity to feature distribution alignment and feature discriminability.

[1] Long M.S., et al. Deep Transfer Learning with Joint Adaptation Networks. ICML, 2017.

[2] Zhu Y. C., et al. Deep Subdomain Adaptation Network for Image Classification. TNNLS, 2020.

[3] Wang W. et al. Confidence Regularized Label Propagation Based Domain Adaptation. TCSVT, 2021.

[4] Wang W., et al. Deep Label Propagation with Nuclear Norm Maximization for Visual Domain Adaptation. TIP, 2025.

[5] Saito K., et al. Open Set Domain Adaptation by Backpropagation. ECCV, 2018.

[6] Cao Z. J., et al. Partial Adversarial Domain Adaptation. ECCV, 2018.

[7] Shu Y., etal. Transferable Curriculum for Weakly-Supervised Domain Adaptation. AAAI, 2019.

Weaknesses 1 (Future Work Inspired by the Theoretical Findings in this Submission): In our responses to #Reviewers 1, 2, and 4, we have thoroughly discussed three key limitations of the proposed loss function. To address these limitations in future work, we plan to: (a) designing an advanced label kernel to handle more diverse domain adaptation scenarios; (b) extending the framework to more complex visual tasks; (c) designing novel label kernels by systematically analyzing how different classifiers vary in their sensitivity to feature distribution alignment and feature discriminability.

Weaknesses 2 (Detailed Descriptions for the Motivation and Workflow of the Proposed Method): In our responses to #Reviewers 1 and 2, we clarify that the motivation of this work originates from an ICML'17 paper, where the authors argued that JMMD is difficult to apply to shallow domain adaptation methods. The following sentence is originated from the ICML paper: · · · while in conventional shallow domain adaptation methods the joint distributions are not easy to manipulate and match.

We will provide the following detailed descriptions for Figure 1 (Lines 36-37) in our revised manuscript: The overview of our revealed theoretical results and proposed JMMD-HSIC. (a) We map the features (upper part) and labels (lower part) of the source and target domains into the RKHS, respectively; (b) The application of JMMD (upper part) and HSIC (lower part) in a subspace-learning framework; (c) The graph embedding interpretation of JMMD (upper part) and HSIC (lower part) in a subspace-learning framework; (d) Learned feature representations of the source and target domains after subspace learning. '$−$' in the module of JMMD means the feature-label dependence difference, and '$+$' in the module of HSIC means separately considering feature-label dependence in the two domains. $**X**$ and $**Y**$ are data feature and label matrices. $\psi$ and $\phi$ are feature and label mappings for a RKHS. $**T**$ and $**B**$ are projection matrices for a projected RKHS. $**T**\psi(**X**)\otimes \phi (**Y**)$ is a tensor-product operator for the covariance. $\text{tr}(**B**^{\top}**K**^{**X****X**}(**K**^{**Y****Y**}\odot**M**^{\text{J}/\text{H}})**K**^{**X****X**}**B**)$ is the deduced concise JMMD/HSIC in a projected RKHS.

Weaknesses 3 (Pseudo-Labels Issue for the Proposed Loss Function): (a) As you rightly pointed out, our model employs pseudo-labeling (prediction results) and trains the model by iteratively updating both the pseudo-labels and our designed loss function. To address the inherent unreliability of pseudo-labels, this submission adopts classifiers specifically designed by two subspace-learning-based domain adaptation techniques (SPL [45], OGL2P [46]). (b) In Section 6 of the supplementary material (Lines 116-127), ablation experiments comparing JMMD, HSCI and our JMMD-HSCI are performed on Office10-Caltech10 dataset, with multiple classifiers verifying the loss's robustness. Furthermore, we compare the performance of two distinct label formats (one-hot hard labels and probabilistic soft labels). As referenced in [4], this challenge can be mitigated by designing a confidence-regularized classifier that alleviates both the over-confidence problem of hard labels and the excessive noise introduced by probabilistic soft labels. Ultimately, this approach enhances the model's robustness against pseudo-label unreliability. (c) In the following two tables, we have also applied the proposed loss function to the shallow subspace learning-based domain adaptation framework (CRLP-L21) described in [4], demonstrating performance improvements compared with CRLP-L21. (d) For future work, we plan to explore: (1) confidence-based filtering methods to discard unreliable pseudo-labels, (2) enhancing the loss function's robustness to pseudo-label issue, and (3) designing more advanced classifiers to obtain more reliable pseudo-labels.

Weaknesses 4 (the Applications of Our Proposed Loss Function in Deep Learning Frameworks): Since this study primarily addresses the challenge of applying JMMD into shallow subspace-learning-based domain adaptation frameworks, we initially do not implement the proposed loss function in end-to-end deep domain adaptation frameworks. Following your suggestion, we have now applied the proposed loss into three different deep learning frameworks (JAN [1], DSAN [2], DLP-NNM [3]), with experimental results on both the Office-31 and Office-Home datasets presented in the following two tables. It can be observed that the proposed loss function still leads to varying degrees of performance improvement across these three frameworks.

[1] Long M.S., et al. Deep Transfer Learning with Joint Adaptation Networks. ICML, 2017.

[2] Zhu Y. C., et al. Deep Subdomain Adaptation Network for Image Classification. TNNLS, 2020.

[3] Wang W., et al. Deep Label Propagation with Nuclear Norm Maximization for Visual Domain Adaptation. TIP, 2025.

[4] Wang W. et al. Confidence Regularized Label Propagation Based Domain Adaptation. TCSVT, 2021.

Weaknesses 1 (the Novelty of JMMD's Kernel Formulated by the Representer Theorem): First, The motivation of this submission mainly comes from a paper published at ICML’17 [3] (precisely the authors of the reference you mentioned [1]), and they applied JMMD into a deep neural network. They argued that JMMD is difficult to be applied into shallow domain adaptation methods. The following sentence is originated from the ICML paper: “· · · while in conventional shallow domain adaptation methods the joint distributions are not easy to manipulate and match.” Second, JDA (Joint Distribution Adaptation) proposed in the reference you mentioned [1] defines the joint distribution difference as the combination of the marginal and class conditional MMD (i.e., their summation), which differs from our definition of joint distribution difference. From the uniformity of JMMD proved in our work, JDA actually represents a special case of JMMD. Consequently, the kernel obtained by JDA through the Representer theorem corresponds to the sum of the marginal kernel and class conditional kernel, rather than the JMMD kernel. Third, their definitions of marginal and class conditional MMD do not involve complex tensor product operations, making their problem inherently less challenging than ours. Fourth, when employing the Representer theorem to derive the JMMD kernel (thereby eliminating its complicated tensor-product operation), we not only uncover its uniformity but also explain—from a graph embedding perspective—the fundamental reason why it compromises feature discriminability. This insight could lead us to propose a novel and more robust loss function, such as the proposed JMMD-HSIC.

Weaknesses 2 (the Importance of JMMD's Uniformity for UDA): First, as addressed in our reply to your raised Weakness 1, our definition of joint distribution difference differs from theirs. By proving the uniformity of JMMD, we identify their method as a special case of ours. This does not imply equivalence, since we can improve JMMD by exploring its limitations and defining more robust kernels. For instance, this submission reveals how JMMD compromises feature discriminability and accordingly designs the JMMD-HSIC kernel. Second, our approach combines not the three components but the class conditional distribution kernel and our newly designed HSIC-based kernel. The rationale for selecting the class conditional distribution kernel is twofold: (a) Marginal distribution kernel neglect label information, often leading to imprecise distribution alignment; (b) While weighted class-conditional kernel can handle class imbalance issue, our experimental datasets exhibit minimal class imbalance, making their results nearly indistinguishable from the class conditional kernel. Third, following your suggestion, we have validated these claims through ablation studies, as shown in the following two tables. Fourth, the uniformity of JMMD enables our to develop more robust loss functions (as demonstrated by our JMMD-HSIC) by identifying its limitations.

Weaknesses 3 (Details about the Sentences in Lines 227-229): First, the primary objective of graph embedding is to minimize the objective function (6) (Lines 219-220). When the weight wij between samples i and j is positive, we should reduce the feature distance (draw them closer) between them to prevent the objective function from increasing; conversely, when wij is negative, we should increase their feature distance (push them further) to decrease the objective function value. Second, through derivation, we obtain (8) (Lines 224-225) and find it maintains the same form as the objective function in (6). From the JMMD (side of (8)) and its corresponding weights (left side of (7) (Lines 223-224)), we observe that when samples belong to the same domain and same class, the weight is negative, indicating we should increase the distance or push them further between two data points from the same classes in the same domain. Conversely, when samples are from different domains but share the same class, the weight is positive, suggesting we should decrease their distance or draw them closer between two data points from the same classes in the same domain. Similarly, analyzing the HSIC (right side of (8)) and its weights (right side of (7)), we find positive weights occur when samples belong to the same domain and the same class, meaning we should decrease the distance or draw them closer between two data points from the same classes in the same domain.

Weaknesses 4 (the Connections between [2] and Ours): The connections of the two are as follows: Special Case Analysis: Their work actually reveals a special case within our JMMD framework - specifically how marginal MMD and class conditional MMD may compromise discriminability. Our formulation generalizes to more distribution metrics satisfying JMMD's concise form, including the weighted class conditional MMD and the proposed JMMD-HSCI. Different Solution Strategies: Our approach fundamentally differs in addressing discriminability degradation. We introduce the JMMD-HSIC kernel with proven its reproducibility, maintaining its validity as a special case of JMMD. Their approach lacks strong theoretical foundation as it modifies the marginal MMD and class conditional MMD empirically through observation.

Question 1 (the Definition of Subspace Learning in this Submission): In response to your raised Weakness 2, we clarify that the motivation of this work originates from an ICML'17 paper, where the authors argued that JMMD is difficult to apply to shallow domain adaptation methods. The following sentence is originated from the ICML paper: · · · while in conventional shallow domain adaptation methods the joint distributions are not easy to manipulate and match. Therefore, the subspace-learning methods referred to in this submission specifically denote shallow subspace-learning frameworks such as Principal Component Analysis (PCA) and Non-negative Matrix Factorization (NMF), etc.

Question 2 (Hard to Obtain Partial Derivative): In response to your raised Question 1, we have provided clarification regarding the subspace-learning framework mentioned in this submission. The primary objective of this work is to address the challenge of obtaining partial derivatives when applying JMMD into the shallow subspace-learning frameworks not the deep subspace-learning frameworks.

Question 3 (Classes Overlapped in Figure 2c and d (Lines 300-301)): Under the domain adaptation setting where target domain labels are unavailable during training, we follow the common practice in the field by employing pseudo-labels (model predictions) to iteratively update the pseudo-labels and our proposed loss function. However, for the visualization in this Figure, we adopted the ground-truth labels following Reference [1]. The inherent unreliability of pseudo-labels remains a critical bottleneck limiting domain adaptation performance. Consequently, it is inherently challenging to learn features that can effectively separate different classes according to their true labels. The observed inter-class overlap primarily stems from the discrepancy between pseudo-labels and ground-truth labels for these samples.

Limitation (Limitation Discussions for the Proposed Method): The limitations of our proposed loss are as follows: (a) Domain Adaptation Setting. The proposed JMMD-HSIC can only address the closed-set domain adaptation setting, where the label spaces of the source domain and target domain are identical. However, we do not explore its application in more complex domain adaptation scenarios, such as open-set domain adaptation [4], partial domain adaptation [5], weakly-supervised domain adaptation [6], etc. (b) Task. This submission only investigates the application of JMMD-HSCI in cross-domain classification tasks, while its potential in other more complex tasks, such as semantic segmentation and object detection, remains under-explored. (c) Classifier. Although this study has verified that JMMD can improve performance across different classifiers, how to design JMMD kernels for different classifiers has not been sufficiently explored.

[1] Long M.S., et al. Transfer Feature Learning with Joint Distribution Adaptation. ICCV, 2013.

[2] Chen Y. M., et al. A Graph Embedding Framework for Maximum Mean Discrepancy-based Domain Adaptation Algorithms. TIP, 2019.

[3] Long M.S., et al. Deep Transfer Learning with Joint Adaptation Networks. ICML, 2017.

[4] Saito K., et al. Open Set Domain Adaptation by Backpropagation. ECCV, 2018.

[5] Cao Z. J., et al. Partial Adversarial Domain Adaptation. ECCV, 2018.

[6] Shu Y., etal. Transferable Curriculum for Weakly-Supervised Domain Adaptation. AAAI, 2019.

Weakness 1 (Convergence Analysis for the Proposed Loss Function): (a) Differentiable. Since the proposed loss function is differentiable, the final objective function to be optimized is also convergent as long as the applied subspace-learning-based domain adaptation framework is convergent. (b) Closed-Form Solution. In this manuscript, the proposed loss function is applied into JDA [15], SPL [45], and OGL2P [46], and the resulting objective functions to be optimized all admit closed-form solutions, ensuring convergence. (c) Computational Complexity. The computational complexity of the proposed loss function is $O(\text{Cn + Cn}^\text{2})$, where $\text{C}$ is the number of classes and $\text{n}$ is the total number of samples from both the source and target domains. Since $\text{C}$ is generally much smaller than the sample size $\text{n}$, the complexity mainly depends on the number of samples. (d) Time/Speed. We analyze the time required for the algorithm to compute the proposed loss function (loss construction) and optimize the corresponding objective function (objective optimization) on two datasets and two subspace-learning-based domain adaptation frameworks (SPL and OGLP2P), with the results recorded in the following two tables.

(a) It can be observed that due to the use of MATLAB, which provides highly efficient matrix computation capabilities, the objective optimization and the loss construction exhibit highly efficient runtime performance. (b) Since the Office-Home dataset is larger in scale, it requires more time compared to the Office-31 dataset. (c) the subspace-learning-based domain adaptation framework applied in this manuscript belongs to a two-stage domain adaptation approach. That is, deep learning is first employed to extract a promising feature representation, followed by further refinement using subspace-learning methods. Since subspace-learning methods entail significantly lower computational overhead compared to deep learning training, they hold substantial research value and have consistently garnered sustained attention from researchers. (d) These results not only validate the efficiency of the two-stage domain adaptation approach but also highlight the necessity of addressing the challenges in applying JMMD within subspace-learning frameworks.

Weakness 2 (Robustness Analysis for the Proposed Loss Function): The results reported in this manuscript are not averaged over multiple runs, for the following reasons: The proposed loss function is applied into three shallow subspace-learning-based domain adaptation methods. Since no stochastic processes are involved, the results obtained from each run are deterministic, with a standard deviation of 0. Referring to the original experimental results in these three papers (JDA [15], SPL [45], OGLP [46]), none of them record or mention the standard deviation.

The loss function we designed is robust, free from stochastic processes, and ensures that the same input will always produce the same output. In Section 6 of the supplementary material (Lines 116-127) and Figure 3 of the main text (Lines 300-301), we conduct ablation studies on Office10-Caltech10 dataset comparing JMMD, HSCI, and our proposed JMMD-HSCI. Different classifiers are employed to validate the robustness of the proposed loss function. Additionally, in our response to your raised Weakness 6, we have provided a detailed discussion regarding the limitations of the proposed loss function.

Weakness 3 (Ablation Study by Individually Employing the JMMD and HSIC Components): In Section 6 of the supplementary material (Lines 116-127), ablation experiments comparing JMMD, HSCI and our JMMD-HSCI are performed on Office10-Caltech10 dataset, with multiple classifiers verifying the loss's robustness. As per your suggestions, we present the experimental results individually using JMMD and HSIC in the following two tables. (a) Considering that the classifiers employed in SPL and OGL2P frameworks require higher feature discriminability, we can observe that HSIC outperforms JMMD. (b) Since the Office-31 dataset has smaller distribution discrepancies, the performance improvement of HSIC over JMMD is more significant on the Office-31 dataset (2.6%, 2.1%) compared to the Office-Home dataset (0.3%, 0.2%). (c) Our proposed loss function achieves optimal results. Although both SPL and OGL2P methods also incorporate loss functions for feature distribution alignment and discriminative feature learning, our approach provides more precise balancing between these two objectives by fundamentally analyzing how JMMD compromises feature discriminability. In other words, we more effectively mitigate the impact of distribution alignment on feature discriminability. Consequently, our method achieves superior performance compared to SPL and OGL2P.

Weakness 4 (Projection Algorithm for Feature Visualization in Figure 2 (Lines 300-301)): In Figure 2, we employ the t-SNE algorithm for feature visualization, which has been widely adopted in domain adaptation field [45,46].

Weakness 5 (Typo in Figure 1 (c) (Lines 36-37)): Thank you for identifying this typo. We will correct it in the revised manuscript.

Limitation (Limitation Discussions for the Proposed Loss Function): The limitations of our proposed loss are as follows: (a) Domain Adaptation Setting. The proposed JMMD-HSIC can only address the closed-set domain adaptation setting, where the label spaces of the source domain and target domain are identical. However, we do not explore its application in more complex domain adaptation scenarios, such as open-set domain adaptation [1], partial domain adaptation [2], weakly-supervised domain adaptation [3], etc. (b) Task. This submission only investigates the application of JMMD-HSCI in cross-domain classification tasks, while its potential in other more complex tasks, such as semantic segmentation and object detection, remains under-explored. (c) Classifier. Although this study has verified that JMMD can improve performance across different classifiers, how to design JMMD kernels for different classifiers has not been sufficiently explored.

[1] Saito K., et al. Open Set Domain Adaptation by Backpropagation. ECCV, 2018.

[2] Cao Z. J., et al. Partial Adversarial Domain Adaptation. ECCV, 2018.

[3] Shu Y., etal. Transferable Curriculum for Weakly-Supervised Domain Adaptation. AAAI, 2019.