Noises are Transferable - An Empirical Study on Heterogeneous Domain Adaptation

Thanks very much for your valuable comments and questions. We address your concerns as follows.

W1. Although the authors draw some interesting results, where the noise can be transferred from the source to the target domain, I think this result is still counterintuitive. If the noise is transferable in semi-supervised heterogeneous domain adaptation, it also can be transferred in the unsupervised heterogeneous domain adaptation, it is unclear why the authors limit this strong conclusion in the semi-supervised scenarios.

A. This is a good question. Although noises are transferable, they cannot be directly applied in unsupervised scenarios. Indeed, we generate source noises based solely on the number of categories. For instance, let's assume that there are 10 categories in the target domain. We sample noises from 10 distinct Gaussian distributions and treat them as samples belonging to different categories. Accordingly, those noises are task-agnostic, and their categories have no practical meaning. Hence, to avoid randomness in the learning process of a specific target task, having a small number of labeled target samples is essential. Also, their category indices are used to establish connections with those of noises. Based on our findings, we can utilize domain adaptation mechanisms to jointly train all samples, including noises, and labeled and unlabeled target samples, to enhance target performance.

W2. According to Section 5, the authors claim that the label information of the source sample might not be the primary factor that influences the performance of SHDA. Since the authors conduct the experiments with some large models like JMEA. It employs the ResNet-50, a very deep pre-trained network, which might contain some label information. Therefore, it cannot sufficiently reflect that the label information is useless. It is suggested that the authors should employ a lighter neural architecture like AlexNet to evaluate the proposed idea.

A. We feel that there may be a misunderstanding about the baselines. In the SHDA problem, mainstream deep transformation approaches, including TNT, STN, SSAN, and JMEA, typically follow a two-step process. They first pre-process source and target samples by extracting heterogeneous features, e.g., $S_{800}$ and $D_{4096}$ (please refer to the Experimental Setup section in our manuscript). Then, to perform heterogeneous domain adaptation, they train lightweight neural network architectures from scratch. Accordingly, JMEA does not contain any label information. Also, we have used lightweight neural network architectures for experiments. Thus, based on the experimental results, we can infer that the label information of the source sample is not the primary factor influencing the performance of SHDA.

W3. Since the authors evaluate the performance in the semi-supervised scenario, it is suggested that the authors should provide the experiment results of training with labeled target data. Because I doubt that some target-labeled data with a reasonable learning rate (to avoid overfitting) might be enough for the ideal performance.

A. We have reported the performance of supervised learning baselines, i.e., SVMt and NNt. Specifically, SVMt and NNt only utilize the labeled target samples to train a support vector machine and a neural network, respectively (please refer to the Experimental Setup section in our manuscript). Also, in our manuscript, we can clearly see from Table 1 that, most SHDA methods are far better than SVMt and NNt. Specifically, on the task of A ($S_{800}$) $\rightarrow$ C ($D_{4096}$), the classification accuracy of STN is 88.41%, which substantially exceeds the best supervised method, i.e., NNt, by 7.82%. Furthermore, it is difficult to achieve ideal performance solely through supervised learning approaches, which can be confirmed in many SHDA studies.

W4. In the part ‘Study on feature information of source samples’, the authors use features with different dimensions to denote different information, for example, D(4096) contains more information than D(800). This might be unreasonable since features with different dimensions might contain equal information.

A. In our experiments, not only do the dimensions differ, but the meaning of each dimension is also distinct. Specifically, we follow the stand SHDA settings and adopt two different features, i.e., $S_{800}$ and $D_{4096}$, to represent images (please refer to the Experimental Setup section in our manuscript). They are completely different feature representations. Also, we do not use the $D_{800}$ features to represent images.

Thanks very much for your valuable comments and questions. We address your concerns as follows.

W1. Could you clarify the distinction between Semi-supervised Heterogeneous Domain Adaptation (SHDA) and Semi-supervised Domain Adaptation (SSDA) according to Definition 1? Both seem to align with Definition 1. It appears that $d_s$ and $d_t$ merely represent specific data dimensions and may not capture the heterogeneity in the nature or type of features between the source and target domains.

A. Thanks for your suggestion, we will revise Definition 1 by introducing the source and target feature spaces, i.e., $\mathcal{X}_s$ and $\mathcal{X}_t$. Under the SHDA setting, $\mathcal{X}_s \neq \mathcal{X}_t$, while in the SSDA problem, $\mathcal{X}_s = \mathcal{X}_t$.

Q1. What was the rationale behind using fixed parameter settings for different SHDA tasks on the same dataset? Doesn't this approach risk not capturing the optimal performance for each task?

A. Indeed, as suggested in the original papers of those baselines, they utilize the same parameter setting across different SHDA tasks on the same dataset. Consequently, adopting such settings could ensure optimal performance to some extent. In our experiments, we solely modify the factors related to source samples, e.g., the number of source samples, to construct distinct SHDA tasks. Thus, to ensure a fair comparison, we adhere to the control variable principle, and maintain a consistent parameter setting for each baseline across different tasks on the same dataset.

Q2. In the section of analysis on the original discriminability of source samples, what led to the choice of $\lambda = 0.4$ and $\lambda = 0.41$?

A. We are sorry for this confusion. In the experiments, we design two strategies, i.e., Category Replicate and Category Shift, to quantitatively explore how the original discriminability of source samples affects the performance of SHDA. In the former strategy, we generate four noise domains, i.e., NK$_3$, NK$_4$, NK$_5$, and NK$_6$. Their LDA values are 40.51, 51.55, 55.77, and 60.57, respectively. We find that all SHDA methods tend to perform better as the LDA values increase, which seems reasonable. To further illustrate the generality of this conclusion, we utilize the latter strategy to empirically create two noise domains, making their LDA values approach to those of NK$_5$ and NK$_6$, respectively. Accordingly, we empirically set the values of $\lambda$ to 0.40 and 0.41, thereby creating the noise domains of NL$_{0.40}$ and NL$_{0.41}$ with LDA values of 57.64 and 60.56, respectively. However, building upon their experimental results, we find that the above conclusion does not hold. Therefore, we construct two counterexamples to empirically demonstrate that the original discriminability of source samples is not the primary source of transferable knowledge in SHDA tasks. We will add the above statement in the revision.

Q3. In the section of analysis on the transferable discriminability of source samples, when using $g_t (\cdot)$ as a single layer fully connected networks with the Leaky ReLU, is there any potential for underfitting the data?

A. Since HCN is trained using extracted features (e.g., $DeCAF_6$) rather than raw samples, even with the adoption of lightweight networks, under-fitting is unlikely to occur.

Thank you for the additional clarification and for updating the manuscript. I consider this work to be a valuable contribution to the field of domain adaptation, and I am inclined to maintain my original score.

A. Thanks again for affirming the value of our work and we appreciate it!

Thanks very much for your valuable comments and questions. We address your concerns as follows.

W1. I think the experiment of “label information” is meaningless. There is no doubt that the order of category indices would not affect the performance since they are just symbols without any semantics. Additionally, I do not think label information can be regarded as category indices.

A. We feel that there may be a misunderstanding about the study on label information of source samples. We want to emphasize that we keep the order of category indices unchanged for target samples and only modify them for source samples (please refer to Figure 1 in our manuscript). As an example, let's assume that there are three categories, i.e., cat, dog, and horse, and their ground-truth category indices are 1, 2, and 3, respectively. In the target domain, we keep the ground-truth category indices unchanged. However, in the source domain, we randomly change the category index of samples belonging to the same category to that of another category. For instance, we change the category indices of cat, dog, and horse to 2, 3, and 1, respectively. As a result, there is no one-to-one correspondence between the categories of the source and target domains. Intuitively, this can affect target performance, especially in homogeneous scenarios where the source and target samples share the same feature extractor. In the SHDA, However, based on our experimental results, we observe that as the orders of category indices for source samples change, the accuracies of all methods remain almost unchanged. This is a surprising and crucial finding. Building upon this, we perform subsequent cross-dataset experiments to explore the impact of feature information on SHDA performance. In those experiments, the categories of the source and target samples are completely distinct. In addition, we can modify label information to category information in the revision.

W2. In Figure 3, the performance of SSAN changes significantly when the feature dimension changes, making the conclusion that feature information is not the dominant factor not convincing.

A. The principles of different methods are distinct, and there may be variations in performance trends. Hence, we observe the overall performance trends of all methods as source domains change, rather than focusing on individual ones. Although SSAN shows relatively large fluctuations in performance, most methods exhibit relatively stable performance. Thus, those results can imply that the feature information of source samples is not the dominant factor affecting the performance of SHDA. We will explain it more clearly in the revision.

W3. In the experiment of Table 1, I think the method of not adopting transfer learning should be included since both true source samples and noises may be unhelpful in this experiment.

A. In our experiments, SVMt and NNt are two supervised learning methods without any transfer mechanisms. Specifically, SVMt and NNt only utilize the labeled target samples to train a support vector machine and a neural network, respectively. Accordingly, we have reported the performance of non-transfer learning methods. Also, based on the results listed in Table 1, we can clearly observe that most SHDA methods are far better than SVMt and NNt. Specifically, on the task of A ($S_{800}$) $\rightarrow$ C ($D_{4096}$), the classification accuracy of STN is 88.41%, which substantially exceeds the best supervised method, i.e., NNt, by 7.82%. Furthermore, it is difficult to achieve ideal performance solely through supervised learning approaches, which can be confirmed in many SHDA studies (Please see details in [1-4]). Therefore, we have good reason to believe that using either noises or true source samples will contribute to improving target performance.

References

[1] Wei-Yu Chen, Tzu-Ming Harry Hsu, Yao-Hung Tsai, Yu-Chiang Frank Wang, and Ming-Syan Chen. Transfer neural trees for heterogeneous domain adaptation. In ECCV, 2016.

[2] Yuan Yao, Yu Zhang, Xutao Li, and Yunming Ye. Heterogeneous domain adaptation via soft transfer network. In ACM MM, pp. 1578–1586, 2019.

[3] Shuang Li, Binhui Xie, Jiashu Wu, Ying Zhao, Chi Harold Liu, and Zhengming Ding. Simultane- ous semantic alignment network for heterogeneous domain adaptation. In ACM MM, pp. 3866–3874, 2020.

[4] Zhen Fang, Jie Lu, Feng Liu, and Guangquan Zhang. Semi-supervised heterogeneous domain adaptation: Theory and algorithms. TPAMI, 45(1):1087–1105, 2023.

Thank you for your responses. However, your replys do not solve my concerns well. So, I would like to maintain my original scores.

1. In the example you mentioned, the category oder affects the performance since both the feature extractor and classification head are transferred. However, since only the feature extractor is reused in your experiments while the classification head is discarded , the category order does not affect. This has nothing to do with homogeneity or heterogeneity. I insist that the experiment of category order is not meaningful.
2. I can not agree with your opinion. In addition to SSAN, the changes of JMEA and DDACL are not negligible.
3. SVMt and NNt are two weak baselines. I would like to know the performances of the SHDA methods in Table 1 without using true source samples or noises.
4. Assigning a sample with two (or more) different category labels would also damage the training. We can not know whether the discriminability or damage causes the effect.
5. Additionally, the conclusion that the primary source of transferable knowledge in SHDA tasks does not lie in the original discriminability of source samples is not convincing since the performance improves when LDA values increase in NK3,4,5 and 6.
6. ''We find that achieving higher target performance becomes remarkably easy by using different means and variances to generate noises.'' This phenomenon doest no support your conclusion about original discriminability.
7. Moreover, category replicate and category shift can not fully represent original discriminability.
8. I suggest to apply the same metric for measuring the two discriminability.

W2. I can not agree with your opinion. In addition to SSAN, the changes of JMEA and DDACL are also negligible.

A2. To further explain this, we can calculate the coefficient of variation (CV). It is a statistical measure used to assess the relative variability, defined as the ratio of the standard deviation (STD) $\sigma$ to the mean $\mu$ [1]. The following tables list the raw data used to draw Figure 3 in our manuscript. Based on those results, we can clearly see that the maximum value of CV for SSAN is 0.04, which is generally considered a relatively small value mathematically. In the revision, we will incorporate those results and discussions into the Appendix for supplementary clarification.

References

[1] https://en.wikipedia.org/wiki/Coefficient_of_variation

Thanks for your patience and meticulous response. We address your concerns as follows.

W1. In the example you mentioned, the category order affects the performance since both the feature extractor and classification head are transferred. However, since only the feature extractor is reused in your experiments while the classification head is discarded , the category order does not affect. This has nothing to do with homogeneity or heterogeneity. I insist that the experiment of category order is not meaningful.

A1. We feel that there may be a misunderstanding about the training manner of Semi-supervised Heterogeneous Domain Adaptation (SHDA) (please refer to Section 3 in the manuscript for the problem definition of SHDA). Unlike source-free domain adaptation, SHDA does not begin by pre-training a feature extractor and then fine-tuning it by using both labeled and unlabeled target samples. In SHDA, all approaches typically follow a two-step process. They first pre-process source and target samples by extracting heterogeneous features, e.g., 800-dimension $SURF$ ($S_{800}$) and 4096-dimension $DeCAF_6$ ($D_{4096}$), and then perform heterogeneous domain adaptation by jointly training the source and target features from scratch. Assuming that the source and target features are homogeneous and share the same feature projection, we believe that the order of category indices of source features will affect target performance.

To empirically verify this, we have conducted an additional experiment on the S domain. Specifically, we randomly and evenly partition all samples into the source and target domains, i.e., S$\_s$ and S$\_t$. For the source domain, we utilize all samples as labeled ones. As for the target domain, we randomly sample 3 samples per category as the labeled ones, and the rest of the samples are considered as the unlabeled ones. This is a homogeneous scenario where the source and target samples are from the same feature space. Following the setting in our manuscript, for source samples within the same category, we randomly change their category index to correspond to a distinct category. The table below lists the performance of NN, which learns a neural network by using all labeled source and target samples. Note that in NN, a shared feature projection for both source and target samples is learned from scratch. We can see that when the category indices of source samples do not follow the ground-truth order, i.e., order 1, the target performance is extremely poor. This implies that the order of category indices for source samples is important in such scenarios. This is in line with our expectation.

In our experiments, however, we find that the target performance is not affected by the order of category indices of source features. The primary reason is that in SHDA, due to the heterogeneity of source and target features, most approaches utilize different feature projections for them. Accordingly, we believe that this finding exhibits distinctive characteristics of SHDA, rather than being a meaningless finding. We will make it more clearly in the revision.

W3. SVMt and NNt are two weak baselines. I would like to know the performances of the SHDA methods in Table 1 without using true source samples or noises.

A3. Similar to W1, we believe that there may be a misunderstanding regarding the training manner of SHDA. Please refer to A1 for details. To the best of our knowledge, all SHDA approaches require source features for classifier training and distribution alignment. As an example, in SHDA, a simple and common distribution alignment mechanism is to align the global and category centroids of the source and target features during adaptation. Therefore, they cannot be directly extended to scenarios without source features.

W4. Assigning a sample with two (or more) different category labels would also damage the training. We can not know whether the discriminability or damage causes the effect. Additionally, the conclusion that the primary source of transferable knowledge in SHDA tasks does not lie in the original discriminability of source samples is not convincing since the performance improves when LDA values increase in NK3,4,5 and 6. ''We find that achieving higher target performance becomes remarkably easy by using different means and variances to generate noises.'' This phenomenon does not support your conclusion about original discriminability. Moreover, category replicate and category shift can not fully represent original discriminability. I suggest to apply the same metric for measuring the two discriminability.

A4. Thanks for your suggestion. To address your concerns and further confirm our conclusion, we have conducted additional experiments to examine how the original discriminability of source samples affects the performance of SHDA. Specifically, we first randomly generate $C$ different means and variances, i.e., $\boldsymbol{\mu\_1}$, $\boldsymbol{\mu\_2}$, $\cdots$, $\boldsymbol{\mu}\_C$, and $\mathbf{\Sigma}\_1$, $\mathbf{\Sigma}\_2$, $\cdots$, $\mathbf{\Sigma}\_C$. Then, we multiply them by a coefficient of $\alpha$ to quantitatively control the magnitude of the discriminability, i.e., $\boldsymbol{\mu}\_1 \times \mathbf{1} \times 1 \alpha$, $\boldsymbol{\mu}\_2 \times \mathbf{1} \times 2 \alpha$, $\cdots$, $\boldsymbol{\mu}\_C \times \mathbf{1} \times C \alpha$, and $\mathbf{\Sigma}\_1 \times \mathbf{1} \times 1 \alpha$, $\mathbf{\Sigma}\_2 \times \mathbf{1} \times 2 \alpha$, $\cdots$, $\mathbf{\Sigma}\_C \times \mathbf{1} \times C \alpha$. Here, $\mathbf{1}$ denotes an all-one matrix with an appropriate size. Accordingly, we create different noise domains by setting distinct values of $\alpha$. Generally, the larger the value of $\alpha$, the greater the discriminability. As a result, we construct 10 noise domains, i.e., NA$\_{0.2}^6$, NA$\_{0.4}\^6$, NA$\_{0.6}\^6$, NA$\_{0.8}\^6$, NA$\_{1.0}\^6$, NA$\_{0.2}\^{10}$, NA$\_{0.4}\^{10}$, NA$\_{0.6}\^{10}$, NA$\_{0.8}\^{10}$, and NA$\_{1.0}\^{10}$. Here, the superscript denotes the total number of categories, and the subscript denotes the value of $\alpha$. Moreover, we adopt the empirical risk of source samples to measure the original discriminability. To this end, we first randomly select 10% of samples from each category as training samples, while the rest are assigned as testing samples. Then, we use those training samples to train a neural network and report the empirical risk on testing ones, i.e., $\mathcal{R}\_s\^o$.

The table below reports the average results of 10 random trials. Note that due to the relatively small values of $\mathcal{R}\_s\^o$, we multiply them by 10 for comparison with those of $\mathcal{TR}\_s$. According to the results, we find that the performance of all approaches is almost constant as the values of $\mathcal{R}\_s\^o$ decline. This once again demonstrates that the original discriminability of source samples is not the primary factor influencing the performance of SHDA. Also, we can see that the change in the values of $\mathcal{TR}\_s$ is smaller than that of $\mathcal{R}\_s\^o$. According to those results, we have good reason to believe that the transferable discriminability of source samples stands as the dominant factor affecting the performance of SHDA.

We will utilize this experiment to replace the previous one. In addition, due to time constraints, in the above experiment, we mainly explore several advanced SHDA baselines, i.e., STN, SSAN, and JMEA, to once again verify that our conclusions are convincing. The results of other SHDA baselines will be included in the camera-ready version.

Thanks very much for your valuable comments and questions. We address your concerns as follows.

W1. The study is biased so the conclusion drawn from the study might not be generalizable. The features are precomputed by some descriptors and are not learnable. However, many semi-supervised domain adaptation methods, especially those based on deep learning, are powerful because they learn the feature extraction networks to generate discriminative features. If the feature is fixed, the real power of these methods cannot be realized, and the study based on this is not generalizable.

A. We feel that there may be a misunderstanding about the research motivation. Please refer to the general response for details.

W2. The effect of the number of labeled target samples has not been studied. For semi-supervised domain adaptation, the number of labeled target samples is a crucial factor for the adaptation performance. However, this study does not cover the investigation on this aspect.

A. We want to emphasize that, in this study, we do not propose a new SHDA approach. Instead, our focus is on exploring the essence of the SHDA, i.e., What knowledge from a heterogeneous source domain is transferred to the target domain? Accordingly, we adhere to the control variable principle. For the same target domain, we explore the above problem by keeping the number of target samples unchanged while changing the source domain. Also, based on our experience in the SHDA, we believe that, within a certain range, as the number of target samples increases, it will only enhance the performance of all baselines without affecting our current conclusions. Hence, we do not conduct experiments involving variations in the number of labeled target samples.

In addition, to empirically verify this, we report the classification accuracies of some approaches w.r.t. distinct numbers of labeled target samples per category in the following table. We can clearly observe that their performance is boosted with the increase in the number of labeled target samples. This is in line with our expectation. In the revision, we will include those results in the appendix for supplementary clarification.

W3. The value of this study is kind of limited. Semi-supervised heterogeneous domain adaptation is a very small field (as can be seen from the literature) and a study on such a small field can only draw attention on a small group of audience. While I agree this is valuable, the value is not very significant. We expect a study paper accepted at this conference to be of high scope so that a broad group of people can learn something from the study.

A. Thank you for acknowledging the value of our research. However, we respectfully and politely disagree that the value of research should be determined by the size of the field. Please refer to the general response for details.