Q1. Overall, it feels like there is a hole that needs to be filled between the entangled importance weight and HFA condition, that is, what happens when HFA is not perfectly achieved (which is almost surely the case), should we change the $\mathbf w$ estimation strategy and estimation characterization?

A1. We appreciate your insightful observation. In practice, achieving perfect Heterogeneous Feature Alignment (HFA) is challenging, which may introduce biases in importance weight estimation. To mitigate this, we adopt a joint optimization approach that simultaneously optimizes Importance Weight Estimation and Heterogeneous Feature Alignment, allowing the two modules to interact synergistically and form a positive feedback loop that enhances overall performance. Specifically, better HFA reduces distribution mismatch, leading to more accurate importance weight estimation, while improved weight estimation provides a more reliable training signal, further refining feature alignment. Of course, when there is a significant deviation in HFA, the estimated importance weight $\mathbf{w}$ may become biased due to residual feature mismatches between domains, which will further affect the adaptation performance. In this case, we need to adjust the $\mathbf w$ estimation strategy. For example, prior information can be incorporated to design regularization terms that correct $\mathbf w$.

Q2. Regarding Theorem 3.5, the result seems rather counter intuitive, because the total variation distance between the two measures can be arbitrarily large when the probability of one class diminishes in distribution shift ($\rho \rightarrow 0$). This kind of implies the bound could be potentially vacuous. Can the authors clarify on this issue?

A2. Thanks for your insightful comments. Theorem 3.5 gives an upper bound on the discrepancy of the feature distribution conditional on the label. We think it is not counter-intuitive since $\rho \rightarrow 0$ only implies a loose upper bound and does not imply that the arbitrarily large total variation. In other words, an infinite upper bound is not equivalent to an infinite total variation. In fact, as shown in Eq. (1), the total variation is a value within the range [0,1]. In addition, Theorem 3.5 reflects the natural limitation that, in extreme cases of label shift, adaptation becomes fundamentally more difficult. Most theoretical results in domain adaptation and importance weighting [1, 2] similarly exhibit such behavior as $\rho \rightarrow 0$. In this paper, we mainly consider the mild case consistent with the mainstream Label Shift [2, 3], and first give a theoretical analysis of the brand-new HLS problem. Based on the theory, we propose a bound minimization HLS framework and achieve good performance. In fact, the extreme imbalance problem has always been an important puzzle, and how to construct more tight upper bounds in this case is a very interesting research problem. $d_{TV}(P,Q)= \mathop {\sup }\limits_{E~\text{is measurable}}\left|P(X\in E)-{Q} (X\in E)\right|.~~~~(1)$

[1] Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, Geoffrey J. Gordon: Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift. NeurIPS 2020.

[2] Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, Animashree Anandkumar: Regularized Learning for Domain Adaptation under Label Shifts. ICLR (Poster) 2019.

[3] Ruidong Fan, Xiao Ouyang, Tingjin Luo, Dewen Hu, Chenping Hou: Incomplete Multi-View Learning Under Label Shift. IEEE Trans. Image Process. 32: 3702-3716 (2023).

Q3. The same concern goes with Theorem 3.8, as it might be the case that for a "bad" predictor, the importance weight could also be arbitrarily large (${\sigma _{\min }(\mathbf{C})}\rightarrow 0$). Is there any guarantee or general explanation on how can this be avoided?

A3. Thanks for your insightful comments. Theorem 3.8 gives an upper bound for Importance Weight Estimation. ${\sigma _{\min }(\mathbf{C})}\rightarrow 0$ leads to a loose upper bound for weight estimation error. Similarly, an infinite upper bound is not equivalent to an arbitrarily large weight estimation error. It typically occurs when the model is poorly trained or when the source and target domains have extreme shifts. In our paper, to avoid this problem, as mentioned in Appendix B.2, we first warm up HLSAN by training without weighting for the first 20 epochs to obtain an acceptable predictor, and then we build estimators for $\mathbf{C}$. In this case, $\mathbf{C}$ is a diagonally dominant real symmetric probability matrix, which guarantees that $\mathbf{C}$ is invertible and its singular values are all greater than 0. Through such way, the extreme cases suggested by the reviewer are largely avoided. Since this issue is a common challenge for label shift, and we have employed certain strategies to mitigate it as much as possible, how to fully resolve it from a theoretical perspective remains an open problem.

Q1. What is the primary role of the unlabeled parallel instances, and would the proposed method become inapplicable if these data were lacking?

A1. Thank you for your insightful question. As the second response to the Reviewer SzhU, parallel instances establish a cross-modal channel that facilitates effective knowledge transfer, linking the source and target domains while mitigating extreme negative transfer, as discussed in Appendix B.3. Thus, our method has a reliance on parallel instances. In reality, parallel instances are common and easy to obtain compared to labeled target data. We then investigate this problem setting and propose an theoretical inspired models. Besides, we also recognize the challenges due to the lack of parallel instances. In future work, we will continue to explore strategies such as self-supervised alignment, semantic similarity, and contrastive learning to explore the task of cross-modal knowledge transfer without parallel instance.

Q2. If the unlabeled parallel instances are crucial, should the experiments analyze the impact of the number of these instances on the final performance?

A2. Thank you for the insightful suggestion. To validate this, we have conducted preliminary experiments to analyze the impact of the number of parallel instances on final performance and the results are shown in the following tables. More thorough investigations will be added to the final version. Based on these observations, we can see that performance gradually improves as more parallel instances are available.

The influence of Parallel Instances for HLSAN.

Q3. While there is existing work on both heterogeneous domain adaptation and label shift, what unique challenges does the Heterogeneous Label Shift problem introduce?

A3. Thanks for your question. We think coupled Heterogeneous Feature and Label Shift is the unique challenges of HLS problem. Unlike traditional label shift, where only the label distribution changes, HLS involves simultaneous shifts in both feature space and label distribution, making standard importance-weighting methods insufficient. Unlike traditional heterogeneous domain adaptation, HLS considers the more general case where heterogeneous features are accompanied by label shift, which makes the existing feature-alignment methods fail to achieve joint distribution alignment. More seriously, our HLS problem can not be decoupled into two sequent or parallel problems since Heterogeneous Feature Alignment and Importance Weight Estimation are intertwined. It is not ideal to simply use existing heterogeneous domain adaptation and label shift techniques to solve the two problems separately. To solve this problem, HLSAN designs two interactive modules of Heterogeneous Feature Alignment and Importance Weight Estimation respectively. By interacting them synergistically, we form a positive feedback loop that enhances overall performance.

Q1. Typos and suggestions.

A1. Thank you for your valuable feedback. We have corrected the typo in the manuscript and checked it carefully throughout. In addition, as suggested by the reviewers, we added a brief summary of the comparison methods, and the related work section has been moved earlier to improve the logical flow. Finally, we have revised Figure 2 with meaningful colors to distinguish learnable and fixed components, added explicit labels (e.g., $T_s, T_t, h$), and clarified the role of the encoder.

Q2. I don't get why a heterogeneous domain should imply a label shift. You can distribute the label in two different modalities. Even in the experiment, the label shifts are simulated.

A2. Thank you for your comment. In reality, since the collected data come from different modalities, it is more likely to cause label shift. For example, when images and text data are collected from different websites, it is almost impossible to maintain the same category ratio of picture and text data, which unavoidably cause the label shift problem. Of course, as the reviewer said, we can distribute label in two different modalities to avoid label shift. Nevertheless, in practice, this approach is not feasible since there is no label data in target domain. In fact, label shift is a well-documented phenomenon in domain adaptation and transfer learning [1,2], and the HLS problem is also raised from reality. Regarding the experiments, in order to evaluate our approach from multiple angles and multiple label shift cases, we follow standard setup in label shift research [3,4] by simulating label shifts to systematically evaluate our method. This allows us to isolate the impact of label shift and better analyze its effects.

[1] Zachary C. Lipton, Yu-Xiang Wang, Alexander J. Smola: Detecting and Correcting for Label Shift with Black Box Predictors. ICML 2018: 3128-3136.

[2] Ruihan Wu, Chuan Guo, Yi Su, Kilian Q. Weinberger: Online Adaptation to Label Distribution Shift. NeurIPS 2021: 11340-11351.

[3] Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, Animashree Anandkumar: Regularized Learning for Domain Adaptation under Label Shifts. ICLR (Poster) 2019.

[4] Ruidong Fan, Xiao Ouyang, Tingjin Luo, Dewen Hu, Chenping Hou: Incomplete Multi-View Learning Under Label Shift. IEEE Trans. Image Process. 32: 3702-3716 (2023).

Q3. The ablation study is done only over two adaptations SP$\rightarrow$EN and SP$\rightarrow$IT. Can we have the ablation study for both experiments' overall adaptations? You choose to take the worst case $\gamma=10$; what are the results for smaller $\gamma$?

A3. Thank you for your suggestions. Due to space constraints, we conducted the ablation study on two representative adaptations (SP$\rightarrow$EN and SP$\rightarrow$IT) and selected $\gamma=10$ as Illustration. In fact, we have done all the experiments. As shown in the following tables. The results can also reveal the same two key findings. 1) Removing any component degrades performance, demonstrating the importance of each term. 2) Incorporating with importance weight enhances performance, indicating the necessity of aligning label distribution shifts.

The accuracy (%) of ablation study of HLSAN with $\gamma = 10$.

The accuracy (%) of ablation study of HLSAN with $\gamma = 5$.

The accuracy (%) of ablation study of HLSAN with $\gamma = 2$.

Q1. How does HLSAN scale with the number of classes ($k$)? Does the RQP estimator degrade for $k > 100$?

A1. Thank you for your thoughtful question. Based on our theoretical analysis (Theorem 3.8) as shown in the following formula, the weight estimation error is sublinear with respect to $k$ so HLSAN can efficiently scale with $k$. Of course, when k is large, such as $k \leq100$, it will inevitably reduce the performance of the RQP estimator due to the increased classification difficulty. In fact, weight estimation with large number of classes has been a recognized difficult problem in label shift. But by increasing the sample size we can mitigate that the impact of the increase in categories at this time. $||\hat{\mathbf{w}} - \mathbf{w}||_2\le c_1\sqrt {\frac{{72k}}{{{n_s}}}\log \left( {\frac{{12k}}{\delta }} \right)}+ c_2\sqrt {\frac{{9k}}{{{n_t}}}\log \left( {\frac{{6k}}{\delta }} \right)}.$

Q2. Can the framework handle non-parallel modalities (e.g., audio→text) without instance pairs $\mathcal O$?

A2. Thank you for your insightful question. Parallel instances provide a cross-modal channel, which enables HLSAN to perform effective knowledge transfer. Different from previous settings with some assumptions to connect source and target domains, instance pairs are vital for our approach since there is no labeled data in the target domain. Therefore, it is difficult for HLSAN to handle non-parallel modalities without instance pairs $\mathcal O$. In fact, we investigate this setting since the parallel instances are now common and easily accessible, such as Image-text data, news broadcasts. We recognize the challenges due to the lack of parallel instances. In the future, we'll explore strategies such as self-supervised alignment, semantic similarity, and contrastive learning to enhance performance in this case.

Q3. What is the computational overhead of adversarial alignment vs. standard DANN?

A3. We appreciate the reviewer’s concern regarding the computational cost. Compared to standard DANN, adversarial alignment introduces additional computations primarily due to the alternating optimization introduces additional updates for $L_O$ and $L_{KT}$. Specifically, each training step requires solving two subproblems sequentially, which adds approximately 60% increase in training time per epoch. Fortunately, our method can level off within 30 epochs in most cases, and the additional computational cost is also affordable in real applications.