Bayesian Domain Adaptation with Gaussian Mixture Domain-Indexing

We thank the reviewer for the detailed and positive comments. The following is our responses to the questions mentioned in the comments.

1. Computational Overhead: The use of CRP and dynamic mixtures increases computational overhead, which might make the method less practical for large-scale or real-time applications without further optimization. It would be helpful if the authors could provide some comments and empirical analysis on the computational overhead of GMDI.

We appreciate this useful suggestion of the reviewer. In the global "Author Rebuttal", we conduct detailed ablation and computational overhead experiments comparing the proposed GMDI with GMDI w/o CRP. Please kindly refer to the global "Author Rebuttal".

2. Typos? Should the left side of Equation (1) be $p\left ( z\mid x, \varepsilon \right )$ or should $\int_{x}$ be added to the right side? A similar issue seems to exist in Equation (5) as well.

Thanks for pointing out the typos. The left side of Equation (1) and Equation (5) should be corrected to $p\left ( z\mid x, \varepsilon \right )$. We will include the correction in the final version.

3. Abstract Claim I do not understand the authors' claim in the abstract, “For classification, GMDI improves accuracy by at least 63% (from 33.5% to 96.5%).” It seems from Table 1, the result for DG-60, the authors picked the worst performance of all baselines, ADDA with an accuracy of 33.5%. It is unclear why the authors claim the improvement is at least 63%.

We apologize for the misleading claim. We will correct it in the final manuscript.

We thank the reviewer for the positive comments, the insightful questions, and helpful suggestions. The following are our responses to the questions mentioned in the comments.

1. GMDI relies on the availability of domain identities but cannot infer them as latent variables. This limits its applicability to scenarios where domain identities are also unknown.

Thank you to the reviewer for pointing out this weakness. Our proposed GMDI focus on inferring domain indices following a mixture of Gaussian distributions, with the number of mixture components dynamically determined by a Chinese Restaurant Process (CRP). Extensive experiments on classification and regression tasks demonstrate the strong domain index modeling capability of GMDI, significantly outperforming state-of-the-art methods. However, we acknowledge that GMDI's reliance on domain identities limits its applicability in situations where domain identities are unknown. We plan to address this issue in future work.

2. The use of the Chinese Restaurant Process and Gaussian Mixture Model can be computationally intensive, especially for large-scale datasets with numerous domains. This could hinder the scalability of GMDI.

We appreciate the reviewer for highlighting this issue. In the global "Author Rebuttal", we conduct detailed ablation and computational cost experiments comparing the proposed GMDI with GMDI w/o CRP and VDI. Please kindly refer to the global "Author Rebuttal" for a comprehensive analysis.

3. In the experiment, the binary classification task is not challenging, which makes the performance advantage not convincing. The multi-classification tasks are necessary.

Thank you for bringing this to our attention. Fllowing VDI, we have tested the performance of GMDI on the multi-classification dataset CompCars. The CompCars dataset is a 4-way classification dataset containing 18,735 samples across 30 domains. Due to the scarcity of datasets with ground-truth domain indices, we leave the extension of experiments on new datasets for future research. More details about the datasets used are available in Appendix H of the paper.

We thank the reviewer for the detailed comments and constructive suggestions. The following is our responses to the questions mentioned in the comments.

1. The learning loss for the proposed model is over too complex, featuring multiple conditional Kullback-Leibler divergences, which might complicate implementation and interpretation.

Thanks for pointing out this shortcoming. The learning loss for the proposed GMDI is indeed somewhat complex. GMDI follows the framework of VDI, which involves multiple latent variables and requires KL divergence constraints on these variables, making this complexity unavoidable. However, by modeling the domain index as a mixture of Gaussian distributions and using the Chinese Restaurant Process (CRP), GMDI demonstrates excellent performance on the relatively more complex CompCars dataset. This indicates its potential for application to even more intricate datasets, showcasing its broad applicability.

2. The paper lacks clear definitions for the 'local' and 'global' domain index within the probabilistic graphical model, which could confuse readers about the model's scope and applicability.

We apologize for the confusion. GMDI follows the setting of VDI.

(1) Local domain index $u$: It contains instance-level information, meaning each data point has a unique local domain index. Essentially, the local domain index can be viewed as an intermediate latent variable.

(2) Global domain index $\theta$: It contains domain-level information, meaning all data points within the same domain share the same global domain index. The global domain index is the true "domain index".

Our proposed GMDI models the global domain index (which contains domain-level information) as a mixture of Gaussian distributions. This mixture provides a higher level of flexibility in a larger latent space, thereby enhancing performance. We will incorporate the above definitions into the final version.

3. While introducing the Chinese Restaurant Process (CRP) adds flexibility to the Gaussian Mixture Model, it also increases computational costs. For the problem described in Figure 1, a fixed-component GMM might have been a simpler and more effective solution, though the number of components may not be that flexible.

We appreciate the reviewer for highlighting this issue. In the global "Author Rebuttal", we conduct detailed ablation and computational cost experiments comparing the proposed method with a fixed-component GMM. Please kindly refer to the global "Author Rebuttal".

4. The authors should also seriously consider empirically comparing the proposed method with a fixed-component counterpart, e.g. by simply using Gumbol softmax to infer the Gaussian component.

Thank you for the insightful suggestion. In the global "Author Rebuttal", we conduct detailed ablation and computational cost experiments comparing the proposed method with a fixed-component counterpart using Gumbol softmax. Please kindly refer to the global "Author Rebuttal".

5. The related work may not be sufficiently discussed. For example [a] discussed an end-to-end approach that learns the domain index using adversarial learning; [b] takes the domain index/identity as a latent dynamical system, coupled with adversarial learning.

Thanks for the mention of these two related works. We follow the setting of VDI, where the concepts of "domain index" and "domain identity" indeed share significant similarities, though they are somewhat different. The two mentioned papers primarily focus on "domain identity", and they are both highly valuable and worthy of discussion. We will discuss them in the related work section of the final version.

----------Minor----------

6. The connection between Equation (5) and Equation (6) is not clearly explained, leaving a gap in understanding the sequential logic of the model's formulation. -The paper's clarity and accuracy in writing could be improved. For instance, the statement "DP requires a predefined number of components" is misleading, as Dirichlet Processes are inherently nonparametric and do not require a predefined number of components.

We appreciate the reviewer for pointing out the issues.

(1) Equation 5 and Equation 6 both represent the posterior factorization of data encoding $z$. $\theta$ in Equation 5 is the global domain index. While GMDI models the global domain index $\theta$ as a mixture of Gaussian distributions, $\theta ^{v}$ in Equation 6 indicates the $v$-th component of the mixture distribution of $\theta$ with the prior $p\left ( v \right )$.

(2) Thanks for pointing out the misleading statement. The Dirichlet Process (DP) is indeed non-parametric. Due to the difficulty of direct construction, we apply the Chinese Restaurant Process by stick-breaking with a predefined bound $K$ to implement the DP. We will correct it with the explanatory sentences above.

7. Several symbols used in the equations are not adequately explained (both in the major paper and appendix), making it difficult to fully grasp the proposed model and its mathematical foundations.

We apologize for the missing explanations of some notations. Below is a detailed explanation of the missing notation：

$\beta$: independent random variable with a Beta distribution in the stick-breaking representation;

$\pi$: the probability vector in the stick-breaking;

$\gamma$: parameter of $q(\beta)$;

$\eta$: parameter of $q(v)$.

We will include the explanations of the above notations in the final version.

8. The proofs and the theoretical part seem to closely follow that of VDI, making it hard to evaluate its novelty.

Thank you for bringing this to our attention. Compared to VDI, the theoretical contribution of the proposed GMDI is that it proves GMDI achieves a more stringent evidence lower bound, as well as a tighter upper bound of the objective. Theoretically, it demonstrates that GMDI can effectively infer optimal domain indices.

Thank you very much for your prompt response. The performance gap between the two models, i.e., GMDI w/o CRP and VDI, is significant, which is what we expected and good result. As you suggested, we further applied a two-tailed paired t-test and found that the statistical significance of the observed performance difference between these two models is significant for alpha=0.05. Because of the positive experimental result, we are expecting you to at least upgrade the current rating score to the original rating score. Thank you very much for you kind support.

To avoid misunderstanding and further clarify the experimental results presented in the global "Author Rebuttal," we would like to provide the following additional explanation. VDI is our baseline, which models the domain index as a single Gaussian. Our proposed GMDI models the domain index as a mixture of Gaussians and incorporates CRP. GMDI w/o CRP is a version that follows your suggestion by not using CRP and instead utilizing a fixed-component GMM (Gaussian Mixture Model with a fixed number of components).

We conducted detailed experiments on the relatively complex TPT-48 and CompCars datasets. The results show that GMDI w/o CRP performs better than VDI but falls short of GMDI. This indicates that using a fixed-component GMM improves performance, but the lack of CRP leads to results inferior to those of GMDI, demonstrating the importance of both GMM modeling and the use of CRP. Please let us know if we need to provide further explanation. Thank you very much.

Thank you for your detailed comments. The following is our responses to the concerns mentioned in the comments.

1. "However, the study primarily builds upon the existing framework of VDI, and its theoretical developments, which are closely aligned with those of NDI, are mostly relegated to the appendix without appropriate citations. This lack of clarity suggests that some key proofs might be incorrectly attributed to this work."

Thank you for pointing out this issue. Our theorems/lemmas are indeed partially based on VDI, and we will carefully review them to ensure that VDI is correctly cited in both the theorems/lemmas in the main body of our paper and the proofs in the appendix in the final version.

Compared to VDI, the theoretical contribution of the proposed GMDI is that it proves GMDI achieves a more stringent evidence lower bound, as well as a tighter upper bound on the objective. Theoretically, it demonstrates that GMDI can effectively infer optimal domain indices.

2. "Additionally, the use of the Chinese Restaurant Process (CRP) to manage an unknown number of Gaussian components is not innovative, as thoroughly discussed in prior research [1]. Therefore, both the technical novelty and theoretical contributions of this paper seem incremental."

Thanks for pointing out this. The main innovation of our proposed GMDI is that, to the best of our knowledge, GMDI is the first to introduce a CRP-based Gaussian mixture model to represent the domain index.

3. "The experiments presented are not sufficiently comprehensive. I can understand that the authors have difficulties in conducting experiments using additional GMM variants, yet such comparisons are essential for a robust evaluation. Notably in the experiments, the CRP model frequently identifies the optimal number of components as two or three. It is important to figure out why CRP outperforms fixed-component GMMs, given that the "optimal" number of Gaussian components is known. A more thorough ablation study could clarify these results."

Thanks for mentioning this. Since fixed-component GMMs are fundamentally similar, none of them can adaptively determine the number of mixture components. It is neither realistic nor meaningful to compare all variants of fixed-component GMMs within a short period. We have already conducted a detailed comparison with the Gumbel softmax version (i.e., GMDI w/o CRP). The experimental results show that our proposed GMDI demonstrates clear advantages. We leave the extension of comparative experiments with additional GMM variants for future research.

Regarding why the CRP model (i.e., GMDI) outperforms the fixed-component GMM (i.e., GMDI w/o CRP), it may be due to the complexities of the training process and parameter search space. Firstly, during training, the varying sample distribution within each batch can make it difficult for the fixed-component GMM (i.e., GMDI w/o CRP) to adapt well under a fixed K, potentially leading to suboptimal performance and convergence challenges. In contrast, the CRP model (i.e., GMDI) can automatically adapt to the sample distribution during training, and experiments have observed that GMDI converges faster to better performance, which aligns with our expectations.

Secondly, due to the complex parameter space, it is challenging to fine-tune both the fixed-component GMM (i.e., GMDI w/o CRP) and GMDI to their optimal performance within a short time frame. However, under fair comparison conditions, GMDI demonstrates superior performance and convergence speed compared to the fixed-component GMM (i.e., GMDI w/o CRP) within a limited search space, while significantly reducing the cost of parameter searching. As shown in the experimental results in the table below, we have tested various hyperparameters, K and $\tau$, as thoroughly as possible. The experiments indicate that even under the fairest possible comparison conditions, the fixed-component GMM (i.e., GMDI w/o CRP) struggles to achieve better performance, thereby validating our conclusions.

（1）GMDI w/o CRP on CompCars dataset: