Synergy Between Sufficient Changes and Sparse Mixing Procedure for Disentangled Representation Learning

W8: Same for the real-world, why not try CG-VAE. If there is some simple reason, authors should state in one sentence why VAE is not suitable for real-world datasets. Also, are there any other alternative methods for real-world, since all four methods are actually some method combinations? Please correct me if my understanding is wrong.

A8: Thank you for your valuable comments. We have included CG-VAE in our experiments with real-world datasets, and the results are shown in the table below. Please refer to the visualization results in Figure 5 of the revised version.

Regarding your question on VAE suitability: While VAEs are widely used, we found that their image generation performance on real-world datasets was suboptimal due to their tendency to produce blurry or less detailed images. This is because of the injected noise and imperfect reconstruction, and with the standard decoder (with factorized output distribution), the generated samples are much more blurred than those coming from GANs. There are other alternative methods like diffusion models, but they require significant computational resources, which makes them less practical for the scope of our current work.

Thanks for your reminder, we have added this experiment results in Appendix E.3.

[8] Li, Zijian, et al. "Subspace identification for multi-source domain adaptation." Advances in Neural Information Processing Systems 36 (2024).
[9] Eastwood, Cian, and Christopher KI Williams. "A framework for the quantitative evaluation of disentangled representations." 6th International Conference on Learning Representations. 2018.

W5: What is CG-VAE-S? It seems like it was used before it was introduced.

A5: Thanks a lot! We apologize for the oversight in the original version of the paper. CG-VAE-S is a variant of the CG-VAE model, where the sparsity constraint is removed. We have now added a clear explanation of CG-VAE-S in the revised version of the paper to ensure that its introduction is properly aligned with its usage. In light of your question, we have added it in the Line 416 of the reversed version.

W6: For synthetic results, why not apply CG-GAN? Also, there are no other baseline methods for comparison.

A6: Thank you for your valuable feedback. As per your suggestion, we have included CG-GAN in our synthetic experiments as shown in Lines 390-395 and Lines 432-444 of the revised paper. Additionally, we have considered a wider range of baseline methods for comparison, including various disentangled representation learning approaches such as IMSDA [3], iVAE [4], $\beta$-VAE [5], FactorVAE [6], and SloW-VAE [7]. Experiment results are shown as follows:

[3] Kong, Lingjing, et al. "Partial disentanglement for domain adaptation." International conference on machine learning. PMLR, 2022.
[4] Khemakhem, Ilyes, et al. "Variational autoencoders and nonlinear ica: A unifying framework." International conference on artificial intelligence and statistics. PMLR, 2020.
[5] Higgins, Irina, et al. "beta-vae: Learning basic visual concepts with a constrained variational framework." ICLR (Poster) 3 (2017).
[6] Kim, Hyunjik, and Andriy Mnih. "Disentangling by factorising." International conference on machine learning. PMLR, 2018.
[7]Klindt, David, et al. "Towards nonlinear disentanglement in natural data with temporal sparse coding." arXiv preprint arXiv:2007.10930 (2020).

W7: It is also very common to compare the generated image with the true by MSE, $R^2$, or cross-entropy (log-likelihood). They are intuitive metrics that are worth seeing.

A7: Thank you for your thoughtful suggestion. Since metrics like MSE and $R^2$ rely on ground-truth latent variables for estimation, they are difficult to calculate in real-world tasks where these ground-truth variables are unknown. Therefore, we have included comparisons using MSE and $R^2$ as additional evaluation metrics on the synthetic experiments. Specifically, we computed MSE based on the approach outlined in [8], and $R^2$ following the method in [3]. Regarding your point about cross-entropy, since the latent variables in our model are continuous, cross-entropy is not an appropriate metric for evaluating disentanglement performance. Instead, we have adopted completeness, disentanglement score, and informativeness as disentanglement evaluation metrics [9]. Experiment results are shown as follows.

Dear qELo, thank you for taking the time and effort to review our paper. Your valuable suggestions have been instrumental in helping us improve the soundness of our experiments. We deeply appreciate your insightful feedback and thoughtful comments.

W1: The sentence of Theorem A3 is a bit confusing. It is a bit hard to read that long sentence.

A1: Thank you for pointing this out. To address this, we have revised the statement by breaking it into two shorter sentences, which we believe will make it clearer and easier to follow. Specifically, the revised A3 is:

A3 (Generalized Sufficient Changes for Subspace Identification): Let $\mathcal{x} _k$ be a subset of $\mathcal{x}$ and $\mathcal{x} _{k, (1)}, \mathcal{x} _{k, 0}$ be two different instance of $\mathcal{x} _k$. And vectors $w(k, \mathcal{u})-w(k,0)$ with $\mathcal{u}=1,\dots,|\text{Pa}(\mathcal{x} _k)|$ are linearly independent.

We hope this modification resolves your concern, and we appreciate your feedback for improving the readability of our work.

W2: Line 158, why $x_k$ contribute $z _j$? And the derivative $\frac{\partial z _j}{\partial x _k}$. Could the author explain why the relationship is not $z_j$ contribute $x _k$?

A2: Thanks for your question. We hope you will find the intuition in Line 78-79. Specifically, by using the sparsity constraint, we can model the ground truth mixing procedure. If $\hat{z}_j$ is not adjacent to $x _k$, i.e., $\hat{z} _j$ contribute $x _k$, then $\frac{\partial \mathcal{z} _{\text{Pa}(x _k)}}{\partial \hat{z} _j} \equiv 0$. In light of your question, we have highlighted this in Lines 169-170.

W3: Line 349, "$q(z|x)$ is the an encoder"

A3: Thanks for your careful review. When we refer to $q(\mathcal{z}|\mathcal{x})$ as 'an encoder', we mean that $q(\mathcal{z}|\mathcal{x})$ is implemented as an encoder neural architecture in practice. Specifically, $q(\mathcal{z}|\mathcal{x})$ is used to approximate the posterior distribution over the latent variable $\mathcal{z}$ given observed variables $\mathcal{x}$. The encoder parameterizes the posterior distribution through a mean and variance in a probabilistic framework of variational inference [1]. Therefore, $q(\mathcal{z}|\mathcal{x})$ serves as the practical tool for inferring $\mathcal{z}$ from $\mathcal{x}$. In light of your suggestion, we have modified it in Line 322 of the revised version.

[1] Kingma, Diederik P. "Auto-encoding variational bayes." arXiv preprint arXiv:1312.6114 (2013).

W4: Eq. 9, why loss $L_s$ is chosen to be the absolute sum rather than the squared sum? Will a squared sum be easier to train and more stable? Any reason or trade-off analysis on this choice? Or does absolute sum have sparsity?

A4: Thank you for your insightful question. The choice of the absolute sum for $L_s$, corresponding to the L1 norm in statistics [2], was deliberate. Specifically, the L1 penalty induces a "sharp" constraint, encouraging the penalized quantities (like the coefficients in linear regression or the partial derivatives in our formulation) to become exactly zero during optimization. This characteristic is particularly desirable in scenarios where we aim to simplify the model or identify the most relevant components in the mixing process. In the meanwhile, the L2 norm (squared sum) tends to stabilize training by uniformly penalizing the magnitude of all coefficients, it does not promote sparsity as effectively. Instead, it typically results in coefficients with small but non-zero values, which may not align with our objective in this context. Therefore, the L1-norm can be used to induce sparsity of the estimated mixing procedure, which aligns with the goal of our theoretical results. In light of your questions, we have replaced L2-norm, i.e., the squared sum, with L1-norm, and conducted experiments on the synthetic datasets, which are shown in the following tables.

In light of your question, we have provided the discussion and experiment results in Appendix E.2.

[2] Tibshirani, Robert. "Regression shrinkage and selection via the lasso." Journal of the Royal Statistical Society Series B: Statistical Methodology 58.1 (1996): 267-288.

W2.2(Q2): I mean my main concern is that, since disentangled representation learning has been studied for almost ten years now, there have been a lot of variants of VAEs that were developed for learning disentangled representations. So I wonder 1) whether their framework can be applied to those variants of VAEs, and 2) if it is possible to compare against those models, such as TC-VAE[3], FactorVAE[4], and HFVAE[5].

A2.2: Yes, you are right! Our method can surely be applied to other variants of VAE like TC-VAE, FactorVAE, and HFVAE. In light of your suggestion, we have extended our experiments to include these VAE variants and evaluated their performance using our framework on simulation datasets with different metrics mentioned in A1, which are shown as follows:

We have added these experiments in Appendix E.1. The results validate that our method can be effectively extended to different VAE variants.

Dear 4KGP, thank you for taking the time and effort to review our paper, which helps us improve the depth of our work. Your valuable suggestions allowed us to present a more comprehensive evaluation and reinforce the generality of our framework. We deeply appreciate your insightful feedback and thoughtful comments.

W1: While they validated their framework on multiple datasets, they only provided qualitative results in terms of the disentanglement of learned representations. I would hope that they can evaluate the results more quantitatively and make comparisons based on metrics of disentanglement, e.g. the metrics used in [1] and [2]. I guess the reason I want to see this is it is difficult to see the disentanglement only based on t-SNE plots, because t-SNE itself will employ an algorithm that performs dimension reduction into 2D space, so it is likely that this algorithm introduces a certain level of disentanglement.

A1: We appreciate your suggestion regarding the evaluation of disentanglement using quantitative metrics. In light of your suggestions, we have acknowledged the importance of quantitative evaluation for disentanglement and followed your recommendation to use metrics in [1]. Specifically, we have considered three other metrics: completeness, disentanglement score, and informativeness. Note that the bigger the values of completeness and disentangle score, the better the performance of disentanglement, and the smaller the values of informativeness, the better the performance of disentanglement. Since these metrics require the ground truth and estimated latent variables, we have conducted experiments on the synthetic datasets and compared the iMSDA [2] as follows:

The results show that our method achieves good performance even when the number of domains is small, which verifies our theoretical results. In light of your suggestions, we have added the experiment results of completeness and the disentanglement score in Lines 432-444. And the experiment results of informativeness can be found in Lines 1250-1255.

[1] Eastwood, Cian, and Christopher KI Williams. "A framework for the quantitative evaluation of disentangled representations." 6th International Conference on Learning Representations. 2018.
[2] Li, Zijian, et al. "Subspace identification for multi-source domain adaptation." Advances in Neural Information Processing Systems 36 (2024)

W2.1(Q1): What is the reason that the authors choose VAEs and GANs as the two specific types of models that they want to implement the framework with?

A2.1*: This is a great observation. Although the proposed framework can be implemented by both VAEs and GANs, we learned from the results that they can facilitate different tasks. For instance, in order to discover learned factors, the latent representation in VAE can be immediately used. On the other hand, for the purpose of multi-domain image generation, where high-quality sample generation is critical, GAN is a particularly well-suited choice.

In light of your questions, we have considered the GC-GAN and GC-VAE in the synthetic experiments, which are shown in Lines 390-395 and Lines 432-444. For real-world experiments, the results of these models can be found in Lines 445-450.

Dear Reviewer 4KGP,

We sincerely appreciate your taking the time to review our manuscript and providing us with your insightful questions and suggestions. As the rebuttal discussion period is limited, we eagerly await any additional feedback you may have. We are more than willing to have further discussions with you.

Best regards,

Authors of submission 4906

W4(Q2): Can you provide some intuition why Sufficient Changes for Component-wise Identification require the second derivative assumption, while this is not the case for the subspace Identification? This could help readers better understand the theoretical foundations of the proposed method.

A4: Thanks for the great question! Subspace identification is weaker than component-wise identification, and hence naturally requires less information in the data generation process and distributions. Let us start with the subspace identifiability, suppose we have two sets of latent variables with changing distributions denoted by $\mathcal{z} _s$ and $\mathcal{z} _c$, respectively. And $\hat{\mathcal{z}} _s$ and $\hat{\mathcal{z}} _c$ are the corresponding estimated variables. Moreover, we let the $i$- and $j$- component of $\mathcal{z} _s$ and $\mathcal{z} _c$ be $z _s^i$ and $z _c^j$, respectively. In order to see whether $\mathcal{z} _s$ is identifiable up to a subspace, we just need to show whether $\frac{\partial z _{s}^i}{\partial \hat{z} _c^j}=0$. To this end, only the first-order derivative of the logarithmic density distribution is needed. However, for the purpose of component-wise identifiability, we have to make additional conditions, such as conditional independence between $z _s^{i}$ and $z_s^{j}$ given all the remaining components. These conditions impose stronger constraints compared to subspace identifiability. Mathematically, this conditional independence implies that the second-order cross derivative of the logarithmic density distribution with respect to $\hat{z} _{i}$ and $\hat{z} _{j}$ is zero, which involves second-order derivatives. In light of your treasured question, we have provided this intuition in Appendix D.2.

Dear Reviewer Fw6r, thank you for your insightful and constructive review. Your comments have significantly helped us refine the clarity and readability of our work as well as the soundness of our experiments. We greatly appreciate the time and effort you dedicated to reviewing our submission.

W1: The notations are heavy and hard to follow. Please provide a comprehensive notation table at the beginning of the paper, and explicitly define each symbol when it's first introduced. For example,

• In equation (2), what is the meaning of two subscripts of $\mathbf{x}_{k}^{(1)}$?
• Other notation also needs to be defined carefully, $\backslash k$ (all index but $k$)?
• In equation (6), it seems there is a missing integral (the second integral has been omitted). There are English typos and errors, e.g., line 157 (contributions --> contribute).

A1: We appreciate your careful review and very helpful feedback. Below, we provided the point-to-point response to the comment and we have updated the paper accordingly.

• We denote by $\mathcal{x} \in \mathbb{R}^n$ a vector of latent variables with the dimensionality of $n$. The subscript $k$ refers to a specific index or position of $\mathcal{x}$, indicating that $\mathcal{x} _k$ is the $k$-th dimension of $\mathcal{x}$. Importantly, $\mathcal{x}$ is a random vector, and each $\mathcal{x} _k$ is a scalar random variable corresponding to one dimension of $\mathcal{x}$. The subscript $(1)$ indicates a particular instance or realization of the random variable $\mathcal{x} _{k}$. Thus, $\mathcal{x} _{k, (1)}$ represents the first instance or realization of the random variable $\mathcal{x} _k$. Thanks for your question, we have provided a notation description on Page 2.

• You are right. $\mathcal{x} _{\backslash k}$ denotes the random vector $\mathcal{x}$ with the $k$-th component excluded.

• Thanks for pointing it out. To avoid confusion, we have added the integral in Equation (6), which can be found in Line 318.

• Typos: Thanks for your careful review. We have gone through the whole paper and corrected the typos.

W2: In synthetic experiments, there is no comparison with any nonlinear ICA methods. Comparing the CG-VAE, and CG-GAN with one or two relevant nonlinear ICA methods can be beneficial for readers to understand the advantages of incorporating domain encoding network and sparse mixing constraints in the disentangled representation.

A2: Thanks for your valuable comment, which helps make the experiments more solid. In light of your suggestion, we have considered two relevant nonlinear ICA baselines: iMSDA [1] and iVAE [2], which leverage domain index as auxiliary variables for identifiability. Experiment results can be found in Lines 390-395 of the revised version.

Specifically, we generate synthetic data following the mixing process defined in Equation (1) and illustrated in Figure 7(a). Since our proposed methods and the baselines are based on variational autoencoder (VAE) structures, we ensure fairness by employing the same encoder and decoder architectures for all methods. Additionally, we use identical hyperparameters, such as the learning rate and the number of training epochs. We consider scenarios with 2, 4, 6, and 8 different domains. The experimental results are presented in the following table:

Experimental results demonstrate that when the number of domains is relatively small, our method achieves significantly higher MCC, which validates the theoretical results of our paper.

[1] Kong, Lingjing, et al. "Partial disentanglement for domain adaptation." International conference on machine learning. PMLR, 2022.
[2] Khemakhem, Ilyes, et al. "Variational autoencoders and nonlinear ica: A unifying framework." International conference on artificial intelligence and statistics. PMLR, 2020.

W3(Q1): In line 220, the index in $z_{\mathbf{o}_1}$ should be $\mathbf{o}_1$?

A3: Thank you for your careful review. We have corrected the index in $z_{\mathbf{o}_1}$ to $\mathbf{o}_1$.

Dear reviewer Fw6r,

Thanks for the time you dedicated to carefully reviewing this paper. It would be highly appreciated if you let us know whether our responses properly address your concerns, despite your busy schedule. Thanks a lot!

Best regards,

Authors of submission 4906