A Revisit of Total Correlation in Disentangled Variational Auto-Encoder with Partial Disentanglement

Dear Reviewer 93Kr,

Thank you very much for your time and valuable comments on our paper. We highly appreciate your recognition of the strengths of our paper. Hopefully, the following responses could resolve most of your concerns and answer your questions.

1.1 "The core idea inspired by ICA is that non-Gaussian is independent", This statement seems oversimplified.

Please kindly notice that, this sentence is not originally stated by us, but cited from the FastICA paper by Hyvarinen and Oja. This sentence is exactly the title of their Sec. 4.1, which illustrates the reason. The intuition is also illustrated in Fig. 7, and the FastICA algorithm is based on this idea, which is explained in Sec. 4 of their paper.

1.2 "if the true number of disentangled latent components is two but we instruct the logcosh-priored VAE to find three, it will yield three components with poor disentanglement instead of finding two disentangled components and one non-informative component" Is there empirical evidence or theoretical justification for this claim about logcosh-priored VAE behavior? If so, could you provide it?

Please kindly notice that the empirical result showing this issue was already in Fig. 6 of our initial submission. Specifically, there are three fully independent latent components but ICA is asked to find six. The orange curves show the poor latent estimated by ICA. However, the green and red curves show that $\beta$-TCAVE and our PDisVAE is able to find three fully independent latent components and three dummy latent components, implying that penalizing TC/PC is a more flexible way for latent disentanglement than assuming a non-Gaussian prior (ICA). Although there is no theoretical proof on this point, we do often observe ICA finish with "pseudo" independent components, which are not really independent after a detailed checking.

2 Important recent literature is overlooked. After a quick search I found:

Thanks for reminding us of these important papers. We apologize that there should have been a section on related works. To answer similar questions asked by other reviewers, we have added a section of related works in Appendix A.5 as an early response.

3.1 Provide CelebA results for $\beta$-TCVAE and FactorVAE, or explain why these were not included.

Please kindly notice that the empirical results showing this issue were already in Fig. 9 and Fig. 13 in our initial submission. Sec. 4.4, paragraph 1, the last sentence, claims that "when $G=K$, PDisVAE reduces to the fully disentangled VAE, e.g., $\beta$-TCVAE or FactorVAE". So, the results are in the $G=12$ groups setting in Fig. 9 and Fig. 13. This is in fact an important point that our PDisVAE is flexible enough to generalize to standard VAE and fully disentangled VAE with particular group configurations.

3.2-3.4 regarding hyperparameters.

In real-world datasets, we don't know the true number of independent groups and each group's rank. So, we tried a series of group configurations. Different group configurations have different disentanglement interpretations as shown in Fig. 9. It should not be assumed as a hyperparameter, but a flexibility of the method. With higher group rank, the latent will be more expressive but the disentanglement will be weaker. The hyperparameter of PDisVAE is $\beta$, which was already discussed in Fig. 4 of our initial submission. The optimal choice of $\beta$ is similar to the findings in $\beta$-TCVAE, which is $\beta\in(2, 10)$, and for a fair comparison, we choose $\beta=4$ for our PDisVAE and $\beta$-TCVAE.

4.1 Testing the method with increasing latent dimensions (e.g., 32, 64, 128) and reporting performance trends. 4.2 Analyzing the impact of different grouping strategies on performance for a fixed latent dimension. For example, comparing random groupings vs. learned groupings.

Please kindly notice that in real-world experiments of Sec. 4.4, we did exactly this ablation study. The results are reported in Fig. 9, Fig. 10 and the group reconstruction videos were in the supplementary material of our initial submission.

Thank you for your recognition of our revision and the suggestion. To answer similar comparison questions asked by other reviewers, we have added a comprehensive table comparison in Appendix A.6 to show the superiority of our PDisVAE. Some metrics and interpretations were already presented in Fig. 7 and Fig. 8 of our initial submission. For your convenience, we show the comparison results here:

The table below presents the partial correlation (PC), latent $R^2$, MSE, and mutual information gap (MIG) evaluated for the partial disentanglement (group-wise independence) in the estimated latent.

ISA and $\alpha$-TCVAE are the two new alternative methods. Since only ISA and our PDisVAE explicitly require group-wise independence rather than strong full independence, only ISA and our PDisVAE obtain the lowest PC. Compared with ISA, PDisVAE is more flexible since we achieve partial disentanglement by adding a PC penalty term to the loss function. Via the PC penalty, the latent distribution is estimated through the aggregated posterior $q(\boldsymbol z)$ from the learned decoder, rather than a fixed $L^p$-nested prior in ISA. For $\alpha$-TCVAE, it is classified as a fully disentangled method but our pdsprites dataset contains group-wise independent latent, therefore, PDisVAE obtains more accurate and partially disentangled latent when evaluated with the true latent (labels).

Dear Reviewer 1ysP,

Thank you very much for your time and valuable comments on our paper. We highly appreciate your recognition of the strengths of our paper. Hopefully, the following responses could resolve most of your concerns and answer your questions.

Structure of the Paper

We apologize for the clarity and the use of vspace. We will reorganize our draft once we finish it and send you a notification.

Notation and Clarity Issues

Please kindly notice that

• $N$ is the number of data points (samples) was already introduced in the first sentence of Sec. 2.1 of our initial submission "Given a dataset of observations $\\{x^{(n)}\\}_{n=1}^N$ consisting of $N$ samples..."
• $n_*$ refers to a specific data point was already clarified in line 206 (Sec. 2.4, paragraph 2, sentence 1) of our initial submission "Intuitively, when we only have a batch $\mathcal B_M \subsetneqq \\{1,\dots,N\\}$ and a sampled $z\sim q(z|n_*)$, where $n_*$ is a specific example point in $\mathcal B_M$..."
• We kindly think that the batch notation here is necessary for solid and rigorous math clarification.

Context of Literature and Definition of Disentanglement and Question 2

• Thanks for this point. We would like to claim at the beginning of the paper that disentanglement cannot be plainly achieved by naive VAE.
• We have added a section of related works to Appendix A.5. We will reorganize our draft once we finish it and send you a notification.
• As stated in texts below Eq. 5, Eq. 2 of $\beta$-TCVAE, and Wikipedia, total correlation (TC) is a natural way of requiring independence from the statistical definition of independence. Our partial correlation (PC) is merely a natural generalization from full independence to group-wise independence. In other words, minimizing PC is an intuitive and explicit way of reducing independence between latent groups from the statistical definition of independence, i.e., Eq. 4 of our initial submission.
• To answer similar comparison questions asked by other reviewers, we have added a comprehensive table comparison in Appendix A.6.

Clarification on Importance Sampling (IS) Approach and Question 4

Thanks for this point. Please kindly notice that the rigorous derivation of the claims in Tab. 1 was already linked to Appendix 3 in our initial submission. To show the superiority of our IS batch approximation method, comparison results have been added to Appendix 3.3. The intuition behind this contribution is that we need a good estimation of TC/PC terms to achieve independence. Without a stable and accurate estimation of TC/PC, it would be harder to achieve the desired independence penalized by TC/PC.

Metrics for Assessing Disentanglement and Question 1

Thanks for this question. For synthetic datasets, we know the true disentangled (independent) latents (labels), so we compute the $R^2$ metric between the estimated latents and the true as a supervised measurement. The pairwise $t$-test of not only the supervised latent $R^2$ but also the unsupervised partial correlation (PC) in Fig. 2(b), and Fig. 5(b) rigorously show significantly better results from our PDisVAE.

Since MIG is a commonly used metric for full disentanglement, we have tried to adapt it to evaluate partial disentanglement (group-wise independence) and presented a table including both $R^2$ and MIG in Appendix A.6.

Concerns with Theorem 1 and Its Proof

Thanks for this point. We realized that we should add a parenthesis in Theorem 1 to eliminate ambiguity, i.e.,

$(x_1, \dots, x_I) \perp (y_1, \dots, y_J) \iff \big(f(x_1,\dots, x_I) \perp g(y_1, \dots, y_J)$ $\forall$ functions $f$ and $g\big)$

The $\impliedby$ is stated as for all functions $f$ and $g$, i.e., if arbitrary choice of $f$ and $g$ makes $f(\boldsymbol x) \perp g(\boldsymbol y)$, then $\boldsymbol x\perp \boldsymbol y$. Since arbitrary choice of $f$ and $g$ holds, then $f = g = \operatorname{identity}$ necessarily holds, i.e., $\boldsymbol x\perp \boldsymbol y$.

Question 3 about group size

The setting of group size is an open question similar to the setting of latent dimensionality.

• Synthetic dataset 2 shows that PDisVAE can find a dummy latent if the true group rank is less than the specified group rank.
• With larger latent dimensionality, we are less likely to omit existed independent group.

Therefore, in the CelebA dataset, for example, we show that having a group rank greater than 1 (group independence) rather than group rank equal to 1 (full independence) can provide more interpretable disentanglement, especially Fig. 9(b). This just indicates the significance of partial disentanglement (group rank higher than 1).

Similar to the full disentanglement models like $\beta$-TCVAE and even all VAEs, it is meaningless to have a huge latent space and this can also hurt the learning effects.

• Regarding Theorem 1. Yes, you are right. The backward is only to prove $(x_1,\dots, x_I) \perp (y_1,\dots, y_J)$, not to prove they can be factorized. If $f(x_1, \dots, x_l) \perp g(x_1, \dots, y_J)$, certainly $(x_1,\dots, x_I) \perp g(x_1, \dots, x_J)$, and that's all we want. Please kindly notice that we do not intend to prove $x_1\perp \dots \perp x_I$ or $y_1 \perp \dots \perp y_J$ in Theorem 1.

• Yes, purely using the original MIG does not effectively assess the concept of group-wise independence, but only single dimensional-wise independence. This is also an important claim in our paper, as stated in Sec. 1 paragraph 3, and Fig. 9(b). To explain the latent $R^2$ measurement in detail, think we generate the data $\boldsymbol x$ using $(z_1, z_2) \perp (z_3, z_4)$, and the algorithm estimated $\hat z_1, \hat z_2, \hat z_3, \hat z_4$. Then, a correct algorithm should be able to find

  • either $(\hat z_1, \hat z_2)$ aligned with $(z_1, z_2)$ and $(\hat z_3, \hat z_4)$ aligned with $(z_3, z_4)$
  • or $(\hat z_1, \hat z_2)$ aligned with $(z_3, z_4)$ and $(\hat z_3, \hat z_4)$ aligned with $(z_1, z_2)$

  through affine (linear: rotation, scale, and translation) transformations, and we pick the maximum one as the $R^2$. If an algorithm fails to find this group-wise independence, there will be no appropriate alignment and the resulting latent $R^2$ will be lower. For example, a fully disentangled VAE like $\beta$-TCVAE cannot find group-wise independence, and hence the above two choices all result in bad alignment $R^2$, which finally results in a low latent $R^2$ measurement.
  Except for those supervised metrics, we also have the partial correlation (PC) metric, which directly measures whether $(\hat z_1, \hat z_2) \perp (\hat z_3, \hat z_4)$ from the statistical definition of group-wise independence. See, texts below Eq. 1 and Eq. 5 of our paper. Also please kindly notice that total/partial correlation is just the mutual information, see the Wikipedia of mutual information.

• The derivation in Appendix 1.3 is very similar to the derivation of the ELBO decomposition in $\beta$-TCVAE, i.e., Eq. 2 and Appendix C in $\beta$-TCVAE. This derivation is for building relationships between $\beta$-TCVAE and our PDisVAE, showing that our PDisVAE is a generalization of their $\beta$-TCVAE, in terms of the ELBO decomposition.

  • To answer your question, "math" means that line of equation is in the form of KL divergence to explicitly express independence/disentanglement. "code" means how people implement those terms in the code.

Dear Reviewer xQT7,

Thank you very much for your time and valuable comments on our paper. We highly appreciate your recognition of the strengths of our paper. Hopefully, the following responses could resolve most of your concerns and answer your questions.

Related works

Thanks for reminding us. We apologize that there should have been a section of related works. To answer similar questions asked by other reviewers, we have added a section of related works in Appendix A.5 as an early response. We are working on running necessary comparison experiments. We will reorganize our draft once we finish it and send you a notification.

Numerical comparison

Please kindly notice that rigorous statistical comparisons were already in Fig. 2(b), Fig. 5(b), and Fig. 8 of our initial submission.

Specifically, when we know the true latents (labels), so we compute the $R^2$ metric between the estimated latents and the true as a supervised measurement. The pairwise $t$-test of not only the supervised latent $R^2$ but also the unsupervised partial correlation (PC) in Fig. 2(b), and Fig. 5(b) rigorously show significantly better results from our PDisVAE.

Since MIG is a commonly used metric for full disentanglement, we have tried to adapt it to evaluate partial disentanglement (group-wise independence) and presented a table including both $R^2$ and MIG in Appendix A.6. Specifically, to answer similar comparison questions asked by other reviewers, we have added a comprehensive table comparison in Appendix A.6 to show the superiority of our PDisVAE. Some metrics and interpretations were already presented in Fig. 7 and Fig. 8 of our initial submission.

Dear Reviewer xQT7,

This totally makes sense. That means this rank-3 group mainly contains background color but also some other information (like some tiny facial features you observed). Please kindly notice that semantic meaning does not always correspond to an independent component or an independent group. Not only our PDisVAE but also fully disentangled VAEs like $\beta$-TCVAE and FactorVAE are essentially trying to find "statistically" independent groups/components, but an independent component/group might not have one semantic meaning and that depends on reasonable interpretation.

It is possible that a group contains more than one semantic meaning. For example, it is likely that females have more warm backgrounds and males have more cold backgrounds. In this case, the background warm/cold is entangled with gender. In this case, we cannot separate these two semantic meanings since they are statistically dependent/entangled.

In our example of background color, especially Fig. 9(b), we interpret that group as background color based on our human understanding. However, we cannot rigorously prove that the background color is totally independent of the tiny facial feature changes you observed. This is actually an important point we want to stress in this paper, like in Sec. 1 paragraph 3, Fig. 7(a), and Fig. 9(b). We can summarize the following four possibilities:

• one semantic meaning corresponds to one latent component (fully independent)
• one semantic meaning corresponds to several entangled latent components (a latent group)
• several semantic meanings correspond to one latent component (semantic meanings are entangled and encoded by one latent component)
• several semantic meanings correspond to several latent components (semantic meanings are entangled and encoded by several latent components)

This is the key reason we generalize fully disentangled VAE to partially disentangled VAE (PDisVAE) since PDisVAE considers all these possibilities that exist in nearly all real-world datasets (maybe with the probability of 1). We view this as our paper's key take-home message that we really need to jump out of the stereotype that one latent component should correspond to one semantic meaning.

For example, in the partial dsprites (pdsprites) dataset shown in Fig. 7(a), although we humans think $x$ location and $y$ location are two separable semantic meanings, they are statistically dependent/entangled with each other, so we cannot separate them but put them in one group, and that is why fully disentangled VAEs (e.g., $\beta$-TCVAE) fails with this dataset (Fig. 7(b)). You can think $x$ and $y$ as two semantic meanings or say $(x, y)$ "location" is one semantic meaning, but the ground truth is that $x$ location and $y$ location are entangled, not statistically separable, and hence should be encoded by a latent group of at least rank-2.

A similar reason also holds for the color distribution we plot in Fig. 9(b). If you use a fully disentangled VAE, you can only interpret that the background color (from red to blue, a curve in HSV space) is encoded by one latent component, but that might not be the fact. We do show in Fig. 9(b) that with more latent components entangled with each other as a group, the background color semantic meaning can be expressed more fully (a 2D manifold or a restricted 3D region that is not evenly distributed).

To answer your question again, let us give another example. In Fig. 1(c) of the $\beta$-TCVAE paper, you will find that they claimed those latent variations as gender, but background color changes. That means gender and background color (two semantic meanings) are very likely to be entangled with each other, and this is actually reasonable. No one can promise an absolutely perfect correspondence between semantic meaning(s) and a latent component/group. All researchers can do is validate the correctness of their method on synthetic dataset, like we do in Sec. 4.1 and Sec. 4.2, and to get more interpretable (but cannot promise perfect correspondence) disentanglement results on real-world datasets. Generally speaking, it is nearly impossible for all kinds of disentangling methods to find pure correspondence between a latent component/group and one semantic meaning on real-world datasets. At least there are some noises including other semantic meanings of tiny magnitude. This kind of result should be acceptable in the field of representational learning (disentanglement), especially on real-world datasets where there is no true latent. Otherwise, any interpretation from any method could have small flaws (that can even come from random seeds or the floating point precision of the training device).

If Reviewer xQT7 still has more questions on this point, we'd love to provide more detailed discussions on it.

Best, Authors

Dear Reviewer AfSM,

Thank you very much for your time and valuable comments on our paper. We highly appreciate your recognition of the strengths of our paper. Hopefully, the following responses could resolve most of your concerns and answer your questions.

Comparison with Stühmer, J., Turner, R., Nowozin, S. (2020). Independent subspace analysis for unsupervised learning of disentangled representations. In International Conference on Artificial Intelligence and Statistics (pp. 1200-1210). PMLR.

Thanks for reminding us of this paper. We appreciate this nice suggestion. We have added a section of related works to Appendix A.5. The comparison with the aforementioned paper and other methods has been added to Tab. 4 in Appendix A.6. We will reorganize our draft once we finish them and send you a notification. For your convenience, we show the comparison results here:

The table below presents the partial correlation (PC), latent $R^2$, MSE, and mutual information gap (MIG) evaluated for the partial disentanglement (group-wise independence) in the estimated latent.

Since only ISA and our PDisVAE explicitly require group-wise independence rather than strong full independence, only ISA and our PDisVAE obtain the lowest PC. Compared with ISA, PDisVAE is more flexible since we achieve partial disentanglement by adding a PC penalty term to the loss function. Via the PC penalty, the latent distribution is estimated through the aggregated posterior $q(\boldsymbol z)$ from the learned decoder, rather than a fixed $L^p$-nested prior in ISA. Therefore, PDisVAE obtains more accurate and partially disentangled latent when evaluated with the true latent (labels).

Although the ISA paper proposed the group-wise independence method through the $L^p$-nested distribution, they did not conduct experiments on synthetic datasets with truly partially disentangled latent. Their application on the dsprites dataset is also a fully disentangled dataset. Compared to their ISA method, therefore, we have comprehensive experiments on both synthetic and real-world datasets to validate and show the existence of partially disentangled latent in many disentangling problems and show the effectiveness of our method in finding those group-wise independent latents.

PDF takes long to render.

We have changed those large PDF figures to PNG figures and the issue was resolved.

Integrate into disentanglement library.

Due to the current time limit and unpublished situation of this draft, we plan and would like to incorporate this method into the disentanglement library in the future once it is accepted.