Disentanglement Learning via Topology

Thank you for your time and the thorough review. We will improve the presentation according to the suggestions. Below we address specific concerns one by one.

W1: The paper lacks a clear reasonable explanation as to why topological constraints are meaningful/effective for disentanglement representation learning.

A: Thank you for the feedback. We are adding a clarification deducing the smallness of RTD from the disentanglement conditions and topological properties of symmetry action. Here is the principal explanation. The Lie group(oid) symmetry $G$ action on the support of data distribution is continuous and invertible. This implies that for any subset of the support of data distribution, the image of the subset under symmetry $g\in G$ has the same homology or the same group of topological features. The preservation of topological features at multiple scales can be tested with the help of the representation topology divergence (RTD). If RTD is small between a sample from $X$ and its symmetry shift, then the groups of topological features at multiple scales under such symmetry $G$ action are preserved.

Also the smallness of RTD implies the smallness of the disentanglement topological measure from (Zhou et al 2021) based on the geometry scores of data subsets conditioned to a fixed value of a latent code. Such subsets for different fixed values of the latent code are also related via the symmetry shift action, and if RTD between them is small, then the distance between their persistence diagrams and hence the metric from loc cit is small as well.

W2: The new loss function was already proposed in a 2022 ICML paper [a]. The main contribution of this work is applying it to disentanglement, making the explanation of the above issue crucial for this paper.

A: Thank you for the remark. As we mentioned in Appendix, the paper [a] describes briefly an application of topological metric to evaluation of interpretable directions in a simple synthetic dataset. They compare topological dissimilarity in data submanifolds corresponding to slices in the latent space conditioned by a fixed latent code value. In our work, we propose the new differentiable loss function TopDis (Eqn. 3), which measures the topological discrepancy, as evaluated by RTD, between an arbitrary data sample from decoder and its symmetry shift obtained via group(oid) action shifts preserving the Gaussian distribution.

W3: The experiments focus on models with some disentanglement capabilities, but the effectiveness of this regularization term on vanilla VAEs has not been studied.

A: Thank you for the valuable feedback. Here are the results for VAE+TopDis. As these results demonstrate, VAE+TopDis outperformed VAE as measured by the 4 disentanglement metrics.

We had also described the experiments for VAE vs VAE+TopDis-C models in Appendix in Table 6.

(cont'd below)

Thank you for your valuable feedback.

1. In [1], the topological metrics is measured between samples whose latent codes necessarily belong to a hyperplane conditioned on the fixed value of one of $z_i$. During the training, the latent codes of data points obtained from the encoder never lie on a hyperplane, so this approach is not applicable for the optimization of the topological loss term. Instead, we work with an arbitrary data points sample obtained from the encoder on which acts the symmetry group(oid) shift.
2. For DCI calculation, we used the evaluation code from the commonly used disentanglement lib: https://github.com/google-research/disentanglement_lib . We note that we trained the models for 1 million iterations compared to 300k iterations used in several other works which can be a possible source of disagreement. Also, in most recent papers only more advanced methods were compared but not the vanilla VAE baseline.
3. We have added this and other improvements to the main text, the additions are highlighted by yellow in the revision. We are also adding, as a byproduct, the proposition clarifying the relation between the two approaches to the definition of disentanglement, the factor-independence based and the symmetry based.

Concluding remark. Please respond to our post to let us know if the clarifications above suitably address your concerns about our work. We are happy to address any remaining points; if the responses above are sufficient, we kindly ask that you consider raising your score.

W3: The evaluation of how the proposed methods handle the tradeoff between disentanglement and reconstruction is limited. Besides Table 4 and Table 8, the authors are encouraged to report the reconstruction errors of the proposed method with and without "gradient orthogonalization" for a complete comparison with the baselines. Did the "gradient orthogonalization" apply to the baselines as well?

A: Thank you for your question and remark. Figure 17 in Appendix illustrates the positive effect of gradient orthogonalization on reconstruction error during training. We do not apply gradient orthogonalization in each experiment but only when the proposed loss can slightly hinder the reconstruction objective.. In our experiments, we orthogonalize the gradient of the proposed TopDis loss w.r.t. the reconstruction loss. Since there is a known tradeoff between reconstruction quality and disentanglement, our primary goal is to develop an additional loss that would enhance the underlying base model but would not result in increase of reconstruction error. While in our experiments we do not apply gradient orthogonalization separately for the baseline models, it is possible to apply the gradient orthogonalization of disentanglement-promoting loss (e.g. Total Correlation for FactorVAE, additional KL-divergence for BetaVAE, etc.) for the baseline model. While a similar technique is known, to the best of our knowledge, it has not been used in the context of learning disentangled representations before.

Q2: How the γ should be selected for different VAE-based methods? Does TopDis improve disentanglement when β is already very large?

A: Thank you for your questions.

1. In our experiments, we used the following greedy procedure to tune the weight $\gamma$ of the TopDis loss. First, we tune hyperparameters of baseline approaches (i.e. beta-VAE, FactorVAE, ControlVAE, etc.), by selecting ones having both high disentanglement quality and reasonable reconstruction error. A minor exception is the DAVA model, where the hyperparameters are tuned adaptively during training. Then, we select the best $\gamma$ from the range [1, 6] by the disentanglement quality. We observe that optimal $\gamma$ often coincide for different datasets.

2. High values of $\beta$ typically lead to high reconstruction error, we do not explore this setup in our experiments. Our goal is to enhance the underlying model to provide both good disentanglement and reconstruction quality.

Concluding remarks. Please respond to our post to let us know if the clarifications above suitably address your concerns about our work. We are happy to address any remaining points during the discussion phase; if the responses above are sufficient, we kindly ask that you consider raising your score.

W1: It is unclear how the hyper-parameters in Eqn (4) affect the performance. There are γ_1 and γ_2 in Table 9 (appendix N), but there is only one γ in Eqn (4).

A: Thank you for your remark. The sensitivity w.r.t. to $\gamma$ in Eqn (4) is provided in Appendix Q. In Appendix Q, $\gamma_1$ denotes the weight for Total Correlation loss from the FactorVAE model while $\gamma_2$ denotes the weight for TopDis loss from the equation (4). We have added the necessary clarification to the Appendix Q and renamed $\gamma_1$ to $\gamma_{TC}$ (stands for Total Correlation) and $\gamma_2$ to $\gamma_{TD}$ (stands for TopDis) as more suitable ones.

W2: In Table 1, vanilla VAE performed better than some other methods (e.g. FactorVAE on 3d Shapes, and β-TCVAE on MPI3D, etc), evaluations of VAE+TopDis and β-TCVAE+TopDis are not included in Table 1.

A: Thank you for your remarks.

1. We provide evaluations of β-TCVAE+TopDis and we have updated Table 1 accordingly. Addition of TopDis improves β-TCVAE as measured by the 4 disentanglement metrics.

2. Also, we are running experiments with VAE+TopDis, please find the results below. We see that VAE+TopDis outperformed VAE as measured by the 4 disentanglement metrics.

3. Indeed, vanilla VAE performed better than some other methods in some cases. This is consistent with actual papers: superiority of Vanilla VAE w.r.t beta-VAE for MPI3D was also observed by Locatello et al, 2020 (see Figure 7, bottom row). We have decided to double-check them and will report results in Appendix, as in most recent papers only more advanced methods were used but not the vanilla VAE baseline. Also we trained the models for 1 million iterations compared to 300 000 iterations used in several other works which can be a possible source of disagreement.

We would like to emphasize that our main objective was to improve the SOTA methods, and that is what we achieved. The TopDis loss have consistently improved the performance of β-VAE, β-TCVAE, ControlVAE, FactorVAE and DAVA models in terms of disentanglement scores (MIG, FactorVAE score, SAP score, DCI disentanglement score) while preserving the reconstruction quality.

Q1: My main concern is the relation between the proposed TopDis and disentanglement, is there any theoretical guarantee or deduction?

A: Thank you for the question. (See the answer to W1.) Firstly, we add the proposition establishing the relation of the probability based definition of disentanglement and the group based definition. Secondly, we add the clarification deducing the TopDis minimization from the disentanglement conditions and properties of symmetry groupoid action, see the answer to W1.

Q2: The authors applied the proposed TopDis to infer disentangled directions in a pretrained style-GAN, is there some quantitative results? Then dose the method can be applied to other disentangled methods?

A: Comparison of methods dedicated to the unsupervised discovery of disentangled directions in StyleGAN is qualitative since the FFHQ dataset doesn't have labels. Our goal is to demonstrate the applicability of the TopDis loss for this problem. See section 5.3 for details.

Concluding remarks. Please respond to our post to let us know if the clarifications above suitably address your concerns about our work. We are happy to address any remaining points during the discussion phase; if the responses above are sufficient, we kindly ask that you consider raising your score.

Thank you for your time and thorough review. We will improve the presentation according to the suggestions. Below we address specific concerns one by one.

W1: The relation between the constrain on latent space and disentanglement is still unclear, the TopDis is based on VAE-framework, which is based on Probability, and the paper referred to the definition of disentanglement based Group. And the paper failed to connect the above two framework,and making the proposed TopDis only kind of an intuitive necessary condition,as shown in Figure3.
A: Thank you for the valuable feedback.
1.We are adding a clarification on the relation of the probability based definition of disentanglement and the group based definition. Let $q(z)$ be the aggregate posterior distribution over a latent space, aggregated over the whole dataset $X$. And let $q(z_i)$ be the similar aggregate distribution over the latent code $z_i$. The formula (1) is valid and defines symmetry group(oid) shifts if we replace the standard normal distribution by any distribution over the real line, we use it with the distribution $q(z_i)$ over the $i-$th latent codes.

Proposition. a) If the distribution $q(z)$ is factorized into product $q(z)=\prod_i q(z_i)$, then the shift defined by the formula (1) and acting on a single latent code $z_i$ and leaving other latent codes fixed, preserves the latent space distribution $q(z)$. This defines the $G_i$ groupoid action on $z$ for any $i$, whose action is then extended to points of the initial dataset $X$ with the help of the decoder-encoder. b) Conversely, if $q(z)$ is preserved for any $i$ by the shifts acting on $z_i$ and defined via formula (1) from the distribution $q(z_i),$ then $q(z)=\prod_i q(z_i)$.
Proof. a) The shift defined by (1) for the distribution $q(z_i)$ acting on the latent space, preserves also any $q(z_j)$ for $j\neq i$. b) The result follows from the case of an arbitrary distribution over a pair of random variables $z_1, z_2$. For two variables, it follows from the Bayes formula that the shifts of $z_1$ preserve the conditional $q(z_2\vert z_1)$. Since the group(oid) action is transitive it follows that the conditional does not depend on $z_1$, and hence $q(z_1,z_2)=q(z_1)q(z_2)$.

To the best of our knowledge, we are the first to impose the preservation of the distributions under the symmetry groupoid action condition in the standard normal distribution variational autoencoder framework. On the one hand, this implies that such symmetry action is necessarily defined not by a group but by the groupoid and also using the specific shifts as in equation (1). On the other hand, this distribution preservation by groupoid is the main ingredient making the two definitions of disentanglement compatible. We are adding this result clarifying the correspondence between the two definitions to the paper.

2.We are adding a clarification deducing the smallness of RTD from the disentanglement conditions and properties of symmetry action on manifolds.Briefly, the Lie group(oid) symmetry $g$ action on the support of data distribution is continuous and invertible. This implies that for any subset of the support of data distribution, the image of the subset under $g$ has the same homology or the same group of topological features. The preservation of topological features at multiple scales can be tested with the help of the representation topology divergence (RTD). If RTD is small between a sample from $X$ and its symmetry shift, then the groups of topological features at multiple scales are preserved.

Also the smallness of RTD implies the smallness of the disentanglement measure from (Zhou et al. 2020) based on the geometry scores of data subsets conditioned to a fixed value of a latent code. Such subsets for different fixed values of the latent code are also related via the symmetry shift action, and if RTD between them is small, the distance between their persistence diagrams and hence the metric from loc cit is small as well.

W2: From Appendix L, the best performance hyperparameters are quite different across different methods and different datasets, is there any guidance or criterion to choose the hyper-parameter?

A: In our experiments, we used the following greedy procedure to tune the weight $\gamma$ of the TopDis loss. First, we tune hyperparameters of baseline approaches (i.e. beta-VAE, FactorVAE, ControlVAE, etc.), by selecting ones having the high disentanglement quality while keeping reasonable reconstruction error. A minor exception is the DAVA model, where the hyperparameters are tuned adaptively during training. Then, we select the best $\gamma$ weight for TopDis from the range [1, 6] by the disentanglement quality. We observe that optimal $\gamma$ often coincide for different datasets.

(cont'd below)

Thank you for your time and thorough review highlighting strengths of our paper. We will improve the presentation according to the suggestions. Below we address specific concerns one by one.

W1: We suggest the authors to “move” the gradient orthogonalization part to the main paper.
A: Thank you for the suggestion, we have moved the details on gradient orthogonalization to the main part.

W2: Move more details on formal definition of RTD to the main paper
A: Thank you for the suggestion. We are adding more details on the formal definition to the discussion of the RTD definition in Section 3.1.

W3: Clarify how $G$ can be applied in the pixel space.
A: Thank you for your question. The equation $f(g(z))=g(f(z))$ concerns the outcome of the learning process, which consists, in particular, in finding a priori unknown symmetries in the data. Given the decoder map $f$, this equation defines the symmetry action on $X$ preserving the data distribution.

W4: Clarify notations \gamma_1 and \gamma_2 in the ablation table in Appendix Q.
A: Thank you for your remark. In Appendix Q, \gamma_1 denotes the weight for Total Correlation loss from the FactorVAE model while \gamma_2 denotes the weight for TopDis loss from the equation (4). We have added the necessary clarification to the Appendix Q and renamed \gamma_1 to \gamma_TC (stands for Total Correlation) and \gamma_2 to \gamma_TD (stands for TopDis) as more suitable ones.

W5: Can RTD be applied in latent space instead of pixel space?
A: Thank you for your question. In the footnote on page 5, we mean that it is possible to compute TopDis loss either between the images (i.e. in pixel space) or between their representations (for example, produced by another pretrained model). It can be beneficial for different reasons to compare the multiscale topology of data in a representation space. For example, common disentanglement datasets have typical image resolution 64x64. However, in case of significantly higher image resolution, it could be beneficial to utilize representations instead of images. As a proof of concept, in the subsection 5.3 and Appendix H, we provide the results on unsupervised discovery of disentangled directions in StyleGAN where we compute the RTD in the representation space instead of pixel space. With these experiments, we illustrate that RTD finds meaningful directions using the representation space as well.

W6: Clarification on VAE+TopDis
A: Thank you for the question. We are now running the experiments for VAE+TopDis. Here are the results for VAE+TopDis vs VAE :

The reconstruction loss for VAE+TopDis: 9.54±0.19 (dSprites), 3489.533±1.502 (3DShapes), 1376.218±0.316 (3DFaces)

As we see, VAE+TopDis outperformed VAE as measured by the 4 disentanglement metrics.

In the experiments in Table 6 in Appendix G the training procedure was identical to that of Table 1 in the main paper, except that we have used a slightly smaller architecture in Table 6 for dSprites for both VAE and VAE+TopDis-C compared with the experiments from Table 1. For this reason, the metrics values differed in Table 1 and Table 6 for VAE at dSprites dataset. However, we highlight that the experimental setup for 3D Shapes is identical in both Table 1 and Table 6. And that our setup is always consistent when we compare a base model with and without TopDis loss.

Also, our main objective was to improve the SOTA methods, and that is what we achieved. The TopDis loss have consistently improved the performance of β-VAE, β-TCVAE, ControlVAE, FactorVAE and DAVA models in terms of disentanglement scores (MIG, FactorVAE score, SAP score, DCI disentanglement score) while preserving the reconstruction quality.

Concluding remarks. Please respond to our post to let us know if the clarifications above suitably address your concerns about our work. We are happy to address any remaining points during the discussion phase; if the responses above are sufficient, we kindly ask that you consider raising your score.