Unpicking Data at the Seams: VAEs, Disentanglement and Independent Components

Thank you once again for your review, we have improved the paper and look forward to any further feedback.

• Re [1]: we hope to have addressed your main concern in the general response. Our work offers an intuitive constructive proof of disentanglement starting from the ELBO through to a description of the factor distributions. We draw close analogy to the linear case for intuition. We also now include an identity of the key relationship between posterior covariance and log likelihood Hessian, in place of previous Taylor approximations.
  • As noted above, we will reconsider how to reference [1].
• Readability: thank you for the feedback, we have reworded the main theoretical section (same content) to hopefully be more readable. We have updated the figure captions to summarise the key logial steps.

Minor:

• L167 - this is now removed as we replace the det-ELBO approximation with the identity
• Eq 9 (now 11) - $s_i$ is defined a few lines above as a singular value of the Jacobian.
• The $U$-basis is a basis for $\mathcal{X}$ defined by the right singular vectors of the Jacobian (solid red lines in the image), i.e. of the matrix $D$ in the linear case (since $x=Dz$). This section steps through the simpler linear case in a way that then holds in the non-linear case.
• typo: fixed, thank you.

Questions:

• Ds: These are the Jacobian and Hessian of the decoder, desribed beneath the equation. We have adjusted the wording to hopefully improve clarity.
• L156-160: That is correct, but here we are referring to a link betwen orthogonality of partial derivatives (related to linear independence) to disentanglement (related to statistical independence).

Thank you again for your review, we hope to address your main concerns below and look forward to any further feeback.

• Experimental part:
  • Our paper completes a thread of theoretical works to fully explain disentanglement in VAEs. This result has wide generality since VAEs can be used for many data types and VAEs are combined with other modelling paradigms, e.g. state-of-the-art diffusion models. Many works demonstrate disentanglement (e.g. [5]) and several empirically verify that disentanglement and orthgonal Jacobian go hand-in-hand [3,4], and the effect of beta is demonstrated in [5,7]. We fully agree that empirical results are necessary to validate to theoretical claims in general, and everything that we explain has been empirically verified.
  • The field of machine learning currently has far more empirical results than theoretical explanations and many papers are published showing new empirical results without firm theoretical basis (which we are not criticising). Given we are explaining well-documented and important phenomenon, fundamental to the aims of machine learning, we hope that it is acceptable for a paper to be published to help redress the imbalance between empirical results and theoretical understanding.
• Deductive reasoning:
  • Thank you for the feedback, we have updated the wording of the main theoretical section and hope it is now more readable and easier to parse.
• Figures:
  • We have split the figure so that it is easier to view alongside the main text and also updated the captions so they better summarise the key logical steps.

[7] Taming VAEs

Many thanks for reviewing our work.

We have improved the presentation of our main results as you requested ("Deductive reasoning"), and hope that these are now easier to follow. These step through how a diagonal covariance matrix causes the manifold distribution to factorise into independent components that are axis-aligned directions in latent space -- a mathematical way to describe "disentanglement". If this is now more clear, we ask that you might consider increasing your score.

Separately, if, as you say, you are less familiar with parts of the literature, it would be very helpful if you could independently confirm the issues we raise with [1]. We show 2 simple cases where textbook inequalities are incorrectly applied, which are immediate to verify, invalidating the proof. This would help clarify to all/AC that our work ought not to be compared to [1] (which we anticipate being retracted).

Many thanks,

Paper Authors

[1] Embrace the Gap: VAEs Perform Independent Mechanism Analysis (2022)

Thank you again for your review and in particular for raising reference [6] (your [2]), of which we were unaware.

• [6]: this work is very interesting, relevant and complementary to our own and we have made reference to it. As now noted in our paper, [6] provides a proof of identifiability of abstract conformal maps based on Moebius tranforms, whereas we provide a constructive proof based on the SVD of the Jacobian with analogy to the linear case for insight. It will be of interest to further consider the relationship between these works. Many mathematical results are proven in many ways and the relationships between proofs can offer new insights, so we hope that concerns of overlap or differences between these approaches are resolved.
• Differences to prior works [1,2,3] are addressed in the note to all reviewers.
• Proofs of Theorems 1 and 3 are given before the theorem is stated. For the purposes of narrative, we develop the argument and conclude with what has been proven.

[6] Function Classes for Identifiable Nonlinear Independent Component Analysis

Further to our response above, we have reflected further and made the following modifications in response to reviewer comments:

• succinctly summarised the proof of Theorem 1 in Appendix A1 (fhCg, YxoF, KWdN)
• succinctly stated the proof of Corollary 1 in more formal terms in Appendix A2 (YxoF, KWdN)
• include Fig 2 (right) to help illustrate section 4.3, when column-orthogonality of the Jacobian is no longer assumed (YxoF).

For ease of reference, we highlight material changes in brown.

We very much hope this makes the paper more digestible and we look forward to any further clarifications.

Thank you for your detailed response to address our concerns regarding prior work and novelty.

Prior work

• Reverse triangle inequality: if I am not mistaken: from $| \Vert a \Vert - \Vert b \Vert | \leq \Vert a-b \Vert$, we can conclude the following for the squared norm case:
  • $| \Vert a \Vert - \Vert b \Vert |^2 = | \Vert a \Vert - \Vert b \Vert | \cdot | \Vert a \Vert - \Vert b \Vert |$
  • where we can bound each term of the right hand side by $\Vert a-b \Vert$,
  • thus leading to a correct conclusion in [1]
  • Am I missing something?

Strong assumptions

• Dimensionality: I agree that these assumptions are strong, though these are general in ICA theory, and as IMA was formulated for this case, I think it is reasonable to make the theoretical connection.
  • The authors of [1] show in their Figure 5 that the same phenomenon happens, providing empirical evidence suggesting that the theory holds in a more general case.
  • Indeed, later work extended IMA to the setting of lower-dimensional latent factors (Ghosh et al., 2023)
• Variance:
  • Please note that [3] also considers a $\beta\to 0$ setting. Also note that [1] acknowledges in the discussion that "Whether this theoretical regime [i.e., $\beta\to 0$] matches common practice remains an open question"
  • What is your explanation that the experiments in [1] back up their claim in their experimental settings that $\gamma^2\to \infty? yields better disentanglement results?

Novelty:

• As [1] also connects the orthogonality to identifiability theory. Maybe I am misunderstanding what you mean by _"first constructive proof"? My understanding of [1] is that it also accomplishes both A and B.
• Thanks, I missed the exponential family VAE point. As [1] also has more general results than Gaussians - their Assumption 1 admits a broader family of priors. Could you please compare how much it differs from your result?

Based on these concerns, I am keeping my score.

Thank you for your reply.

We detail below the errors in [1], which one would typically expect to make a paper an invalid basis for another not to be published. Further, [1] proposes a theoretical explanation under strong assumptions that our work doesn't make, which would again typically mean our work is not precluded from being published. We hope this resolves comparison to [1]. To differentiate our paper positively, note that our approach differs substantially: the stated "main result (Thm. 1)" of [1] (correctness aside) is an approximation, which we supersede with a precise identity as the starting point of our novel explanation by which orthogonality leads to disentanglement via the SVD of the decoder Jacobian.

Novelty: Re $\beta$, we generalise previous understanding of $\beta$ to exponential family $p(x|z)$ (as used in [5]). [1], Appx A3 specifically makes "the assumption of a Gaussian decoder". The prior is an independent aspect of the model, $p(x|z)p(z)$, so is an orthogonal consideration.

Material Errors in [1]: We are happy to be corrected, but we are fairly sure of the following 3 errors - any of which render the conclusion (Theorem 1, the paper's main result) unsafe:

1. [p.33, after "triangle inequality..."]:     $|\mathbb{E}[$ $\lVert a\lVert^2 - \lVert b\lVert^2$ $]|$ $\leq$ $\mathbb{E}[$ $\lVert a - b\lVert^2$ $]$       [where $a = x-f$; $b = -\sum\tfrac{\partial f}{\partial z_k}...]$
   • (dropping expectations for clarity) this has the form   " $| \lVert a\lVert^2 - \lVert b\lVert^2 | \leq \lVert a - b \lVert^2$ "        (*)
   • triangular inequality:       $| \lVert a\lVert - \lVert b\lVert | \leq \lVert a - b \lVert$       $\Rightarrow$    $| \lVert a\lVert - \lVert b\lVert |^2 \leq \lVert a - b \lVert^2$     (by squaring, as you say)
     • this is not the same as (*) as norms are squared inside the absolute operator on the LHS.
   • counter example to (*):     let $b = x>0$,   $a = x+1$
     • $\Rightarrow$     $| \lVert a\lVert^2 - \lVert b\lVert^2 | = | 2x + 1 | > 1 = \lVert a - b \lVert^2$, contradicting (*)
2. [next step, p.33]:   $\mathbb{E}[$ $\lVert (c - e) - (d-e)\lVert^2$ $]$ $\leq$ $\mathbb{E}[$ $\lVert c - e\lVert^2$ + $\lVert d - e\lVert^2$ $]$         [$c = x$, $d = f(z)-\sum\tfrac{\partial f}{\partial z_k}...)$, $e = f(\mu)]$
   • this has the form of the standard triangular inequality $\lVert a - b \lVert \leq \lVert a\lVert + \lVert b\lVert |$  
   • but, as above, all norms are squared, expanding the LHS gives an additional cross term, without which the inequality does not hold in general
     • the inequality may still hold if the cross term is bounded, but this is not claimed in the paper
3. [first step, p.34]: drops the $K$ term, which bounds the decoder's Hessian and higher derivatives (in earlier Taylor expansion)
   • this equates to a step in the work of Poole & Kumar, but is not justified (e.g. in Assumption 1)
   • since K is unbounded, any conclusions omitting it are invalid in general
   • in particular the identity we state defines the relationship between posterior covariance and the decoder's Hessian, proving it cannot be dropped in general.

As above, we hope that this resolves comparison to [1]. We regret having to highlight issues with this work and trust that our comments are taken at face value and our work considered in good faith. We look forward to addressing any further clarifications on our paper.

Thank you for responding, we respond to each query below.

We hope that are responses make clear the novelty of our work over previous. In particular, no prior work (that we are aware of) reaches the conclusions in Theorems 1-3 or shows that disentanglement equates to factorising the distribution over the mean manifold.

Key points:

• Correctness of [1]:
  • as a fundamental of mathematics, if a proof step does not hold, then the overall proof does not hold. We have shown this to be the case for 2 steps in the proof of Thm 1 (above, e.g. by counter-example), as the reviewer acknowledges. As such, Thm 1 in [1] is not proven to hold under the assumptions stated and we would expect it to be accepted that [1] is not a reasonable comparison to draw and so comment no further on it.
  • we also reiterate that our Eq 8 makes precise the approximate relationship in Thm 1 (we say approximate due to O notation, which is not in Eq 8)
• factorising the data distribution is disentanglement. We show that independent factors of p(z) are pushed forward to independent factors of p(x). This is our main contribution, described in Theorems 1 & 2 and Eqs 14 & 17
  • e.g. if features are independent, p(x = {big, blue, square}) = p(big) p(blue) p(square)
• "first constructive$^*$ proof": we not only show that the data distribution factorises into independent components, but also what those components are, i.e. push-forward distributions of 1-D independent components in Z.
  • * standard mathematical notation: proofs can be non-constructive (e.g. existence of Nash Equilibria but without providing an example of one), or constructive, as here

Other points

• Assumptions in [1]: as above, we do not comment on assumptions in [1] beyond restating that we do not make them.
• $\beta$ in posterior collapse (PC) - to clarify:
  • PC occurs if the likelihood can learn the full data distribution, i.e. $p(x|z)=p(x)$ (hence $Var[x|z] = Var[x]$ (*))
    • as seen for text where $p(x)$ and $p(x|z)$ are both multinomial distributions (e.g. see Bowman et al, 2015, cited immediately above Cor 3.1).
  • Reducing $\beta$ is found to mitigate PC in practice.
  • By Thm 3, reducing $\beta$ reduces $Var[x|z]$, thus for some $\beta$, $Var[x|z]$ drops below $Var[x]$ (a constant) and $p(x|z)$ cannot match the variance of $p(x)$ hence PC is impossible by (*).
• Column orthogonality: This is not assumed per se. In §3 we include a precise identity (Eq 8) that justifies why diagonal posterior covariances implies Jacobian column-orthogonality (in expectation), observed empirically. Our main result shows why this orthogonality implies disentanglement (orthogonality in introduced in Assumption A3).
• Linear example. correct, we first present the main argument in the linear case for intuition in the non-linear case. (In itself, that p(x) factorises as a product of distributions over 1-D sub-spaces is well known since p(x) is Gaussian.)

Questions

• Variational inference approximates the posterior by $q(z|x)$ (assumed Gaussian in VAEs). Laplace approximations approximate a distribution by a Gaussian. Opper & Archambeau, 2009 show that VI with Gaussian $q(z|x)$ is equivalent to an averaged Laplace approximation. So this is not an assumption we make, rather a way to interpret the common VAE assumption.
• Typo - fixed

Remarks

• L482: ".. induced implicitly in the decoder of a VAE." - "implicitly" refers to the VAE
• link from Hessian to Jacobian: clarified.
• Contribution list: fixed.
• Notation: we use D throughout to refer to decoder derivatives (following ICLR notation guidelines), particularly as several Hessians are referenced, but we will review the notation for clarity.
• Eq 9 derives from Eq 8 assuming a Gaussian likelihood, which we have made clear.

We address related comments from reviewers KWdN & fhCg. In particular,

• we aim to resolve comparison to reference [1]; and
• we are pleased that fhCg notes our findings are "significant if true" and believe our proofs, which involve no complex steps (and are now summarised in the appendix), confirm our claims.

Main points

• Reference [1]
  • incorrectness: we kindly ask KWdN and fhCg to please fully address our proof that [1] is incorrect in 3 places, the first 2 in misapplying the (reverse) triangle inequality (see Prior Work). We would anticipate this as grounds for retraction and so comment no further on its assumptions or relationship to other works.
  • mentioning incorrectness of [1] in our work (fhCg): we do not detail issues with [1] in our work as our work does not rely on [1] being incorrect. As stated above (see Novelty), even if it were correct, the main result of [1], Thm 1, proposes an approximate relationship (due to $O$-notation), that we state precisely by an established identity (our Eq 8). Eq 8 is then the starting point for our main result: a constructive proof of how disentanglement follows. We trust that this clarifies the significant difference between our work and [1], independent of correctness; and, having shown that [1] is both provably incorrect and does not overlap with our main, resolves the comparison.
• Identifiability (fhCg) - identifiability relates to the uniqueness of a solution up to certain symmetries, or other conditions. Our main focus is not on identifiability so we make no formal definition, rather uniqueness of solutions follows mainly from that of the SVD (column permutation and sign) on which proofs rely.

Clarifications to KWdN

• Equation 8: We confirm that, as written, Eq 8 is an equality that states the precise relationship that prior works derive an approximation to [e.g. 1, 2].
• Disentanglement definition: As we note in §2 (Disentanglement), disentanglement is not a "well-defined" mathematical concept, more a phenomenon observed when generated samples change by a single feature as one latent variable is varied. Thms 1/2 & Eqs 14/17 give a mathematical description of disentanglement as the distribution over the manifold factorising into statistically independent components that are functions of independent latent variable aligned with the standard basis.
• Constructive proof: constructive proofs generally provide clearer intuition. Here, independent components in latent space map to independent components in data space, which explains the mechanism behind how changing a single latent variable changes a single independent component/feature of the data.
• ("Unseen fixes": please elaborate on what fixes you believe do not appear.)

Dear Reviewers fhCg & KWdN:

• over the review process you have each drawn multiple comparisons between our work and [1].
• we have demonstrated fundamental differences and, moreover, that [1] contains clear errors in its main result, which is unfortunate but we would expect to lead to its retraction.
• we hoped this would swiftly resolve matters given the reviewers knowledge of [1] and we are surprised to have not had confirmation on this most fundamental point.

Having addressed all concerns raised, we ask in the time that remains that the errors we identify in [1] (as published) are confirmed and that our work is reviewed on its own merit without comparison to [1].

[1] Embrace the Gap: VAEs Perform Independent Mechanism Analysis (2022)