Can Diffusion Models Disentangle? A Theoretical Perspective

Dear reviewer 34h9,

We thank you for the thoughtful comments and for recognizing the originality and importance of our work. We appreciate your engagement with both the theoretical and empirical parts of the paper, and we address your concerns below.

1. On experimental details in Section 5

We defer the experimental details to the Appendix E to focus on the exposition of our theoretical results. But we will consider adding the dataset description back to the main text in the extra space of the camera-ready version of the paper.

2. On theoretical-empirical gaps and the role of Eq. (13)

We appreciate this concern and agree it deserves clarification. Eq. (13) is a practical implementation of Eq. (11), tailored for high-dimensional image/speech data. Specifically:

• We omit the balancing term for empirical simplicity;

• Style guidance is applied by zeroing out content, equivalent to the style regularizer in Eq. (11) when the input data has ISM structure;

• For MNIST colorization/CIFAR-10 denoising/voice conversion, the ISM assumption is not explicitly enforced, but the structure is approximately satisfied in these tasks either directly or after the input is transformed into self-supervised representation [1,2].

Thus, Eq. (13) is not theoretically disjoint from Eq. (11)—rather, it's an approximated empirical proxy aligned with our theoretical principles. We will make this relationship clearer in Sec. 5 and Appendix E.

[1] O. D. Liu, H. Tang, and S. Goldwater, “Self-supervised predictive coding models encode speaker and phonetic information in orthogonal subspaces,” in Interspeech, 2023.

[2] Kamper, Herman, Benjamin van Niekerk, Julian Zaïdi, and Marc-André Carbonneau. “LinearVC: Linear Transformations of Self-Supervised Features through the Lens of Voice Conversion,” 2025.

3. On Eq. (7): Why not make $z_{\phi}(X)$ constant

We appreciate the opportunity for clarifying. While the regularizer in Eq. (7) enforces an upper bound on mutual information between content and input, the score matching loss term still requires $z_{\phi}(X)$ to be predictive of the data’s conditional score. If $z_{\phi}(X)$ were constant, the model could not satisfy the conditional score matching objective. Thus, the interplay between the loss and regularizer ensures that $z_{\phi}(X)$ retains just enough content information for prediction without overfitting. This tension promotes disentanglement.

4. On score function using ground-truth generator (lines 1122 and 1204)

In Appendix A and B, we use the true generative function $f(Z,G)$ only for theoretical analysis, such as constructing counterexamples (e.g., for demonstrating non-editability in Example 1) or establishing upper bounds of the optimal conditional score matching loss in the proof of Theorem 4.1 and Theorem 4.2. In practice, the model learns a score estimator $s_{\theta},$ and the generator is not known. We will clarify in the text that these instances refer to analytical derivations only, not model implementation.

5. On use of $C_2$ in line 1077

It is used in line 1086, but we agree it appears too early, and will consider moving it to the later part of the proof.

6. On use of $z_{\phi}(X)$ in Eq. (13)

In Eq. (13), $z_{\phi}(X)$ is passed to the score estimator $s_{\theta}(X_t, z_{\phi}(X), G, t).$ The second term in the loss zeroes out $z_{\phi}(X)$ to encourage style-specific signal separation (style guidance). The loss structure is consistent with Eq. (11) but omits the balancing term since our model is able to work without it empirically.

7. On CIFAR10 background being black

In our CIFAR-10 experiments (Sec. 5, Fig. 3), we perform denoising, not colorization and the input is not always black. The colorization task is conducted on MNIST, which is gray-scale. In principle, our method works for any background color.

8. On relation to deterministic disentanglement formalizations

Thank you for asking this important question. Classical deterministic disentanglement assumes that invertible functions exist to recover latent variables (e.g., $Z=z(X),\,G=g(X)$). However, diffusion models are stochastic and non-invertible, making such assumptions inapplicable. Our proposed $(\epsilon,\nu)$-disentanglement and $\epsilon$-editability generalize these notions to stochastic settings by relying on mutual information and distributional distances rather than pointwise mappings.

These criteria reduce to classical notions in the low-noise limit, where DMs become nearly deterministic and when the content and style variables are discrete. We will add a paragraph in Section 3 connecting our framework to deterministic disentanglement theories.

9. On theoretical derivation in the appendix

Thank you for this candid feedback. We understand that Appendix B–D contains dense derivations. We will improve the clarity by:

• Restating the theorems and definitions before proofs.

• Adding brief and intuitive summaries before technical steps

• Clarifying notation and referencing which result each equation builds on

---

We thank you again for the insightful comments and hope our response helps to address your concern and makes our paper more suited for the NeurIPS conference. Happy to discuss further and consider further revisions.

Dear reviewer cYrQ,

Thank you for the thoughtful and constructive feedback. We address the concerns below:

1. Clarifying the connection between multi-view setup and ISM assumption

Thank you for this important observation. We agree that multi-view disentanglement (Theorem 4.2) and the independent subspace model (ISM) (Theorem 4.5) address distinct settings. The former analyzes general nonlinear, stochastic mappings under weak supervision, while the latter provides stronger guarantees in a special case with linear structure.

Our intent is not to claim that ISM is a general case of multi-view disentanglement. Instead, we show that when the ISM structure holds, we can derive finite-sample convergence guarantees using gradient dynamics (Thm 4.5). The general framework (Defs. 3.1-3.4) encompasses both settings. We will make this relationship clearer in the revision by explicitly distinguishing the scope and assumptions of each theorem.

2. Assumption of known style function

This is a valid concern. The assumption of a known style function appears only in the “content disentanglement” setting (Def. 3.3, Thm. 4.1), where the goal is to disentangle content given a fixed style – a setting aligned with applications like voice conversion and image editing.

In practice, we approximate the style function using a pre-trained model (e.g. speaker embeddings from ECAPA-TDNN, which achieves almost perfect speaker verification error rate [1]). While this is not a ground-truth oracle, it provides a fixed and consistent representation of style across training and inference, which makes the theoretical setup meaningful. This assumption mirrors common practice in the voice conversion literature and is not unique to our framework.

We agree it is an idealization, but as a weaker alternative to requiring full latent access, it enables principled analysis under partial supervision. In real-world scenarios where even pre-trained models are unavailable, we propose the multi-view formulation (Def. 3.4, Thm. 4.2), which removes the need for known style and instead leverages paired observations. We will highlight the distinction more clearly.

[1] Desplanques, B., Thienpondt, J., & Demuynck, K. “ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification”. Interspeech 2020.

3. Clarifying Definition 4.3 (independent subspace model)

Yes, Def. 4.3 assumes a linear mixing model to latent factors, which is a simplified case used to derive finite-sample convergence guarantees. As mentioned in the paper (Sec. 4.3), the ISM assumption is not required in general – our earlier results (Theorems 4.1 and 4.2) apply to nonlinear, stochastic mappings under mild Lipschitz assumptions. The ISM-based result (Thm 4.5) serves to complement these general results with stronger guarantees and clearer intuition about training dynamics under structure.

Even though the ISM assumption does not always hold for raw data, such structures often emerge in the latent space of self-supervised representations. For instance, previous study [2] has found that self-supervised representation models have learned orthogonal subspaces for content (phonetic) and style (speaker) information, as mentioned in Section 4.3 of our paper. Further, practical algorithms making the ISM assumption have recently achieved competitive results in disentanglement tasks such as voice conversion [3] and image editing [4], demonstrating the applicability of the ISM assumption to complex data.

[2] O. D. Liu, H. Tang, and S. Goldwater, “Self-supervised predictive coding models encode speaker and phonetic information in orthogonal subspaces,” in Interspeech, 2023.

[3] Kamper, Herman, Benjamin van Niekerk, Julian Zaïdi, and Marc-André Carbonneau. “LinearVC: Linear Transformations of Self-Supervised Features through the Lens of Voice Conversion,” 2025.

[4] L. Rout, N. Raoof, G. Daras, C. Caramanis, A. Dimakis, and S. Shakkottai, “Solving linear inverse problems provably via posterior sampling with latent diffusion models,” in NeurIPS, 2023.

4. Dimensionality assumptions in multi-view setting

We appreciate the opportunity for clarifying. No, we do not require that the observation $X\in \mathbb{R}^{d_X}$ has the same dimensionality as $(Z,G).$ Our theory allows for arbitrary encoder functions mapping inputs to latent spaces of lower dimension. For instance, the encoder can output $\hat{Z}\in \mathbb{R}^{d_Z},$ even if $d_Z<d_X.$ The score function operates in observation space, but the latent representations can reside in lower-dimensional subspaces. We will clarify this in the definitions and theorem statements.

5. Speaker embedding as known style function

We appreciate the opportunity to clarify. When we refer to using speaker embeddings (e.g., d-vectors), we treat the pretrained embedding extractor $g(\cdot)$ as a proxy for the true style function. This is standard in voice conversion pipelines and aligns with how weak supervision is commonly defined in practice. We never claim that this represents the true style variable $G$, but rather a fixed, known surrogate used during both training and inference, consistent with Def. 3.3. We will revise the text to make this distinction clearer and emphasize that our theory supports both fixed-style and fully-learned (multi-view) scenarios.

6. Logical progression across theorems

Thank you for pointing this out. The three theorems build progressively: Theorem 4.1 (Content disentanglement): proves approximate disentanglement under known style supervision. Theorem 4.2 (Multi-view disentanglement): proves approximate disentanglement under multi-view supervision with learned content and style encoders. Theorem 4.5 (ISM content disentanglement): focuses on linear latent structure and shows finite-sample global convergence with gradient descent. The first two theorems highlight approximate disentanglement under weak supervision. Theorem 4.5 is a special case of Theorem 4.1 when the data distribution has ISM structure, which allows us to study the training dynamic of disentanglement and prove stronger disentanglement guarantees. The special case of Theorem 4.2 with ISM is omitted since its proof is analogous to Theorem 4.5, though we provide a sketch of the proof strategy in Line 245-248.

7. Assumption discussion and relaxation

We agree and thank the reviewer for the suggestion. We already mention in Appendix B.7 and C.4 that some assumptions (e.g., exact independence between $Z$ and $G$, or i.i.d styles) can be relaxed to approximate independence settings.

Further, under the content disentanglement setting, the assumption of access to the true style variable can be relaxed by assuming instead access to a sufficiently good style estimator $G’$, such as a reliable speaker embedding model in voice conversion. Such an estimator is essentially a perturbed version of the true style variable, and since the score function is assumed to be Lipschitz in our theory, it is robust against such perturbations and all theorems proved in our paper still hold, albeit with additional error terms due to the style estimation error.

We will consolidate these discussions into the main text and include a new limitations and future directions paragraph addressing how assumptions might be relaxed in practice.

---

We thank you again for the insightful comments and hope our response helps to address your concern and makes our paper more suited for the NeurIPS conference. Happy to discuss further and consider further revisions.

Dear reviewer qfQD,

We sincerely thank you for careful reading and for highlighting both the clarity of our theoretical contributions and the originality of our disentanglement framework. Below, we respond to the key concerns raised regarding the empirical connection, equation derivation, and experimental scope.

1. Empirical validation of theory and derivation of Eq. (13) from Eq. (11)

We appreciate the reviewer’s careful inspection. Eq. (13) is a practical implementation of Eq. (11) tailored to image domains (MNIST, CIFAR-10) where the true ISM structure is unknown or approximate. We drop the balancing loss in Eq. (13) for simplicity since empirically the model does not rely on the loss to escape bad local minima.

Eq. (13) is equivalent to Eq. (11) without the balancing loss and when the input data is an ISM. The form in Eq. (13) looks different from Eq. (11) primarily because the (13) is using conditional score matching while (11) is using unconditional score matching. The two are equivalent as discussed in Section 2, and our theory can be easily adapted to conditional score matching as well, as discussed in line 250-251.

Further, the style-guided regularization losses in (13) (denoted as $L_r’$) and (11) (denoted as $L_r$) are related. Suppose we assume ISM and restrict the function class to be a variant of the dual encoder score network in Fig. 2, i.e. $s_{\theta}(X_t,\hat{Z},G,t)=Us_Z(X_t,\hat{Z},t)\mathbf{1}[\hat{Z}=\mathbf{0}_{d_Z}]+Vs_G(X_t,G,t),$ and suppose the all-zero vector is outside the support of the content estimator $\hat{Z}$, then we have $L_r’=\mathbb{E}||Vs_G(X_t,G,t)+N_t/\sigma(t)||^2.$ This takes the same form as $L_r$ in (11), except in two aspects: i) this is conditional score matching; ii) $s_G$ is a score function conditioned on $G$ instead of an unconditional one in (11). We have already discussed i), and ii) poses no issue either since the score function conditioned on $G$ still satisfies the ISM assumption and can still be handled by our theory.

2. Role of balancing loss and intuition

Intuitively, the balancing loss tries to balance between the subspace matrices $(U,V)$ and the subspace score networks $(s_Z^{\theta_Z},s_G^{\theta_G})$ to have similar operator norms (by viewing the subspace score networks as operators). Otherwise, suppose $(s_Z^{\theta_Z},s_G^{\theta_G})$ have operator norms much larger than $(U,V)$, then $U \approx 0_{d_X\times d_Z}, V\approx 0_{d_X\times d_G},$ since their combined operator, the full score network $s_{\theta}$, has finite norm. Further, since the gradients $\nabla_{\theta_Z} L^{\lambda_r}$ and $\nabla_{\theta_G} L^{\lambda_r}$ scale with $U,V$, this implies $\nabla_{\theta_Z} L^{\lambda_r}\approx 0$ and $\nabla_{\theta_G} L^{\lambda_r}\approx 0,$ even though the learned content and style spaces $R(U)$ and $R(V)$ are nowhere near the true spaces $R(A_Z)$ and $R(A_G)$. This results in a bad local minimum.

The balancing loss also does not affect the output of the optimal score network, since for any “imbalanced” optimal parameters $(U,V,\theta_Z,\theta_G)$, there always exist another $(U,V,\theta_Z’,\theta_G’)$ that achieves zero balancing loss. The argument based on singular value decomposition is provided in 1306-1309 of Appendix D.3.

Therefore, the balancing loss helps the model to avoid bad local minima but does not change the location of at least one global minima for the regularized score matching loss. On the other hand, we empirically observe the model to converge near a global optima without the balancing loss, so we discard it for our experiments. The balancing loss is thus largely a proof artifact, and we leave analysis without such a loss for future works.

3. Multi-view disentanglement experiments

We appreciate the reviewer highlighting this. Our speech experiments (Sec. 5, Fig. 5) do instantiate multi-view disentanglement: different speakers provide style views, and the shared emotion across recordings is the content. This matches the Definition 3.4 setup. We acknowledge that this connection could be more explicitly stated and will revise the text to clarify this mapping between theory and practice. We are also actively developing additional vision-domain multi-view experiments.

4. Role of $U$ and $V$ in Eq. (10), and learnability

Thank you for pointing this out. U and V are learnable linear projections that align the outputs of the content and style encoders with the overall score function, motivated by Lemma 4.4. We adopt this architecture to reflect the ISM-based decomposability of scores into subspace components. Their gradients are computed and optimized during training (see gradient flow Eq. 12). We will update the paper to explicitly define U and V as learnable parameters.

5. Meaning of $n$ in Eq. (11)

Sure. $n$ is the number of samples in the training set. It is mentioned in the “Intuition” section in Line 243-244.

6. Intuition behind style guidance regularizer and mutual information

We agree this point deserves further clarification. The key idea is that when the score network is forced to explain part of the data using only style, the model is implicitly discouraged from encoding style information redundantly in the content branch. For general latent variable models, this is a heuristic regularization rather than a formal MI minimization, and we will add a clearer explanation and acknowledge.

For ISM, however, this regularization does lead to MI minimization. To see this, consider the extreme case where no style guidance regularizer is used, then the content encoder can simply copy from the input X and maximize the score matching loss $L_0$ by letting $U$ covering the entire data space, and in this case the mutual information between the content encoder and X, $I(\hat{Z};X)=I(X_{t_0};X)$ is maximized; As style guidance weights $\lambda_r$ increase, the style encoder starts to play an increasingly important role in score prediction until it can predict the style component of the score perfectly. In turn, the content encoder becomes less and less predictive of the style component. Further, by Theorem 4.5, the subspaces $R(U)$ and $R(V)$ recover the content and style subspaces up to vanishingly small error for sufficiently large $\lambda_r$, and thus $I(\hat{Z};X)\approx I(Z_{t_0};X)<I(X_{t_0};X)$ by data processing inequality and that the true content encoder is injective. For a more precise relation between the style regularizer weight and $I(\hat{Z};X)$, one can upper bound $I(\hat{Z};X)$ in terms of subspace recovery errors using an argument similar to Appendix D.1.

7. Comparison to prior work in image experiments

We have made a comparison with related works in two additional disentanglement experiments on the CelebA image dataset. First, we perform denoising with the same setting as CIFAR 10 and compared our approach with two standard baselines, conditional GAN [1] and beta-VAE [2] adapted to the content disentanglement setting by conditioning on the observed style variable, which are representative of the approach taken in the papers suggested by the reviewer. As shown in the Table below, our method is superior to the baselines and follow the same trend with the style regularization weight as in our experiments on MNIST and CIFAR10.

Further, we perform face editing with gender as the style variable. As shown in the table below, our method is superior to the conditional InfoGAN [1] and betaVAE [2] baseline in terms of style transfer (CLIP score) and content preservation (LPIPS score).

[1] Kim, Hyunjik, and Andriy Mnih. "Disentangling by factorising." International conference on machine learning. PMLR, 2018.

[2] Chen, Xi, et al. "Infogan: Interpretable representation learning by information maximizing generative adversarial nets." Advances in neural information processing systems 29 (2016) Moreover, we plan to perform larger-scale experiments on more diverse datasets in the future.

8. Miscellaneous: wrong figure and limitations

Thank you for pointing it out. We acknowledge the formatting error in Fig. 6(d) and will fix it in the camera-ready version. We will also combine our scattered discussions of limitations into a dedicated section in the final version for clarity.

---

We thank you again for the insightful comments and hope our response helps to address your concern and makes our paper more suited for the NeurIPS conference. Happy to discuss further and consider further revisions.

Dear reviewer KBNb,

Thank you for the thoughtful feedback and for acknowledging the strengths of our paper. Below, we address the main concerns raised:

1. On the ISM assumption and generalization to complex data

We agree that ISM represents a simplified model; however, we emphasize that ISM is only one part of our broader theoretical framework. Sections 4.1 and 4.2 provide results under general nonlinear, non-invertible mappings with only subgaussian and Lipschitz assumptions (see Theorems 4.1 and 4.2). ISM is used in Section 4.3 specifically to establish stronger guarantees such as finite-sample global convergence, which are rare in existing literature. The theoretical contributions are general.

Further, even though the ISM assumption does not always hold for raw data, such structures often emerge in the latent space of self-supervised representations. For instance, previous study [1] has found that self-supervised representation models have learned orthogonal subspaces for content (phonetic) and style (speaker) information, as mentioned in Section 4.3 of our paper. Further, practical algorithms making the ISM assumption have recently achieved competitive results in disentanglement tasks such as voice conversion [2] and image editing [3], demonstrating the applicability of the ISM assumption to complex data.

[1] O. D. Liu, H. Tang, and S. Goldwater, “Self-supervised predictive coding models encode speaker and phonetic information in orthogonal subspaces,” in Interspeech, 2023.

[2] Kamper, Herman, Benjamin van Niekerk, Julian Zaïdi, and Marc-André Carbonneau. “LinearVC: Linear Transformations of Self-Supervised Features through the Lens of Voice Conversion,” 2025.

[3] L. Rout, N. Raoof, G. Daras, C. Caramanis, A. Dimakis, and S. Shakkottai, “Solving linear inverse problems provably via posterior sampling with latent diffusion models,” in NeurIPS, 2023.

2. On additional experiments (e.g. CelebA)

The main contribution of the paper is to develop a theoretical framework for general diffusion model-based content-style disentanglement, and our experiments are intentionally structured to validate different settings (content, multi-view, and ISM) in a controlled yet representative manner. As noted in the paper (Section 5 and Appendix E.3), we also include real-world evaluations on speech tasks (e.g., IEMOCAP) using voice conversion and emotion recognition to demonstrate applicability beyond synthetic data.

We have further verified our theory on two more disentanglement tasks on the CelebA image dataset. First, we perform denoising with the same setting as CIFAR 10 and compared our approach with two standard baselines, infoGAN [4] and beta-VAE [5] adapted to the content disentanglement setting by conditioning on the observed style variable, which are representative of the approach taken in the papers suggested by the reviewer. As shown in the Table below, we observe the same trend as our previous experiments.

Further, we perform face editing with gender as the style variable. As shown in the table below, our method is superior to the conditional GAN baseline in terms of style transfer (CLIP score) and content preservation (LPIPS score).

Moreover, we plan to perform larger-scale experiments on more diverse datasets in the future.

[4] Kim, Hyunjik, and Andriy Mnih. "Disentangling by factorising." International conference on machine learning. PMLR, 2018.

[5] Chen, Xi, et al. "Infogan: Interpretable representation learning by information maximizing generative adversarial nets." Advances in neural information processing systems 29 (2016)

3. On empirical comparisons with content-style disentanglement methods

We appreciate this suggestion and agree that comparisons with existing baselines would strengthen the empirical section. However, as our primary focus is theoretical, our experiments are designed to validate the theoretical predictions (e.g., convergence trends, editability, MI bounds) in a controlled manner.

That said, we do include comparisons with several baselines in the speech domain (Fig. 5), such as pitch shifting, TriAAN-VC, and nearest-neighbor VC methods. In our response to 2, we also compare our method with two standard baselines for disentanglement, conditional InfoGAN and beta-VAE. For image data, direct comparisons with methods such as StyleDiffusion [Wang et al, ICCV 2023] are non-trivial due to differences in task setup and model supervision, but we plan to incorporate these in future versions.

4. On novelty compared to DisDiff and mutual information-based methods

We respectfully disagree that our contribution lacks novelty. While DisDiff [Yang et al., 2023] introduces MI-based regularization for DMs, it is a fully empirical work. Our work makes the first theoretical contributions toward identifiability and convergence guarantees of DM disentanglement under stochastic, non-invertible mappings – challenges not addressed in DisDiff. Our formulation of $(\epsilon,\nu)$-disentanglement and $\epsilon$-editability (Defs. 3.1-3.2), and the theoretical results (Theorems 4.1-4.5) including finite-sample convergence, are novel and unique to our paper.

Moreover, our use of MI regularization is not merely adopted from prior work but is grounded in our theoretical derivation to achieve approximate identifiability. These elements distinguish our work both in terms of scope and depth.

5. On title scope

We will change the title to “Can diffusion models disentangle content from style? A theoretical perspective” to limit the scope of our theory to content-style disentanglement.

---

We thank you again for the insightful comments and hope our response helps to address your concern and makes our paper more suited for the NeurIPS conference. Happy to discuss further and consider further revisions.

Thank you again for the clarification.

You are absolutely right that DisDiff includes a theoretical result, and we appreciate the opportunity to clarify the distinction. The main theorem in DisDiff [Yang et al., 2023] demonstrates that their proposed regularization objective upper bounds the mutual information between content and style. However, this result does not provide identifiability or convergence guarantees for disentanglement within the diffusion process itself. In fact, the authors themselves acknowledge this limitation in Section 4.3 of their paper:

Given the difficulty in identifying sufficient conditions for disentanglement in an unsupervised setting, we follow the previous works in disentangled representation literature, which propose necessary conditions that are effective in practice for achieving disentanglement.

As we demonstrate in Example 1 of our paper, minimizing mutual information between latent variables alone does not guarantee disentanglement, even in idealized settings. This underscores the need for supervision (e.g., labels or views) and structural assumptions (e.g., ISM) to ensure recovery, as provided in our theoretical framework.

By contrast, our theory introduces an information-theoretic formalization of approximate disentanglement and editability under weak supervision settings (e.g., partial labels and multiple views), and proves global convergence guarantees under the ISM assumption. We agree that the phrasing “first theoretical contribution” was too strong and will revise it in the final version to reflect the specific novelty in our identifiability guarantees under weak supervision.

We believe this distinction highlights an important shift from empirically motivated regularization to theoretically grounded disentanglement objectives in the context of diffusion models.

Thank you for the detailed response, I think the comment that "DisDiff is fully empirical work"is not rigorous, DisDiff presents a therom of disentangling for diffusion model and gives a proof on the therom in appendix. Also I think " first theoretical contributions" is not a rigorous claim for this paper.

Considering that my concerns are still not fully solved, I will keep my rating in this period.