Mechanisms of Projective Composition of Diffusion Models

We thank the reviewer for their support for our paper and helpful suggestions.

• The reviewer suggests a quantitative analysis of our CLEVR experiments. We agree that this is an excellent idea and have performed the analysis. The results are shown in the table below and will be included in the camera-ready version should the paper be accepted.
• To produce the table below, we generated 100 samples using each composition method, and manually counted (to avoid any potential error in using a classifier) the objects in correct locations (i.e. locations corresponding to the conditioners of the distributions being composed) in each generated image. In the table below we record the histogram of object counts in correct locations.
• Regarding the reproducibility of Figures 3 and 5, we provide additional samples in Figures 8 and 10, respectively, in the appendix. Further length-generalization is also explored in Figure 9. In addition, the new table below quantitatively confirms the reproducibility of the results of Figure 3 (when attempting to compose 3 single-object distributions as in Figure 3, the empty-background (projective) composition correctly produced images containing 3 objects in 99/100 trials, while the Bayes composition never produced an image containing 3 objects in 100 trials).

Composition of location-conditioned CLEVR distributions

• N denotes number of distributions being composed (hence the expected number of objects) -- we test N=1 through N=6
• "Single-object empty" composes single-object object distributions with an empty background
• "Single-object Bayes" composes single-object object distributions with an unconditional background
• "Bayes-cluttered" composes 1-5 object distributions (with location label assigned to a single randomly-chosen object) with an unconditional background
• The table shows the histogram of manual counts, that is, each column lists the number of images that contained the given number of objects in correct locations.

We thank the reviewer for their time, insightful questions, and constructive critiques.

Overall, we want to emphasize that the goal of this paper is to understand and predict when composition will work — and just as importantly, when it will fail. That is, we want to theoretically explain prior empirical observations about when composition worked or failed; we do not aim to introduce any new methods.

We paraphrase and respond to your specific questions below.

Q1: In real-world settings, when we can expect the existence of a diffeomorphism that separates the conditions we want to compose?

• First on a technical note, we actually only require “$C^1$ diffeomorphisms”, i.e. the feature-map and its inverse should be differentiable. We will clarify this in the revision.
• Our “diffeomorphic” assumption is very closely related to existing assumptions in the literature on “disentangled feature representations.” For example, the long line of work on learning disentangled representations implicitly assumes that such a disentanglement is (at least approximately) possible. (e.g. [1] on VAEs and [2] on GANs). That is, if we are in a setting where we have a neural network that maps to and from a “disentangled” feature space (e.g. a VAE or a BiGAN), then this neural network defines our requisite $C^1$ diffeomorphism (technically this assumes the encoding and decoding networks are differentiable everywhere, which we can guarantee e.g. if if the network uses smooth activation functions).
• Finally, we do not believe that disentangled representations always exist for any distributions we might wish to compose--- and we are equally interested in understanding these failure cases. For example, “style” and “content” features are typically believed to be disentangled in the existing literature, and thus we expect style+content compositions to work. On the other hand, some concepts may be impossible to disentangle in any reasonable features space. In such cases, a diffeomorphism may not exist, and we do not expect these concepts to compose well (an example is the horse+dog composition in Figure 6).

Q2: How robust is the theory? Is it really necessary to perfectly satisfy Factorized Conditionals?

• Although our theory technically requires perfect independence, which is indeed a strong condition, our CLEVR experiments empirically study a case where the conditions hold only approximately, and explore both the robustness of the theory as well as its limits in this imperfect case (please see response to reviewer nnqo for further detail).

Q3: What are the practical implications/usefulness of the result in Lemma 6.3 which says that, even if projective composition is possible at $t=0$, reverse diffusion may not correctly sample from it?

• The fact that reverse-diffusion sampling may not work even when composition is possible at $t=0$ explains the “negative-result” in Figure 5, and may help explain other failures of composition in the literature.
• Most notably, this result helps explain empirical findings in Du et al. (2023), who showed that HMC sampling (which in particular allows sampling directly at $t=0$) was necessarily to enable successful composition in many cases. Our theory helps explain why HMC sampling worked when standard diffusion sampling did not. We discuss this further in Appendix J.1.

Q4: Text-to-image evaluations?

• Our goal is primarily to theoretically explain existing empirical evaluations in the literature. In text-to-image settings, the Bayes composition (used in Du et al. (2023) and other works) is often approximately projective, as discussed in Section 5.4. Therefore, existing empirical results in text-to-image settings are typically already constructed in a way that is compatible with our theory. We therefore accept the existing experimental results of Du et al (2023) and others and seek to understand/explain them (both successes and failures) through our theory. Of course, there is much more to study and explore empirically in text-to-image settings that we hope to explore in future work.

Q5: Quantitative evaluation?

• We performed some additional quantitative evaluations of our CLEVR experiments: please see the table of results and description of the experiments in our response to Reviewer DBEj.

If the reviewer’s concerns have been adequately addressed, we kindly ask they consider raising their score to support acceptance.

References:

[1] Isolating Sources of Disentanglement in VAEs RTQ Chen, X Li, RB Grosse, DK Duvenaud. NeurIPS 2018.

[2] A style-based generator architecture for generative adversarial networks T Karras, S Laine, T Aila. CVPR 2019.

We thank the reviewer for their support for our work and insightful questions, to which we respond individually below.

Weaknesses

Q1: The Factorized Conditional assumption, critical for the theoretical guarantees, may be too strong and not fully reflective of practical scenarios.

• Theorem 6.1 shows that it is enough to satisfy Factorized Conditionals in some feature-space, even if the assumption is not satisfied in pixel-space: that is, as long as there exists some feature-map which “disentangles” features in the appropriate sense, then distributions will compose correctly. We further discuss the empirical evidence for such disentangled features spaces, as well as the robustness of the theory to approximate satisfaction of the conditions, in our answers to Q1 and Q2 below.

Q2: Experimental validation is limited to synthetic datasets, leaving open questions about applicability in more complex, real-world settings.

• Our goal is primarily to theoretically explain existing empirical evaluations in the literature. In text-to-image settings, the Bayes composition is often approximately projective, as discussed in Section 5.4. We therefore accept the existing experimental results of Du et al (2023) and others using the Bayes composition, and seek to understand/explain them (both successes and failures) through our theory. Of course, there is much more to study and explore empirically in text-to-image settings that we hope to explore in future work.

Q3: Some proofs and technical derivations are only sketched, which could hinder reproducibility and complete understanding.

• We provide complete proofs of all claims in the Appendix. In particular, Theorem 5.3 is sketched in the main text but proved formally in Appendix G. Theorem 6.1 and Lemma 6.2, 7.1, and 7.2 are proved in Appendices H, I, and J.

Questions

Q1: Can you provide additional empirical evidence on real-world datasets to assess whether the Factorized Conditional assumption holds approximately in practice?

• This question is closely related to the existing literature on “disentangled feature representations.” This long line of work (e.g. [1] on VAEs and [2] on GANs) implicitly assumes that such a factorized representation is (at least approximately) possible, and there is substantial empirical evidence supporting this for at least some concepts. For example, “style” and “content” features are typically believed to be disentangled in the existing literature, and thus we expect style & content to form Factorized Conditionals. We mention this connection in Section 7, but will elaborate on it in the revision.

Q2: How robust are your theoretical results if the independence assumptions are only approximately satisfied? Could the framework be extended to account for partial dependencies?

• Currently, the theory requires that the independence assumptions be satisfied exactly, but developing robust versions is an important direction for future work.
• Empirically, we use the CLEVR experiments to probe the robustness of the theory. In the CLEVR setting, Factorized Conditionals holds only approximately, due to the possible occlusions and shadowing effects between different objects. Our experiments show that that projective composition is approximately, but not exactly, achieved. To push the limits of this robustness, in Figure 9 we attempt to length-generalize up to 9 objects (which works up to about ~6 objects and then degrades).

Q3: Could you elaborate on potential strategies to mitigate the sampling challenges (as noted in Theorem 6.1 and Lemma 6.3) in practical implementations of your composition operator?

• Yes! As you note, Lemma 6.3 tells us that even if projective composition is possible at t=0, reverse diffusion may not correctly sample from it. Practically, this suggests that non-diffusion sampling methods that enable sampling directly at t=0, such as variants of Langevin dynamics, may be necessary to achieve projective composition in practice (when it is possible at t=0). This is consistent with empirical findings in Du et al. (2023), who showed that HMC sampling was necessary to perform composition in many cases. We discuss this further in Appendix J.1.

Q4: Are there any plans to integrate or test your framework with more complex, high-dimensional real-world image datasets to further validate its practical impact?

• We agree this is an important area for future work; we consider the present work as the first step in this direction.

References:

[1] Isolating Sources of Disentanglement in VAEs RTQ Chen, X Li, RB Grosse, DK Duvenaud. NeurIPS 2018.

[2] A style-based generator architecture for generative adversarial networks T Karras, S Laine, T Aila. CVPR 2019.

We thank the reviewer for their support for our work and insightful questions, to which we respond individually below.

Weaknesses:

Q1: The main theoretical results only cover sampling in the pixel space, and a theoretically successful result is lacking in the feature space. Yet, feature space composition is an important application in diffusion decomposition.

• This is certainly true. In fact, we show theoretically that in feature space, diffusion sampling may not work even when projective composition is possible at t=0 (Lemma 6.3). This is consistent with empirical findings in Du et al. (2023), who showed that HMC sampling (which in particular allows sampling directly at t=0) was necessarily to enable successful composition in many cases. Our theory helps to explain why HMC sampling worked for Du et al. when standard diffusion sampling did not. We discuss this further in Appendix J.1. This result contributes to our overall goal of understanding when composition will work — and just as importantly, when it may fail.

Q2: [It is unclear how practical the definition of Factorized-Conditionals will be in real world composition.]

• Please see response to Q4.

Questions:

Q1: Can authors explain the intuitive meaning of Lemma 6.3 and discuss some remarks on it?

• Lemma 6.3 intuitively says that, even if projective composition is possible at $t=0$, reverse diffusion (or indeed any annealing method) may not be able to correctly sample from it. Specifically, the lemma proves (using a counterexample) that it is possible for a set of distributions to all vary smoothly in time, while their composition changes extremely abruptly— making any annealing-based sampling method very challenging.

Q2: In appendix B, training EDM2 on CLEVR, what is the conditional variable? Is the generation trained to be conditioned on the number of objects, shape of objects, color of objects, position of objects, or a mixture of them?

• Appendix B includes two different conditioning setups. In Figures 7, 8, and 9 we condition on the 2d location of the object (or the location of one randomly-chosen object, for multi-object distributions). In Figure 10, we condition on the color of the object. In all experiments we condition only a single attribute (either location or color) at a time, with all other attributes sampled randomly and not conditioned on. Thanks for the question — we will clarify these points in the final draft!

Q3: In appendix B.2, after training the diffusion model, how to use the trained diffusion model to get/define a generation model following certain $p_i$?

• For the location-conditional models, the $p_i$‘s correspond to different location conditioners. Specifically, in these experiments, we choose a fixed set of locations $i$ that we wish to compose, and obtain the score of $p_i$ by forwarding our conditional diffusion model conditioned on location $i$.
• Similarly, for the color-conditional models, the $p_i$‘s correspond to different color conditioners. There are only 8 colors so we assign a $p_i$ to every possible color, and we obtain the score of $p_i$ by forwarding our conditional diffusion model conditioned on color $i$.

Q4: How practical will the definition of Factorized-Conditionals be in other real-world diffusion composition, and how far it can be generalized to other successful compositions, such as composing different text prompts with/without different region masks to generate an image?

• This question is closely related to the existing literature on “disentangled feature representations.” For example, “style” and “content” features are typically believed to be disentangled in the existing literature, and thus we expect style & content to form Factorized Conditionals. Regarding composing different text-prompts, we expect similar intuitions about disentanglement to carry over --- see our Figure 6 for example, which composes using different text-prompts.
• Regarding region masks, we believe that either explicit masks or simply text-conditioning that includes location information can indeed be very helpful for achieving Factorized Conditionals (please see Section 7 for further detail).

We thank the reviewer again, and hope these responses are helpful.