Improving Compositional Generation with Diffusion Models Using Lift Scores

Thank you for your valuable comments. We address your concerns as follows:

Chamfer Distance

We chose Chamfer Distance because it (1) applies to uniform distributions and (2) is sensitive to out-of-distribution samples. KL is inapplicable for out-of-distribution samples with undefined density ratio in uniform distribution settings. Wasserstein Distance is more robust to outliers, which doesn't meet our requirement to sensitively capture unaligned samples.

Segment Anything for t2i

Vanilla Segment Anything lacks text prompt support. Grounded Segment Anything [1] would be a good candidate. We used CLIP/ImageReward following standards in previous works [2,3], but we will add a discussion in Conclusion.

TIFA score [2, 4]

New TIFA experiments show improvements across categories:

entire prompt vs individual condition

We used entire prompt. We add new experiment with minCLIP (minimum CLIP score across subjects):

Similar performance gains were observed, indicating CompLift primarily improves the weaker condition (typically missing object).

CLIP vs ImageReward

We manually labeled 100 samples from Fig. 10 for presence of both black car and white clock. With 62 positive and 38 negative samples, we calculated metric performance:

CLIP and ImageReward perform similarly, both better than TIFA. CLIP slightly preferred due to imbalanced data.

decisions feel a bit ad-hoc

We choose $\tau=250$ as the median activated pixel count among all images. Tests at 25th and 75th percentiles showed the median works best - lower $\tau$ reduces accuracy due to estimation variance, while higher $\tau$ increase rejection rates.

Regarding $\epsilon_\theta(x_t, c_{compose})$: this design is from empirical observations. Intuitively, if object $c$ exists in image $x$, then for most noisy images $x_t$, $\epsilon_\theta(x_t, c)$ should be closer to $\epsilon_\theta(x_t, c_{compose})$ than the unconditional $\epsilon_\theta(x_t, \varnothing)$ in the corresponding pixels. We'll add this explanation to the paper.

connections with CFG

Our paper uses the constrained distribution:

$

x_0 \sim p_{\text{generator}}(x_0), \quad \text{s.t. } \log p(x_0 \mid c_i) - \log p(x_0) > 0, \quad \forall, c_i.

$

Now we show that [5,6] tries to satisfy the constraint using soft regularization. Using Lagrangian relaxation and multipliers $\lambda_i \geq 0$, the objective can be transformed into:

$

\mathcal{L}(x_0, \lambda) = \log p_{\text{generator}}(x_0) + \sum_{c_i} \lambda_i \Bigl( \log p(x_0 \mid c_i) - \log p(x_0) \Bigr), \quad \lambda_i \geq 0.

$

Since $\nabla_{x_t}\log p_\theta(x_0) \propto \epsilon_\theta(x_t, t)$, and [5, 6] assume an unconditional generator, the derivative matches Equation 11 in [6]:

$

\nabla_{x_t}\mathcal{L}(x_0, \lambda) \propto \epsilon_\theta(x_t, t) + \sum_{c_i} \lambda_i \Bigl( \epsilon_\theta(x_t, t \mid c_i) - \epsilon_\theta(x_t, t) \Bigr), \quad \lambda_i \geq 0.

$

CFG [5,6] uses fixed $\lambda_i=w$, not guaranteeing constraint satisfaction.

Are Figure 4, 7, samples cherrypicked?

Yes: Figure 4 shows the ones with most-improved CLIP scores. Figure 7 uses random prompts with clear pixel separation and aesthetic quality. We will add more samples to Appendix and captions to make the selection clear.

[1] https://arxiv.org/abs/2401.14159

[2] https://arxiv.org/abs/2305.13308

[3] https://arxiv.org/abs/2301.13826

[4] https://arxiv.org/abs/2303.11897

[5] https://arxiv.org/abs/2207.12598

[6] https://arxiv.org/abs/2206.01714

Thank you for your valuable feedback. We address your concerns below.

Overstatement of minimal

We will modify the abstract to make the contribution accurate as "significantly improved the condition alignment for compositional generation".

Limited comparisons: no T=50 for vanilla CompLift; missing EBM baselines on CLEVR/text-to-image

We have conducted additional experiments to address both concerns:

1. We compared against EBM+ULA [1] on text-to-image generation (ULA is the default text-to-image sampler in their repo).

2. We ran vanilla CompLift with T=50 to enable fair comparison.

The consolidated results are in the table below. CompLift outperforms EBM+ULA across all model variants, and the cached version performs similarly to vanilla CompLift with the same T=50, with CLIP scores very close while ImageReward scores are slightly lower for the cached version.

We wish to clarify that our main focus is demonstrating vertical improvement - how CompLift boosts the base method's performance. The horizontal comparison to other baselines serves as supportive evidence that this boost helps achieve state-of-the-art performance.

Structure issues

Thank you for the constructive feedback. We will try our best to make the concept more clear. In particular, we will make the following modifications:

1. Self-inclusive caption - summarize the experiment setup in the caption of Figure 1, including the component distribution, the algebra, the data generation, and the training.
2. Early introduction of 2D dataset - briefly mention the 2D synthetic dataset in Section 3, including the description about the data generation, the component distribution, the algebra, and the accuracy metric. We will add a sentence to refer reader to Section 6 and Appendix D for more details.
3. Intuitive explanation - add several more explanation of the Compose function, using the examples in the 2D dataset.

2D dataset missing generation details

We will add more text to Section 6.1 about the dataset generation. In short, the distributions follow the generation way in [1] - they are either Gaussian mixtures or uniform distribution. We sample 8000 data points randomly for each component distribution, and train 1 diffusion model for each distribution. We will include more parameters of the distributions in Appendix D.

Unclear noise sharing performance loss in negation

Our hypothesis is that sharing the same noise introduces some bias in the estimation, and it makes CompLift over-reject samples as a conservative way. With more trials, the bias of the estimation is amplified, thus makes more samples over rejected. After taking a deeper look in Figure 2, we also observe similar sharing-noise regression for Product and Mixture, though the regression is very slight for those 2 algebras. We will modify the explanation in the paper to make this hypothesis more clear.

Cached CompLift description unclear

We provide more details about the algorithm in Appendix C. Algorithm 5 and 6 are in pseudo-code styles and might be easier for the reviewer to parse. We will add a sentence in Section 4.3 to refer readers to Appendix C for more context. Please let us know if there remains such an issue, we will keep making the paper easier to read.

EBM-related abbreviations

Thanks. We'll add explanations in Section 6.1 for all abbreviations (EBM, ULA, U-HMC, MALA, HMC).

Typos

Thanks. We'll remove the self-reference and change $z$ to $z_t$ in paragraph L317.

[1] https://arxiv.org/abs/2302.11552

Thank you for your thoughtful and constructive feedback. We address your concerns as follows.

Relationship to CAS [1]

Thank you for pointing out this important related work, which we previously overlooked. We will add reference to this valuable work, and incorporate a discussion of CAS in the "Related Work" section of the revised version. A summary of the relationship is as follows:

Our work can be seen as an extension of CAS, which investigates the potential of using CAS as a compositional criterion to decompose the alignment requirements of a complex prompt into multiple acceptance criteria. To approximate CAS, we employ ELBO estimation to reduce computational cost, as an alternative to the Skilling-Hutchinson estimator used in the original CAS paper. It would be interesting to explore how Skilling-Hutchinson-based estimation performs in a compositional setting. It may yield higher accuracy at the expense of greater computational overhead, which we plan to investigate in future work.

Comparison with Attent-and-Excite [2] on text-to-image generation

Thank you for the great question. We conducted a new experiment using Attent-and-Excite [2], focusing on the additional improvement achieved by incorporating CompLift. We observed consistent performance gains with both SD 1.4 and SD 2.1. Note that SD XL is not included due to the lack of support in the original Attent-and-Excite code.

The idea of "counting the activated pixels" in Section 5.1 seems quite confusing... How did you set the threshold $\tau$?

Thank you for raising this thoughtful concern. We agree that performance could be further improved by making $\tau$ an object-specific hyperparameter. For simplicity, we currently set $\tau$ as a uniform threshold.

We chose $\tau = 250$ as the median value among the number of activated pixels across all images. We also experimented with the 25th and 75th percentiles, and found the median performed best in practice. A lower $\tau$ leads to less accurate rejection due to ELBO variance, while a higher $\tau$ increases the rejection rate.

We will include more details on the derivation of $\tau$ in the Appendix. Intuitively, $\tau = 250$ corresponds to ~1.5% of the total number of the latent pixels in SDXL (128x128 latent space) and ~6.1% in SD 1.4/2.1 (64x64). We find this small threshold sufficient, since our focus is on identifying "missing object" issues. If an object is missing, it tends to result in almost no activated pixels. Additional discussion will be added to the Appendix.

Does the same trend hold for the recent work "Reduce, Reuse, Recycle" [3] based on MCMC?

Thank you for this suggestion. We also observed a clear overall advantage of our method over MCMC-based methods like "Reduce, Reuse, Recycle" [3]. While certain MCMC variants such as U-HMC and MALA perform comparably in specific scenarios (e.g., the first test case in Product and second in Mixture), they often generate samples outside the target distribution in other settings, unlike our method.

We will update Appendix D to include visualizations comparing results from the MCMC-based methods.

Typo: page 8 line 408: ImageResizer -> ImageReward

Thank you for catching this typo. We will correct it in the revised version.

[1] https://openreview.net/forum?id=E78OaH2s3f

[2] https://arxiv.org/abs/2301.13826

[3] https://arxiv.org/abs/2302.11552

Thank you for your valuable feedback and questions. We address your concerns below:

On CompLift's dependence on underlying generative model

We agree and will add this theoretical limitation to our Conclusion. While theoretically CompLift cannot improve if the base method produces no accurate images, in practice even weak generators often improve with ≤5 candidate images.

On "minimal computational overhead" claim

We'll remove the ambiguous term "minimal" and claim only "requiring no additional training." For text-to-image generation, the (n + 2) · T additional forward passes can be parallelized to reduce latency. Currently, generation takes ~15s and lift score calculation ~30s on a 4090 GPU, with GPU memory as the bottleneck. Ideally, we can further parallelize to the latency of O(1) forward pass given enough GPU memory.

On cached CompLift's accuracy-speed tradeoff

Every column in Fig. 5 has a corresponding accuracy row in Table 2 (Cached CompLift has T=50 by default). We'll add a footnote clarifying this. In practice, we perceive small accuracy regression on Mixture and Negation task when switching from vanilla CompLift to Cached CompLift. However, the accuracy remains significantly higher than other baselines. The tradeoff exists, but seems to be mild and acceptable given the substantial speed improvement.

On lift score equivalence to point-wise mutual information [1]

We'll update the Related Work section to reflect this equivalence. Our paper applies point-wise mutual information (PMI) as an acceptance/rejection criterion, focusing on missing-object cases and composing PMI for multiple objects.

On real-world text-to-image improvement

The seemingly trivial CLIP improvement is due to the low magnitude of CLIP scores. Here, we provide another perspective to interpret the numbers. We compare the CompLift selector to the perfect best-of-n selector, which has direct access to the metric function. The percentage gain is calculated as (CompLift metric - baseline metric) / (perfect selector metric - baseline metric). On average, the gains are ~40% for vanilla CompLift and ~30% for cached CompLift. We will add more explanation to the Appendix.

On CompLift's compositionality

Our approach is to (1) evaluate individual concepts separately, and (2) compose an acceptance/rejection criteria from these multiple individual criteria, as Algorithm 3. Thus, the CompLift criteria seems compositional from our perspective. We wish to mention Composable Diffusion [2] as an example to elucidate such a perspective. Essentially, the approach uses (1) individual score on each concept, and (2) compose the score from these multiple individual scores. Such a factorize-and-compose way helps reduce the hardness to comply with complex prompt. As shown in our experiments, it improves performance on complex multi-object generation by evaluating individual conceptual alignment before making a composed decision.

On limited algebraic operations for text-to-image

We acknowledge this limitation. While we tested all algebras in the 2D dataset, we found no existing mature benchmark for OR/NOT algebras in text-to-image generation. We'll note this in our Conclusion for future work.

On fair comparison with Composable Diffusion

We recognize the challenge in comparing these approaches. Composable Diffusion generates candidates with no internal selection mechanism, while CompLift is a post-hoc filter. They serve different purposes and are not mutually exclusive—CompLift can enhance generation models by leveraging semantic alignment for filtering results. Our main goal in the experiments is to show such an enhancement, instead of a direct replacement, of other baselines such as Composable Diffusion.

[1] https://arxiv.org/abs/2310.07972

[2] https://arxiv.org/abs/2302.11552