Compositional VQ Sampling for Efficient and Accurate Conditional Image Generation

We thank the reviewer for their careful assessment of our work. Below we aim to clarify points where appropriate.

In direct response to each of the bullet points raised in "weaknesses", in the order of appearance:

1. A stronger condition weighting increases confidence while reducing diversity, akin to a temperature parameter, and therefore is more sensitive to mode collapse (as in the sunglasses case). This is true of any temperature parameter in probabilistic sampling. The entanglement of (sunglasses/reading glasses) with the condition weighting merely indicates a higher prevalence of sunglasses in the dataset compared to reading glasses. In the limit $w\rightarrow\infty$, we would expect the method to always output the same image, as would be the case of setting temperature to 0 in any analogous situation.
2. The speedup relative to the baseline compositional methods ($2.3-12\times$) compensates for the requirement of additional forward passes ($~3\times$), which is on a comparable order of magnitude. The requirement of additional forward passes is stated in the paper as a known and necessary limitation, and in our view this is not a weakness of the contribution itself.
3. We agree that it would be nice to show our approach outperforming the baselines at greater resolution in addition to the resolutions shown. Besides increased compute requirements, there is no fundamental property of our method which limits its application to 128x128, 256x256 (c2i) or 512x512 images (t2i). However, we had limited compute for our experiments which we exploited to the greatest degree possible, and in doing so, we believe we have adequately demonstrated the benefits and potential impact of our method.

Regarding the reviewer's observation that "the method can be seen as a compositional aMUSEd": we suggest that terming our method "compositional aMUSEd" is potentially unfaithful to the origins of generative masked transformers, which pre-date aMUSEd by around 3 years. Only part of our contribution (that of adapting a pre-trained masked generative transformer model for compositional text-to-image generation without any fine-tuning) relates to aMUSEd.

Other reviewers have commented on our paper's novelty as a strength. We have formally derived why our method for compositional image generation works in theory. Our derivation is novel for discrete processes, and accordingly requires theoretical considerations above, beyond and independent of those of continuous processes, to which we dedicate over 2 full pages of our paper (spanning pages 4-6). We show empirically that the method does in fact work consistently with the theory, and not only this, but it is dramatically faster (2.3-12 times) and dramatically more accurate (cutting error by 63% on average) than a strong continuous baseline, all while having comparable parameter efficiency and FID scores. We hope this clarifies the novel aspects of our work and its potential impact, and if not, we are very happy to continue discussion until the end of the discussion period.

Our earlier comment aimed to respond fully and faithfully to each of the reviewer's concerns. We would like to check with the reviewer whether we have adequately addressed these concerns, before this stage of the discussion period ends?

We would like to gently follow up from our previous message to check if the reviewer feels we have adequately addressed the concerns/questions, and if so if the reviewer might kindly increase their rating as a result?

We thank the reviewer for their constructive questions and suggestions for improvements. We aim to address each of these points below, and we respond to suggested improvements in the most recent revision of our paper (see points below for specific changes).

Addressing the points under "weaknesses" in-order:

1. We have dedicated more than two pages to the derivation of compositional conditional generation for masked generative models. Without this derivation (one of our core contributions), there would not have been sufficient motivation to implement, train and extensively evaluate our approach (another of our core contributions). We include this derivation in detail so as to elevate the quality of the presented work by providing a rigorous motivation to our approach. The reviewer states that this "may be a weak contribution", so we would like to draw attention to:

   • The state-of-the art results for accuracy and efficiency while maintaining competitive FID, across 3 very different experimental settings, compared to already strong continuous compositional baselines
   • The capacity of our method to be applied to an existing pre-trained T2I model without any additional fine-tuning for improved control over T2I outputs
   • The lack of any similar derivation, implementation, evaluation or presentation of results in the literature as of the time of submission

   We hope the reviewer will kindly revise their assessment of our contribution in light of these clarifications.

2. We thank the reviewer for this suggestion: We agree that the full Figure 13 will enhance clarity if placed in the main text. This is now present in the most recent revision (top of page 3) which was possible without going over the 10-page limit.

3. The speedup relative to the baseline compositional methods ($2.3-12\times$) compensates for the requirement of additional forward passes ($~3\times$), which is on a comparable order of magnitude. The requirement of additional forward passes is stated in the paper as a known and necessary limitation and is not a weakness of the work itself in our view.

4. The reviewer's statement that FID is "relatively weak" may be imprecise or misleading in the context of the improved accuracy. We are keen to point out that the FID score on FFHQ is poor relative only to baselines which achieve considerably higher error rate, indicating reduced diversity but considerably greater control over the output image. Compared to the next-best baselines in terms of error rate on FFHQ, our model achieves comparable (1 component) or better (2 and 3 components) FID (Table 3, main text). This is an average 83.9% reduction in error rate on FFHQ while still reducing FID by 3.2% on average.

In response to the questions:

1. The introduction of more input components restricts the sample space, reducing diversity in favour of control. This explains the increase in FID and is a direct consequence of the probabilistic formalism of our method. Despite this increase in FID in Table 3, our method still attains lower FID scores than the method with the next-lowest error rate for 2 out of 3 input component settings for FFHQ.
2. We found that up to and including 8 objects are reliable for OOD synthesis of Positional CLEVR, with 9+ objects failing more with the addition of more input conditions. This is expected, since errors are expected to accumulate with the addition of conditions (residual variance is additive).
3. Since -c and c are not independent attributes (they are mutually exclusive), the independence assumption is explicitly violated. The result is that the contributions perfectly cancel out, thus generating an image as if the condition c or -c were not specified (i.e. P(x) as opposed to P(x|c) or P(x|-c)).
4. This is a very interesting and insightful question which, while beyond the scope of our contribution, we would be excited to explore in future work. Carefully observing the effect of dropping the input conditions after a few time steps could further improve generation efficiency with minimal effect on performance, e.g. since the number of sunglasses-specific tokens may already be sufficient in the partial image to serve as cues for a full and accurate image.
5. We used the official ICLR2025 LaTeX template for everything. We believe the presentation of references to be correct, but we welcome any specific correction if this is not the case, which we will action as soon as possible in a subsequent revision.

Our earlier comment and revised submission aimed to respond in detail to each of the reviewer's concerns, recommendations and question (see also: revised paper). We would like to check with the reviewer whether we have adequately addressed these concerns, before this stage of the discussion period ends?

We thank the reviewer for their constructive comments, questions and suggestions. We respond below to each point under "weaknesses" and "questions" in the same order that they appear in the review.

1. Our derivation requires theoretical considerations above, beyond and independent to those of continuous processes, to which we dedicate over 2 full pages of our paper (spanning pages 4-6). We show empirically that the method does in fact work consistently with the theory, and not only this but it is dramatically faster (2.3-12 times) and dramatically more accurate (cutting error by 63%) than composed diffusion models. For instance, the elimination of the unconditional constant term (Appendix D.1) is crucial to our contribution (since there's no simple way to estimate the prevalence or density of a particular attribute in the data, especially for continuous variables) and is not a necessary consideration for continuous composition. To be more specific about the trade-offs offered by masked generative models (we also direct the reviewer to the cited literature, line 051): masked generative models allow trade-offs of sampling temperature (to control diversity vs. accuracy), the number of tokens added per timestep (to control quality vs. efficiency) and also the outright improvement of enabling generation of images larger than those in the training set in addition to in-painting and out-painting without any extra implementation.
2. Collectively, the 3 datasets studied represent very different compositional tasks. The two variants of CLEVR are actually very different from each other. Positional CLEVR requires the interpretation of a continuous input signal (object co-ordinates) while Relational CLEVR relies on the precise specification of attributes of pairs of objects as well as their relation to each other, making it far more difficult (reflected in the significantly larger error rate for all baseline methods compared to the other two datasets). The FFHQ dataset differs from the CLEVR datasets in a unique way: the there are only 3 binary input attributes (totalling 8 unique combinations), however realistic sampling requires far more in the way of diversity and coverage. We further demonstrated our method's generalisation to the text-to-image setting with a pre-trained model, without any fine-tuning or specialised training objective.

Questions:

1. This is an interesting and insightful question, but unfortunately it lies beyond the scope of our present work with masked generative models. We suspect that in autoregressive models errors may accumulate more with time due to over-fitting spurious long-term correlations in the data, possibly resulting in poorer generations, however we leave implementation and extensive analysis and evaluation of compositional autoregressive models for image generation to future work.
2. Responding to each of the two sub-points:
   • The range of weights does not require tuning: we used the same range $[-3,3]$ for generating a set of 7 images for the other two FFHQ attributes (see Appendix E.2). We used the value of w=3 for all quantitative experiments for all 3 datasets to achieve our SOTA results. We also trialled $w=2$ but found $w=3$ to give across-the-board accuracy improvements on all datasets, indicating that dataset-spcific tuning of $w$ is not necessary.
   • The relative weight of each component specifies the relative importance of the corresponding feature being present in the generated image, at the cost of diversity. Specifying a higher weight for "glasses" (e.g. $w=3$) than for "smile" (e.g. $w=1$) means that the model will be more confident that the output image contains glasses, but the specific depiction of glasses will be less diverse (e.g. always sunglasses), meanwhile the same will not be true for smile (less confidence in the depiction of a smile but otherwise more diverse depictions of facial expression).

Our earlier comment aimed to respond in detail to each of the reviewer's concerns and questions. We would like to check with the reviewer whether we have adequately addressed these concerns, before this stage of the discussion period ends?

We would like to gently follow up from our previous message to check if the reviewer feels we have adequately addressed the concerns/questions, and if so if the reviewer might kindly increase their rating as a result?

We thank the reviewer for their constructive comments, questions and suggestions. We address each of these points directly below and, where appropriate, the revised paper clarifies and addresses these points (specific line numbers below).

Addressing the points under weaknesses in-order:

1. We thank the reviewer for drawing attention to lines 191 which previously had a typo. This line should state the assumption that the input conditions are independent of each other conditional on x, i.e. $P(c_1,c_2|x)\propto P(c_1|x)P(c_2|x)$ for a pair of attributes $(c_1,c_2)$, which the reviewer rightly states is the product of experts assumption. This is now clarified in the most recent revision (lines 194-195).
2. The reviewer's statement "essentially adopts ... classifier-free guidance (CFG)" is potentially misguided. The reviewer's reference [3] does not refer to classifier-free guidance nor to discrete generative models. The relationship between our method and masked CFG is analogous to the relationship between product of experts and a single classifier, i.e. they are conceptually related but otherwise distinct, with one being a prerequisite for the other. CFG does not offer any specific advantages for compositional image generation, which is the explicit focus of our contribution. Our motivation to implement and extensively evaluate our method (which is state-of-the art in accuracy and efficiency) is strongly informed by the significance of our result for composing the outputs of masked generative models, which requires theoretical considerations beyond those of continuous approaches (for instance the elimination of the conditioning terms from equation (2)).
3. This is not a gap in the evaluation; however, we accept that the reason for this should be made more explicit in the paper. In Boolean algebra, A OR B is identical to NOT (NOT A AND NOT B). This would correspond to $-(-a + -b)=a+b$ for conditional log-probabilities a and b, which is also the way we implement A AND B. This is a contradiction (A AND B is not equivalent to A OR B), which indicates that Boolean operations don't always translate to composition of log probability distributions: arithmetic negation is distributive, while Boolean negation is not, i.e. NOT(A AND B) does not imply NOT A AND NOT B. We have clarified this in Appendix D.4 of the most recent revision.

In response to the reviewer's question under "questions":

1. We agree that the relationship between equations (2) and (3) is not immediately clear, primarily due to space limitations. We attempt to make this clearer below and this is now in Appendix D.2 of the latest revision (now referenced on line 223):

   Equation (3) holds if and only if the joint distributions $s_t,c_i$ and $s_t,c_i$ are independent conditional on $s_{t+1}$, i.e. $P(s_t,c_i,c_j|s_{t+1})\propto P(s_t,c_i|s_{t+1})P(s_t,c_j|s_{t+1})$ for all pairs of distinct input conditions $c_i,c_j$ (this is the product of experts assumption, extended to sequential conditional generation).

   $P(s_t,V|s_{t+1})=P(V|s_{t+1})$ for any random variable or collection of variables $V$, since $s_t$ is entirely determined by $s_{t+1}$ since the generation process is purely additive, meaning the representation of $s_t$ is contained by that of $s_{t+1}$. Therefore, the statement of the extended product-of-experts assumption can be simplified to:

   $P(c_i,c_j|s_{t+1})\propto P(c_i|s_{t+1})P(c_j|s_{t+1})$

   i.e. the extended product of experts assumption is equivalent to the original product of experts assumption and therefore (3) follows from (2) with the substitution of a single categorical variable $x$ with a stateful discrete generative process $s_{t+1}|s_t$, provided that the process is purely additive (such as unmasking or autoregressive sampling).

We would like to gently follow up from our previous message to check if the reviewer feels we have adequately addressed the concerns/questions, and if so if the reviewer might kindly increase their rating as a result?