Causally Motivated Diffusion Sampling Frameworks for Harnessing Contextual Bias

• (W2) This approach relies heavily on text prompt conditioning, limiting its effectiveness in handling complex cases. As the reviewer mentioned, text prompt based conditioning might have some limitations such as ignoring some objects in the complex prompt while the classifier-guidance-based approach [1] could bypass the issue. However, the approach of [1] is not scalable as they need to train a classifier per attribute (e.g., minority score) that can take noised image $x_t$ as input. They also require labeled dataset to train the classifier. On the other hand, ours does not require any additional training/finetuning process nor dataset preparation (assuming $p(C')$ is precomputed which is reasonable as we are going to release). Ours also can be applied to off-the-shelf pretrained Diffusion Models, which is a huge benefit in practice. Even though we might have the limitations of the long text conditioning, we believe our proposed methods have our own benefits compared to [1]. For example, our CB- can be easily applied to various scene generations as shown in Fig. 4, 5, and 7 in the main paper. However, it is very difficult for [1] to do the same thing as ours for two reasons: 1) it is unclear if their method can be used with pretrained StableDiffusion (SD), and 2) having a noise-aware classifier that can deal with thousands of diverse classes of SD is non-trivial.

[1] Don't Play Favorites: Minority Guidance for Diffusion Models, ICLR'24

• (Q1) Clarifying the precomputing step of $p(C')$. We apologize for the confusion. We provide a pseudo code for elaborating the process to retrieve (i.e., precompute) the contextual bias $p(C')$. Please see the Clarifications in "Response to all reviewers". As for the required time, for (a) unconditional image generation and (c) conditional image generation steps, we respectively use 5 days with 10 GPUs (RTX A5000). For (b) and (d) steps, we respectively use 2-3 days with 10 GPUs (RTX A5000). In total, we roughly used 15 days to obtain the final $p(C')$. As for the approximated space complexity, we use 1 TB for storing 1,130,195 images. (Note that one million images are obtained from the (a) step and can be removed after (b) step, and another one million images are obtained from the (c) step.)

• (W1, Q2) Bias of Diffusion, VLM, and LLM As the reviewer mentioned, our work assume that the bias of diffusion models are highly related to VLM and LLM. As mentioned in Section D.2 of the main paper, we tried to justify our assumptions by the fact that many large-scale models are heavily connected to one another because they share a public dataset, and LLMs could be used to generate text caption data. More specifically, diffusion models leverage pretrained LLM or VLM for obtaining text representations during the training process, from which we can suppose diffusion models are significantly affected by pretrained LLM or VLM. Furthermore, StableDiffusion and LLaVA are pretrained with LAION dataset which use CLIP to preprocess the data. That is, the bias from CLIP can affect the bias of pretrained StableDiffusion and LLaVA. Thus, while we agree that the contextual bias may not be exactly the same, they are related enough to be significantly useful as seen by our experiments. We are happy to include more discussion of this in the final paper.

control mechanism over diffusion remains overly rough. For instance, for a weak diffusion model, its understanding of the prompt is quite superficial. Even if your method makes additional modifications to the prompt, it may not accurately reflect in the generated images.

As the reviewer pointed out, weak text-guided Diffusion Models (e.g., SD 1.x or 2.x) could have an issue in applying multiple objects (c.f., catastrophic neglect by [1]).

However, as the reviewer might already know, there have been some good research papers dealing with such issues [1-3]. Since their methods are implemented on top of the pretrained text-guided diffusion models as ours is, we can easily apply, let’s say, Attend-and-Excite during the sampling process to strengthen the subject tokens. For example, suppose a prompt $y$ is "A cat and a dog", and a sampled contextual bias $c’$ is "library". As shown in the rightmost column of Fig. 6 in Attend-and-Excite, we can see that the limitation (i.e., catastrophic neglect) of the text-based guidance could be mitigated successfully. Again, our method is not exclusive with the existing works [1-3], and thus we can mitigate the limitations of text-based conditioning by combining those works with our methods. Please note that the problems that the previous studies [1-3] aim to solve are far from the problem we are solving $^{1)}$, and thus combining [1-3] with our method does not affect our technical novelties.

Secondly, the stronger version of Diffusion Models (e.g., SDXL) can deal with such issues better. Considering that the power of Diffusion Models will get stronger, we believe the weakness of text-based guidance will be even more mitigated at the end of the day.

To summarize, the problem the reviewer pointed out (i.e., catastrophic neglect) can be mitigated by advanced sampling techniques [1-3] or by enhanced power of the base models (e.g., SDXL). Those techniques can be naturally combined with our proposed methods without degrading our technical novelties.

[1] Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models, SIGGRAPH 2023

[2] Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment, NeurIPS 2023

[3] Expressive Text-to-Image Generation with Rich Text, ICCV 2023

$^{1)}$ The problem we would like to solve is that pretrained large diffusion models are designed without considering any contextual bias which can be useful in multiple scenarios to control (c.f., L075 - L098 in the main paper). Thus, we suggest the way to obtain the contextual bias and the way to control it under the causal framework.

For this task, pre-storing 1TB of datasets seems excessively costly.

After preprocessing, we can have the 1,130,195 samples of $p(C')$ (c.f., Eq. 7 in the main paper), which occupies only 81 MB. The end-users can simply leverage the precomputed $p(C')$ as we will release it, which means they do not need 1TB storage for running the preprocessing stage. We apologize for the confusion.

It is essential for this work to demonstrate the necessity of this approach.

• Wrapping up the necessities of our methods that are mentioned in our paper:

  • As mentioned in L077-078 of our paper, CB+ could be useful if a given condition is not detailed enough to describe the whole scene. The generated sample can miss some objects that the user wanted to put together without directly mentioning, and CB+ can naturally autofill some visual components that have not been explicitly conditioned.

  • As mentioned in L085-088, if non-trivial and diverse object combinations are desired from the generated images, CB- can be a solution. This is because we can extrapolate the object combinations of the generated sample beyond the co-occurrence statistics.

• During the rebuttal, to more clarify the reviewer’s concern, we further provide the use-cases of CB- below:

While contextual bias may not be an issue for all users, we contend that in creativity use-cases, e.g., inspiring or augmenting the creativity of content makers, contextual bias could be an issue because it limits the creativity of the images. Particularly, we suggest that there are naturally at least two distinct stages in creative content generation using diffusion models: (1) Ideation and (2) Refinement. In the ideation stage, the content maker may only have a vague topic in mind as encoded by a simple prompt as a starting point, such as "an astronaut". Our CB- approach can then generate many diverse feasible scenarios around this prompt. Our diverse scenarios can then inspire new ideas for the content creator, such as "An astronaut coming across a huge dinosaur/elephant on an alien planet" or "A space-born girl exploring the Milky Way" (Fig 3 in the Supplementary Material). After the ideation stage, the content creator can refine the idea by manually augmenting and updating the prompt to arrive at their final creation, such as "A space-born girl exploring the Milky Way. She is excited for the upcoming visit to Earth for the first time of her life". Our CB- approach is primarily useful for the ideation stage. Once an idea is selected, manual prompt design can be used to refine the final created image but this is not useful for generating new ideas.

For example, suppose there is a person who needs creative thinking about a rabbit. This person could work in advertising companies, in the film or book industry, or could be a content creator. With existing sampling methods (Baseline 1 and 2 in Fig. 25 and 26 in the Supplementary Material), by prompting "a cute rabbit", it is not easy for him to get inspired and come up with interesting stories because most of the generated images are not diverse enough and very similar. With ours, however, a lot of interesting scenes can be generated, and he can easily come up with a lot of stories like "a musician rabbit playing the guitar", "a pet rabbit secretly watching a television when the owner is out", "an excited rabbit couple visiting a newly opened market in the town", "a rabbit traveling the forest wearing a backpack, slightly overwhelmed by an upcoming adventure", "a rabbit riding a mowing machine before inviting his fiance", "A rabbit comforting a girl crying in the living room", "A rabbit rowing a boat to tour the town", etc. After choosing one of them (e.g., "a musician rabbit playing the guitar"), he can add more details like "a musician rabbit wearing sunglasses and playing the guitar in the stadium with his band. He is playing rock music". As seen in this example, our proposed interventional sampling could be used to assist people to be creative. Once they get the idea, they can easily refine the prompt.

• (W3, W4) Modern diffusion models can easily generate unconventional combinations like "astronaut riding a horse on Mars". We heartily agree that modern diffusion models can easily generate unconventional combinations. Indeed, this assumption is the basis for our method and is exactly why we use pretrained diffusion models. Our goal is not to create a better diffusion model but to automatically control contextual bias. Or, to put it another way, our method can automatically generate creative and imaginative prompts. Your prompt "astronaut riding a horse on Mars" is rather a good example of why we need interventional sampling (Eq. 7 in the main paper). Let a prompt $y$ be "Astronaut on Mars" (already imaginative but not particularly creative) and a contextual bias $c'$ be "horse" and "man" (a very natural context). These seemingly unrelated $y$ and $c'$ combination successfully draws a creative and imaginative scene on top of the strong power of the modern diffusion models. However, Where did the prompt come from? A human came up with a prompt idea (astronaut on Mars) but then specifically chose words that would not usually co-occur even if they are natural in other contexts (i.e., man riding a horse). Yet, the human effort to create these prompts is not scalable. For example, suppose we need to make 1000 prompts to generate creative and novel scenes where an astronaut appears. We might be able to come up with 10-20 scenes quickly, but it is not necessarily easy to scale up. By leveraging our causally-grounded interventional sampling, however, we can generate thousands of novel scenes without any human labor because the retrieved contextual bias $c' \sim p(C')$ can extrapolate the co-occurring object combinations beyond data-driven correlation.

• (W1) Formulation is overcomplicated and unnecessary for addressing the problem at hand. In this paper, we explicitly model contextual bias using a causal framework. We emphasize that we aim for a principled and theoretically grounded approach to modeling contextual bias as opposed to a heuristic or ad-hoc approach. Thus, we first derive the appropriate equations from a principled viewpoint and then approximate these with foundation models. As opposed to the regular diffusion sampling method, our causal framework enables to strengthen the contextual bias (to assist to autofill the scene) or to weaken the contextual bias (to draw a novel scene with non-trivial object combinations). We thoroughly checked again our derivation and formulation and believe that our formulations in Eq. 3 and 7 are not overcomplicating nor unnecessary. We would appreciate it if the reviewer could provide a simpler but principled way to address the problem of contextual bias under a causal framework.

• (W2) Novelty of the proposed method ---Retrieving co-occurring objects using LLMs and identifying objects appearing in images using VLMs is trivial and has been commonly practiced in the task of text-to-image generation. While individual steps in our method like retrieving co-occurring objects has likely been used before (though we would appreciate explicit references), we cannot find any work that uses these steps in a principled way to address contextual bias in a causal framework. Our novelty stems from how we derive and integrate these steps. The approximations we use via LLMs or VLMs are empirically justified by our strong results. Again, we emphasize that our novelty does not stem from the individual steps but from the principled derivation and specific combination and integration of those steps. If we have missed related works that develop a principled method for handling contextual bias, we would appreciate specific references so that we can improve our paper.

• (W1) The proposed methods boils down to a refined form of prompt engineering. We respectfully disagree with both the implied claim that prompt engineering is not valuable and that our method is merely prompt engineering. First, we argue that prompt engineering itself has great value, and there are many highly cited research papers (e.g., [1-3]) directly related to prompt engineering. Thus, we believe your implicit assumption that prompt engineering is unworthy to be published is unfounded. Second, we argue that there are at least two independent components of our method. The first is the principled implementation-agnostic derivation of handling contextual bias using a specific form of interventional sampling. The second is the actual implementation of our interventional sampling by using prompt engineering. The mere fact that we can use prompt engineering as a way to approximate the necessary sampling should not be viewed as a weakness. To further emphasize the independence of the methodological contributions, we implemented a method for the conditional sampling, i.e., the conditional $p\_{\theta}$ X|y, C'=c' $$, that does not involve prompt engineering. To be specific, since $y$ and $c'$ encode fundamentally different information, we can apply a sampling method Generalized Composable Diffusion Models (GCDM) [4] and incorporate $y$ and $c'$ in the score space during the denoising process. By doing so, we can control the tradeoff between diversity due to $c'$ and the original prompt $y$. Please see the Additional experiment results 2 in "Response to all reviewers".

[1] Language Models are Unsupervised Multitask Learners, 2019

[2] Language models are few-shot learners, NeurIPS’20

[3] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS’22

[4] Enhanced Controllability of Diffusion Models via Feature Disentanglement and Realism-Enhanced Sampling Methods, ECCV'24

We first address the core concern of when there is unwanted bias. Then we will provide extensive qualitative and quantitative results demonstrating that prompt engineering approaches including negative prompting cannot exhibit the creativity yet realism of our approach.

I still have concerns regarding the core assumption that contextual bias is a significant issue for image generation using diffusion models.

While contextual bias may not be an issue for all users, we contend that in creativity use-cases, e.g., inspiring or augmenting the creativity of content makers, contextual bias could be an issue because it limits the creativity of the images. Particularly, we suggest that there are naturally at least two distinct stages in creative content generation using diffusion models: (1) Ideation and (2) Refinement. In the ideation stage, the content maker may only have a vague topic in mind as encoded by a simple prompt as a starting point, such as "an astronaut". Our CB- approach can then generate many diverse feasible scenarios around this prompt. Our diverse scenarios can then inspire new ideas for the content creator, such as "An astronaut coming across a huge dinosaur/elephant on an alien planet" or "A space-born girl exploring the Milky Way" (Fig 3 in the Supplementary Material). After the ideation stage, the content creator can refine the idea by manually augmenting and updating the prompt to arrive at their final creation, such as "A space-born girl exploring the Milky Way. She is excited for the upcoming visit to Earth for the first time of her life". Our CB- approach is primarily useful for the ideation stage. We totally acknowledge that once an idea is selected, manual prompt design can be used to refine the final created image but this is not useful for generating new ideas.

For example, suppose there is a person who needs creative thinking about a rabbit. This person could work in advertising companies, in the film or book industry, or could be a content creator. With existing sampling methods (Baseline 1 and 2 in Fig. 25 and 26 in the Supplementary Material), by prompting "a cute rabbit", it is not easy for him to get inspired and come up with interesting stories because most of the generated images are not diverse enough and very similar. With ours, however, a lot of interesting scenes can be generated, and he can easily come up with a lot of stories like "a musician rabbit playing the guitar", "a pet rabbit secretly watching a television when the owner is out", "an excited rabbit couple visiting a newly opened market in the town", "a rabbit traveling the forest wearing a backpack, slightly overwhelmed by an upcoming adventure", "a rabbit riding a mowing machine before inviting his fiance", "A rabbit comforting a girl crying in the living room", "A rabbit rowing a boat to tour the town", etc. After choosing one of them (e.g., "a musician rabbit playing the guitar"), he can add more details like "a musician rabbit wearing sunglasses and playing the guitar in the stadium with his band. He is playing rock music". As seen in this example, our proposed interventional sampling could be used to assist people to be creative. Once they get the idea, they can easily refine the prompt, as the reviewer pointed out.

Table 3: Performance comparisons of new baselines, suggested by the reviewer.

Table 4: The engineered prompts by the baselines given a prompt of ``Photo of an astronaut''. (0,0) means the first row and the first column of the corresponding figures (i.e., Fig. 3 and Fig. 4 in the Supplementary Material).

I would need to see a more direct and compelling comparison of your method’s effectiveness against baseline methods like prompting or prompting with negative prompts, both quantitatively and qualitatively, to justify the need for your approach.

• We conducted additional experiments with the suggested baselines to verify the advantage of our proposed interventional sampling over the existing methods. The results are reported both quantitatively and qualitatively.
• Experiment settings:
  • Baseline 1 is implemented with simple prompt engineering. Before being used as a condition to SDXL, a prompt is engineered by Gemini with a query of f"briefly modify the given prompt to be creative. Answer in one sentence.: {prompt}". The answer is used as a text input.
  • Baseline 2 is implemented with a combination of prompt engineering and negative prompt techniques. A prompt is engineered by Gemini with a query of f"What would be the frequently co-occurring objects that can be likely placed in the scene generated by the given prompt ‘{prompt}’? Do not answer the words mentioned in the prompt. Answer 10 objects except for {important_obj} one line with comma.", where important_obj is manually predefined as a key object from the prompt, e.g., ''ancient dragon'' from ''A fantasy illustration of ancient dragon''. Once the co-occurring objects are obtained, we use them as a negative prompt during the sampling process to remove contextual bias, as suggested by the reviewer.
  • We use 13 queries. We randomly sample 20 samples per prompt (and per setting, e.g., baselines and ours) and show all of them without cherry picking.
• Results: The image results are shown in the Supplementary Material. We can see that the results from our interventional sampling show better performance in creative scene generation than the baselines. For example, from Fig. 1,3, and 5, the Baseline 1 results such as an astronaut in space and photo of Mars and artistic autumn forest in diverse viewpoints could be considered creative. However, it is limited in generating diverse creative scenes as most of the images contain similar object compositions (c.f., the lowest LPIPS score in Table 3 below). We believe it is not suitable for helping and inspiring humans to be creative. In the case of Baseline 2, from Fig. 10 and 20, we can see that the negative prompt technique removes the contextual bias related to the prompt, e.g., the removed snow in Fig. 10 and 20. However, the diversity is still limited as it is not exactly modeling the interventional distribution from causal perspective, and thus the backdoor path in Fig. 3 in the main paper remains and negatively affects the image generation process. On the other hand, our results show impressive performance in generating diverse and creative scenes. For example, from Fig. 3, we can see an astronaut standing in front of a dinosaur, an astronaut walking through a street with debris, an astronaut standing next to huge elephants, an astronaut standing on the river, etc. (c.f., the highest LPIPS score in Table 3 below). Additionally, we also provide the qualitative prompt-engineered results in Table 4 below. Please let us know if the reviewer wants to see additional engineered prompts for the specific figures.

• (W2) Clarifications of the used models for the figures in the paper. SD 2.1 is used for Fig. 2 and 4 and SD 1.5 is used for Fig. 6. SDXL is used for Fig. 7.

• (W3) FID results with SDXL for fair comparisons. To resolve the concern of the reviewers about the fair comparisons, we conduct additional experiments to measure FID based on SDXL. Please see the Additional experiment results 1 in "Response to all reviewers".

• (W4) How is $c'$ incorporated in the image generation process? In the main paper, to model $p\_{\theta}$ x|y, c' $$, $c'$ is directly added in the prompt space. For example, if $c'$ is {chair, lamp, couch} and $y$ is A photo of a restroom, the input condition is A photo of a restroom, chair, lamp, couch. One step forward, during the rebuttal period, we explore a sampling method Generalized Composable Diffusion Models (GCDM) [1] to implement the CB- context conditioning in a different way than simple text concatenation. Please see the Additional experiment results 2 in "Response to all reviewers".

  [1] Enhanced Controllability of Diffusion Models via Feature Disentanglement and Realism-Enhanced Sampling Methods, ECCV'24

• (W4, Q2) Clarifications on modeling $p(c')$ We apologize for the confusion. (Q2 (a)) As mentioned in L282-284, 1,130,195 samples are used to get $p(c')$. (Q2 (b)) Yes, empirically it is very diverse as more than one million of unconditionally generated images are used to retrieve $p(c')$. We further provide pseudo code for explaining the process more fully. Please see the Clarifications in "Response to all reviewers".

• (Q1) Discussion on the reviewer's suggestion and why it may or may not work compared to CB-. The reviewer suggested another way to obtain $c'$ by asking VLM about "given this image, describe a list of common nouns that do not occur in this scene, but could be reasonably expected to co-occur and increase the diversity". This would correspond to something like sampling from the conditional $p(c'|x,q)$, where $q$ is the question and $x$ is the image. First of all, the input setting is different from ours because our task is text-guided image generation which means we do not have an image input $x$. Perhaps, you meant that we first generate an initial image from the original prompt $y$, then ask the VLM for new objects, and then regenerate with a new prompt. This could be approximately formalized as $p(c'|y) \approx \mathbb{E}_{x}$ p(c'|x,q)p(x|y) $$. Note that $c'$ depends on $y$, which is specifically what our interventional distribution aims to avoid. Furthermore, it is unclear if this has any proper causal interpretation. While this heuristic approach may increase diversity, it does not provide a way to disentangle context in a principled way as our approach does. Please note that the interventional (that we model in the paper) can associate unrelated objects and generate creative scenes, e.g., a tree in a bedroom.

• (Q3) Implementation details about ControlNet and DEADiff To implement Fig. 6, we add the contextual bias $c'$ and the text prompt $y$ in the prompt space. Since both ControlNet and DEADiff take a text prompt as input, we simply sample $c'$ from the precomputed $p(c')$, and add it to the prompt $y$. The new prompt (combining $c'$ and $y$) is then fed into the networks.

I thank the authors for their detailed response to my review. Here are some follow-up questions from the rebuttal:

R0. Can the authors please provide a clear indication of what was changed in the revision from the original submission? Without this, it is very time consuming to identify if previous weaknesses / questions have been addressed in the writeup.

R1. The details of models used in all experiments and figures need to be clearly mentioned in the paper, not just in the rebuttal, unless I missed this in the revision (see R0).

R2. I appreciate the additional experiments on FID with SDXL which extend previous results on a weaker diffusion model - this largely addresses W3. I would still like a discussion on the pros and cons of choosing other contextual biases (Llava and Gemini) to override and steer the existing contextual biases of the LDM.

R3. The details of incorporating $c'$ are now clear to me from the rebuttal (text concatenation to the prompt), but this is still totally unclear from the main paper and needs to be included, unless I missed this in the revision. I appreciate the additional experiment with a different sampling method (GCDM) - my takeaway from these results is that boosting the contribution of retrieved confounder $c'$ during guidance increases diversity, which seems intuitively correct if $y$ and $c'$ encode different information about the image. I suggest including this experiment in the Appendix to further highlight this point.

R4. What are the 1,130,195 samples used to learn p(C')? Since any added diversity depends on the diversity present in this data, it would be nice to have some details about what kind of data this is (how many classes? what kind of classes? do these samples overlap with pretraining distributions of any LDMs /Llava ?)

R5. I remain unconvinced and slightly confused by the authors' response to Q1. In Eq 6, the sampling chain begins by conditioning on unconditionally generated images $x'$, which the authors describe with Llava to get $y'$, which is then used as conditioning to generate $x''$, which is again described by Llava to obtain objects $c'$. My question is, why not just use Llava's learned contextual bias to describe potentially co-occurring objects $c'$ from the original unconditionally generated image $x'$, instead of doing this entire sampling chain? Here, $c'$ depends only on $x'$ and not $y$, preserving the goal of your interventional distribution. Please correct me if I have misunderstood Eq 6 and this process.

R6. It is not obvious to me that the proposed method has a "proper causal interpretation" from the paper. This would be more clear if we knew the distribution (i.e. training data) used to learn $c'$ and how it differed from the $y \to x$ mapping learned by the LDM. Additionally, I need more justification/discussion of the claim that $y$ and $c$ are truly causal of generated image $x$, and not just commonly co-occurring. In my understanding, $x$ and $y$ co-occur frequently in the pretraining distribution of the LDM, and $x$ and $c$ co-occur in the pretraining distribution of the LDM (unconditional) combined with the associations learned by Llava. Please correct me if I have missed something.

In my view, the main paper is still lacking details and needs some work to improve clarity and readability. With the additional experiments, pseudocode, and clarifications provided in the rebuttal, I increase my score to a 7 (which unfortunately I cannot actually do). I would further increase my score to 8 if the missing details (R1 - R4) and my queries above (R5, R6) are addressed directly in the main paper. With this in mind, I will currently maintain my score.

(R0, R1, R3) Paper revision

We appreciate the reviewer for his/her kind advice to revise our paper. We agree with the reviewers’ points; 1) mentioning what SD versions are used per figure, 2) adding more contents/details to the captions for the figures to make them self contained, and 3) including the GCDM experiment to the Appendix. As it already has passed the deadline to directly update the pdf, we will thoroughly revise and update the pdf accordingly for the camera-ready version of our paper.

(R2) I would still like a discussion on the pros and cons of choosing other contextual biases (Llava and Gemini) to override and steer the existing contextual biases of the LDM.

• Pros: naively using the existing contextual biases of LDM is actually simply the regular diffusion sampling (i.e., Fig. 3 (b) in the main paper). To model $p(C’)$ though, we essentially leverage VLM (in the case of CB-) and LLM (in the case of CB+). By leveraging VLM or LLM, we could approximate contextual bias $p(C')$ and control them to strengthen/weaken the contextual bias effects.

• Cons: $p(C’)$ is an approximation of $p(C)$, which means it is not exact $p(C)$. However, we would like to emphasize that the approximating step is essential as we do not have access to all the training data (as mentioned in L104-L110 in our paper). Specifically, if we can have access to all of the training data for the pretrained Diffusion Models (i.e., billions of image $x$, its paired caption $y$), and if we can manually label all of co-occurring objects $c$ for each image, we can obtain the samples from true $p(C)$ which is more accurate than our approximated $p(C’)$. Since it is infeasible in general, we approximate $p(C’)$ by leveraging LLM or VLM.

(R5) why not just use Llava's learned contextual bias to describe potentially co-occurring objects $c'$ from the original unconditionally generated image $x'$, instead of doing this entire sampling chain? Here, $c'$ depends only on $x'$ and not $y$, preserving the goal of your interventional distribution.

What the reviewer suggested is one of the possible implementations of Eq. 7 in our main paper:

$p(X|\text{do}(Y=y)) \approx \mathbb{E}_{c' \sim p(C')}$ p_{\theta} $X|y, C'=c'$ $$, where $p(C') \approx \sum\_{x'} p\_{\phi} (C'=c'|X=x') p\_{\theta}(X=x')$. However, as mentioned in [1], we observe clearly degraded performance of unconditionally generated images from Diffusion Models, and thus we add one more step to get more accurate co-occurring objects information, i.e., get the prompt $y’$ first from the unconditional generation and next get the co-occurring objects $c’$ from conditionally generated images.

[1] the second paragraph of the introduction, “Return of Unconditional Generation: A Self-supervised Representation Generation Method”, NeurIPS’24

(R4) What are the 1,130,195 samples used to learn p(C')? Since any added diversity depends on the diversity present in this data, it would be nice to have some details about what kind of data this is (how many classes? what kind of classes? do these samples overlap with pretraining distributions of any LDMs /Llava ?)

As specifically shown in the pseudo code in "Response to all reviewers", we obtain the 1,130,195 samples of $p(C')$ by following the derived sampling chain. Some examples of $c' ~ \sim p(C')$ could be ["car", "trees", "flowers"], ["couch", "chairs", "table", "kitchen appliances", "island"], ["car", "cow", "man", "dog"], and ["bench", "trees", "people", "car"], where each list contains actually co-occurring objects of diffusion models.

As for the details, we provide the highest and the lowest 100 object count matrices over the 1,130,195 samples in Table 3 and 4. The example for how to count each object is that, given the examples above, the count of "car" is 3 and the count of "trees" is 2. The vocab size (i.e., # of words) is 48774.

As for the overlap with pretrained LDMs and LLaVA, since LDMs and LLaVA are leveraged as core steps to get the samples for $p(C')$ in our proposed sampling chain (c.f., Eq. (6) of our paper, and the pseudo code in the Response to all reviewers), we believe the pretrained knowledge of LDMs and LLaVA is the fundamental ingredient of the retrieved $p(C')$.

(R6) It is not obvious to me that the proposed method has a "proper causal interpretation" from the paper. This would be more clear if we knew the distribution (i.e. training data) used to learn $c′$ and how it differed from the $y \rightarrow x$ mapping learned by the LDM. Additionally, I need more justification/discussion of the claim that $y$ and $c$ are truly causal of generated image $x$, and not just commonly co-occurring. In my understanding, $x$ and $y$ co-occur frequently in the pretraining distribution of the LDM, and $x$ and $c$ co-occur in the pretraining distribution of the LDM (unconditional) combined with the associations learned by Llava. Please correct me if I have missed something.

We first would like to define the meaning of each variable with some examples. We next describe the causal relationships between the variables. (c.f., Section 3.1 in the main paper)

$Y$ is text prompt and $X$ is image. $C$ is an unobserved confounding variable that we would like to approximate by $C'$. In our paper, the confounder $C$ indicates co-occurring objects statistics in our visual world. Suppose we have some paired samples $(Y=y,X=x,C=c)$ for each variable. Some examples would be ($y=$''Photo of street'', $c=$["car", "people", "traffic light", "traffic sign"], $x_{\text{street}}$), where $x_{\text{street}}$ is a corresponding street image, and ($y=$''Photo of living room'', $c=$ ["couch", "lamp", "rug", "window", "plant"], $x_{\text{living room}}$).

As mentioned in [2], conceptually, the object co-occurrence statistics $C$ could affect either $X$ or $Y$ because $X$ and $Y$ are realizations of the visual world in different modalities (i.e., image and text). For example, there could be a tendency that "couch", "lamp", "rug", and "window" typically co-occur in the "living room", which could affect the data generation process. In other words, most living room images or most image descriptions with a word "living room" could keep containing the frequently co-occurring objects ("couch", "lamp", "rug", and "window"). These co-occurrence statistics implicitly but likely make the pretrained diffusion models generate an image with ("couch", "lamp", "rug", and "window") given a prompt "living room". Thus, our proposed causal graph has the edges $C \rightarrow X$ and $C \rightarrow Y$. The edge $Y \rightarrow X$ comes from the pretrained text-guided diffusion models that take $Y$ as an input and output $X$. (c.f., Fig. 3 in the main paper)

Now, we retrieve a confounder $C'$ by approximating the inaccessible hidden confounder $C$ and control it following CB+ and CB- (c.f., Fig. 3 (a), (c) in our paper).

[2] Fig. 5, Visual Commonsense R-CNN, CVPR 2020

Table 4: Objects with minimum counts over 1,130,195 samples of $p(C')$, thresholded by 100.

Table 3: Objects with maximum counts over 1,130,195 samples of $p(C')$.

• (W1-1) Generated images can ignore the confounder. As we mentioned in Limitations from the main paper, the phenomenon that diffusion models could ignore either one of $C'$ of $Y$ is observed, if they are semantically too far. However, we believe this is not a fundamental issue of our proposed sampling methods but rather a limitation of current diffusion models. As the power of diffusion models increases, this phenomenon would be reduced.

• (W1-2) The framework’s dependence on predefined confounders may limit its flexibility. Thank you for pointing out a valid point. Yes, our predefined $p(C')$ might be limited in representing all the possible object-cooccurrence information. However, it contains fairly rich information as $p(C')$ is precomputed over 1130195 samples. Moreover, based on Eq. 7 in the main paper (i.e., $\mathbb{E}_{c'}$ p_{\theta} $X|y, C'=c'$ $$), our interventional sampling method generates an image by combining $y$ with $c'$, which means the generated images contain both $c'$ and $y$, mitigating the limitation of $p(C')$.

• (W2) It is true that the derived sampling chain in Eq. 6 of the main paper is expensive. To make it up, as mentioned in L282-L287 of the main paper, we precompute $p(C')$, and this precomputation significantly reduces the sampling time. Without precomputing, it takes 12 seconds to sample $c'$ while it takes less than 1 second to sample $c'$ from the precomputed $p(C')$. Thus, we can confirm that the sampling cost of $c'$ does not affect the scalability/adaptability of our interventional sampling method.

• (W3) Interestingly, our experiment results show that FID from CB- (weakening contextual bias) is consistently the best. Table 1 in the main paper is based on SD 2.1 and Table 1 shown in the "Response to all reviewers" is based on SDXL. We believe this is because FID measures both quality and diversity, and thus increasing diversity appropriately can improve FID.