SG-Adapter: Enhancing Text-to-Image Generation with Scene Graph Guidance

Thank you for your thoughtful and constructive feedback. Your comments have been invaluable in helping us refine and improve our study. Below, we address your concerns in detail.

---

Weaknesses

Weakness 1, 3, 4: Dataset Size and Comparative Evaluation

Response: Thank you for your suggestions. In addition to experiments conducted on our MultiRels dataset, we have conducted extensive evaluations on Flickr30k[1] to demonstrate the robustness and generalization performance of our method. Please refer to Section 4.6 (L420-L465), Table 3, and Figure 4 for these results.

During the rebuttal period, we have additionally included R3CD (AAAI24) and ISGC (24) in Table 3 to provide a more comprehensive comparative analysis. The updated results are as follows:

For R3CD, ISGC, and SceneGenie, we report inception scores and FID values directly from their respective papers, as their code is not available. These additional results help establish a more robust and fair comparison.

---

Weakness 2: Adaptor vs. Stable Diffusion Fine-tuning

Response: Our analysis identified the primary source of failure as the causal attention process in CLIP. Consequently, our focus has been on rectifying this issue.

We also conducted experiments where only the parameters of the Stable Diffusion (SD) UNet were fine-tuned. However, this yielded suboptimal results, reinforcing the importance of addressing causal attention in the CLIP text embeddings. This focus ensures that our approach remains efficient and targeted, avoiding unnecessary changes to the Stable Diffusion backbone.

---

Weakness 5: Human Evaluation Sample Size

Response: Compared to previous works, our method collects human evaluation data on a much larger scale. For instance:

• Attend&Excite[8] gathered 65 responses across 10 prompts using 4 methods, resulting in 2,600 interactions.
• Dreambooth[9] collected 1,800 interactions with 72 responses and 25 prompts.

In contrast, our study involved 104 responses for each of 20 text prompts across 5 comparison methods, yielding a total of 10,400 interactions. This expanded dataset significantly enhances the reliability and robustness of our evaluation, providing a strong foundation for our conclusions.

We acknowledge the importance of expanding the evaluation set further and will consider additional test cases in future work to address more diverse scenarios comprehensively.

---

We hope these detailed clarifications and additional experimental results address your concerns thoroughly. Thank you again for your valuable feedback, which has greatly helped us strengthen our work.

[1] Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence model.
[2] Image generation from scene graphs.
[3] Pastegan: A semiparametric method to generate image from scene graph.
[4] Diffusion-based scene graph to image generation with masked contrastive pre-training.
[5] Scene graph to image synthesis via knowledge consensus.
[6] R3cd: Scene graph to image generation with relation-aware compositional contrastive control diffusion.
[7] Scene graph to image synthesis: Integrating clip guidance with graph conditioning in diffusion models. [8] Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models. [9] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

Dear Reviewer wQ1r,

We sincerely appreciate your time and effort in reviewing our submission and writing your responses.

However, as the discussion deadline approaches in less than 2 days (only 1 day left for reviewers to post a message), we would greatly appreciate it if you could carefully review our latest responses and provide further clarifications at your earliest convenience. We highly value any opportunities to address any potential remaining issues, which we believe will be helpful for a higher rating. Your timely feedback is extremely important to us!

Once again, thanks so much for your time and effort, we deeply appreciate it!

Best Regards,

Authors of Submission 1595

We appreciate your recognition and valuable feedback on our work. Your comments are insightful and have helped us refine and improve our study. Below, we address your concerns and suggestions in detail.

Weaknesses

Weakness 1: Limited Novelty in Method Design

1. The Importance of Scene Graph Information:
   Our analysis highlights that incorporating scene graph information is essential for rectifying false contextualization caused by erroneous causal attention. Furthermore, how scene graph information is utilized significantly impacts performance.

2. Utilizing Scene Graph Information Effectively:
   We explored alternative methods, such as inserting scene graph embeddings via gated self-attention. However, this approach failed to effectively address the errors, as shown in the third row of Table 1 in the main paper. In contrast, our adaptor demonstrates superior correction capabilities, as validated by the experimental results. This demonstrates the importance of both the adaptor structure and the specific way in which scene graph information is incorporated.

Weakness 2: Experimental Evaluation of Entity Interaction Generation is Not Comprehensive

Instead of relying on traditional scene graph generation methods, which are often trained on limited datasets and prone to long-tail distribution issues, we leveraged GPT-4V to extract scene graphs. Recent research ([1], [2]) combining large language models (LLMs) with scene graphs demonstrates that LLM-based methods, due to their strong generalization capabilities, are less affected by the long-tail distribution problem.

• References:
  1. LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation
  2. GPT4SGG: Synthesizing Scene Graphs from Holistic and Region-Specific Narratives

---

Questions

Question 1: Generating Complex Interactions

To address this concern, we have included additional results in Appendix A.7 Fig.7, covering cases with more complex interactions. For example:

• Crowded scenarios, such as a group of astronauts.
• Uncommon pairs, such as "hamster riding on motorbike."

These cases demonstrate that our SG-adaptor effectively learns general interaction relationships beyond high-frequency combinations in the training set.

Question 2: Scene Graph Detection Conflicts with Text Prompts

Our framework focuses exclusively on entities and their relationships. For unrelated prompt elements such as "4K," "realistic," or specific lens types, we provide the flexibility to process text using either GPT-4 or an NLP parser. For handling lengthy or complex prompts, GPT-4 can be used with additional instructions such as:

• "Ignore prompts unrelated to entities and relationships."

This ensures that the scene graph detection process remains focused on the core elements relevant to the task, mitigating potential conflicts with extraneous text.

Question 3: Impact of Adaptor Parameter Size

We have conducted an analysis of how performance varies with the size of the parameters in the adaptor. Detailed results are provided in the following table.

The findings show that as the number of parameters increases:

• FID worsens (higher values).
• Alignment with Scene Graph slightly improves.

We have chosen a balance point between FID and scene graph alignment to ensure optimal performance without over-parameterization.

---

We hope these clarifications and additional experiments address your concerns. Thank you for your thoughtful feedback, which has helped us strengthen our work.

Dear Reviewer Bw61,

We sincerely appreciate your time and effort in reviewing our submission and writing your responses.

However, as the discussion deadline approaches in less than 2 days (only 1 day left for reviewers to post a message), we would greatly appreciate it if you could carefully review our latest responses and provide further clarifications at your earliest convenience. We highly value any opportunities to address any potential remaining issues, which we believe will be helpful for a higher rating. Your timely feedback is extremely important to us!

Once again, thanks so much for your time and effort, we deeply appreciate it!

Best Regards,

Authors of Submission 1595

We appreciate your recognition and valuable feedback on our work. We apologize for the issue of the citation format and have fixed it in the updated manuscript. Your suggestions on our dataset and experiment are helpful in improving our work. Detailed responses are provided as follows:

Weaknesses.a

Response: Thanks for your comment we have rectified the format of citations in our manuscript.

Weaknesses.b

Response: We construct such a high-quality dataset to intuitively demonstrate the effectiveness of our method in generating multiple relationships with accurate correspondences. The relation combinations in our experiment part do not appear in our training dataset, showing that our method has a certain generalization ability. Additionally, as we illustrated in Section.4.6, our method could also be extended to large-scale datasets like Flickr30k[1] and show consistent improvement.

Weaknesses.c

Response: Thanks for your valuable suggestion. Besides the results on our MultiRels, we have included experimental results that were trained on the large-scale dataset Flickr30k. You can see Section.4.6 L420-L465, Tab.3 and Fig.4 for reference. During the rebuttal period, we provided additional results in Appendix A.7 to show our model's generalization performance and ability to generate multiple entities. To better show the effectiveness of our method, we extended Tab.3 as follows:

For R3CD, ISGC and SceneGenie, we just report the inception score and FID from their paper since they do not provide the code.

Questions.1

Response: Yes. The SG-Adapter could alleviate the color leakage problem since we could regard color as a specific relationship, e.g., the caption "a blue cat" could correspond to the relation ["a cat", "is", "blue"]. We have provided this case in Appendix A.7.

---

We hope these clarifications and additional experiments address your concerns. Thank you for your thoughtful feedback, which has helped us strengthen our work.

[1] Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence model.
[2] Image generation from scene graphs.
[3] Pastegan: A semiparametric method to generate image from scene graph.
[4] Diffusion-based scene graph to image generation with masked contrastive pre-training.
[5] Scene graph to image synthesis via knowledge consensus.
[6] R3cd: Scene graph to image generation with relation-aware compositional contrastive control diffusion.
[7] Scene graph to image synthesis: Integrating clip guidance with graph conditioning in diffusion models.

Dear Reviewer 3Tge,

We sincerely appreciate your time and effort in reviewing our submission and writing your responses.

However, as the discussion deadline approaches in less than 2 days (only 1 day left for reviewers to post a message), we would greatly appreciate it if you could carefully review our latest responses and provide further clarifications at your earliest convenience. We highly value any opportunities to address any potential remaining issues, which we believe will be helpful for a higher rating. Your timely feedback is extremely important to us!

Once again, thanks so much for your time and effort, we deeply appreciate it!

Best Regards,

Authors of Submission 1595

We appreciate your commendation and valuable feedback on our work. We apologize for any confusion caused by the figures and notations. Below are the clarifications for your questions, and we have updated the corresponding parts in the manuscript accordingly:

Questions 1-4, 7

Response: We sincerely appreciate your suggestions regarding naming, formatting, and figure optimization. These improvements have been incorporated into the revised manuscript for your reference.

Regarding the naming of each attention mask:

• For SG Adaptor: $\mathbf{M}^{sg}$ represents the attention mask used by our adaptor, which leverages scene graph information to refine the embedding of each token. We refer to this as the Token-Triplet Attention Mask.
• For CLIP Text Encoder: The other two attention masks, $\mathbf{M}$ and $\mathbf{M}^{\tau}$, are associated with the CLIP text encoder. Specifically:
  • $\mathbf{M}^\text{causal}$: The default causal attention mask.
  • $\mathbf{M}^{\tau}$: A modified attention mask informed by scene graph information, referred to as the Triplet-Triplet Attention Mask.

Question 5

Response:
Given the role of our adaptor $f(\cdot)$, the attention mask should be $\mathbf{M}^**sg**$. The previous use of $\mathbf{M}$ was a placeholder. To avoid confusion, we have replaced all instances of $\mathbf{M}$ with $\mathbf{M}^**sg**$ throughout the manuscript.

Question 6

Response:
Table 2 and Line 413 discuss the ablation study of our proposed SG Mask $\mathbf{M}^{sg}$. The mask $\mathbf{M}^{\tau}$ addresses the limitations of causal masks by rectifying contextualization issues within the CLIP forward process. However, as demonstrated in Section 3.1 and Appendix A.1, directly applying $\mathbf{M}^{\tau}$ in the CLIP forward process does not yield the desired results.

To address this, we designed an adaptor and derived a new attention mask, $\mathbf{M}^{sg}$, based on the same underlying idea. This mask is specifically tailored to rectify the output of the CLIP text encoder, ensuring improved performance and adaptability.

---

We hope these clarifications and additional experiments address your concerns. Thank you for your thoughtful feedback, which has helped us strengthen our work.