T$^3$-S2S: Training-free Triplet Tuning for Sketch to Scene Generation

Dear Reviewer wtqN,

Thank you for your constructive and insightful comments for helping us improve our paper. Please see below our response to your specific questions.

Q1: Transfer prompt balance to other multi-instance T2I scenarios, such as AAE and StructureDiff.

A1: Thank you for the suggestion. Following your suggestion, we have added experimental results on transferring the Prompt Balance module to Attend-and-Excite. The results are shown in Figure 10 of Appendix F. Despite the limitations of SD V1.4, the PB module effectively balances the embedding strength between instances without layout guidance, enhancing their representation. While both instances are successfully generated in most cases, some attributes become entangled, resulting in artifacts like a cat with frog legs or a pig with bird wings. This nonetheless demonstrates the PB module's effectiveness, given the lack of spatial separation for the instances. We will explore other experiments when time is allowed.

Q2: Explanation for the CP module and the same type of multiple instances with different attributes.

A2: (1) There may be a misunderstanding here. The setting of $K$ is not strictly tied to the number of instances. In a feature space with an embedding of 1024, corresponding to 1024 feature maps, the top 2 strongest patterns in all maps can encompass all instances, especially after Prompt Balance, which significantly increases the likelihood of instance selection. This is discussed in our Top $K$ analysis in Appendix B, where we provide statistics on the probability of each instance being selected. Lines 521-527 also mention that increasing $K$ initially improves generation quality but eventually introduces more noise, as shown in Figure 8.

(2) Generating instances of the same type with different attributes is indeed challenging for multi-instance synthesis. To verify this, we have designed three prompts, including one group of five colored roses and two groups of five colored cats. As shown in Figure 14 (c), our approach accurately assigns distinct properties to identical instances. However, for uncommon attributes like a red cat, the model struggles due to limitations in the original SDXL, though it successfully generates a green cat as prompted.

Q3: Details of characteristics prominence, whether "applied on cross_attention(I, T)" or "on I + cross_attention(I, T)".

A3: Thank you for the careful reading. Our characteristics prominence module is applied on cross attention(I, T). To avoid confusion, we have added the explanation in Line 335 to make it clearer.

Q4: Quantitative experiments and user study.

A4: Thank you for the suggestion. Given that our current approach involves sketch-based multi-instance scene generation, existing benchmarks should be adjusted for our evaluation, such as adding sketch inputs for T2ICompBench (Huang et al., 2023). Therefore, we have designed 20 sketches of complex scene composition, each with more than four sub-prompts, encompassing various terrains (plains, mountains, deserts, glaciers, cities) and diverse instances (rivers, bridges, stones, castles). We have utilized CLIP-Score (Hessel et al., 2021) for the global prompt and image, and evaluated the CLIP-Score for each background prompt and instance prompts by cropping the corresponding regions. Additionally, we have conducted a user study to assess different variants of our approach, using a 1-5 rating scale to evaluate image quality, placement, and prompt-image consistency. Details can be found in "Metrics" of Section 5.1, "Quantitative Evaluation" of Section 5.2, and Appendix E. The results of quantitative experiments and user studies are shown below, both of which demonstrate good improvements of our T$^3$-S2S approach over the ControlNet model and variants:

Best regards,

Authors

Dear Reviewer,

I hope this message finds you well.

First of all, I would like to express my heartfelt gratitude for the detailed and insightful comments you provided during the review process. From your feedback, we can clearly see that you thoroughly read our manuscript and offered highly constructive suggestions, which have been immensely valuable in improving our work. Following your comments, we have conducted additional experiments and made significant revisions to our draft, which we believe have greatly strengthened the manuscript.

We submitted our rebuttal and the revised manuscript a few days ago and are eagerly awaiting your thoughts. Your feedback is crucial not only for ensuring that all concerns are addressed but also for presenting our work in the best possible light.

It is especially encouraging to know that you are currently one of the reviewers who has given us a positive evaluation, and your perspective gives us hope. We deeply appreciate your constructive input and would be incredibly grateful if you could continue to support us by reviewing the revisions and providing further encouragement. Please trust in your judgment, as it is vital for us during this critical stage.

Thank you again for your time and effort in guiding us through this process. We look forward to hearing your thoughts soon.

Best regards,

Authors

Dear Reviewer wtqN,

Thank you for your thoughtful response and for addressing my concern.

This is the best news we’ve received recently, and we truly appreciate your decision to increase the score to 8.

We are committed to open-sourcing our code in the future, as we believe it will greatly benefit the development of the community. Your encouragement further motivates us to make this happen.

Thank you once again for your support and recognition!

Best regards,

Authors

Thanks to the authors for their serious reply, which solved my concern.

I decided to raise my score to 8.

In addition, I hope the authors can open-source the code in the future, which will be beneficial to the development of the community.

Dear Reviewer 7JUo,

Thank you for your constructive and insightful comments for helping us improve our paper. Please see below our response to your specific questions.

Q1: The clarity of the writing and notation

A1: Thank you for the helpful comment. We have improved the clarity, presentation and notation to make our contributions more clearly.

Q2: Comparison of specific Inference time.

A2: Thank you for your constructive comment. All our operations are performed using parallel matrix calculations, and our approach remains unaffected by the number of instances, ensuring stable speed. The inference time for a single prompt increases slightly from 18s to 29s on a single Nvidia V100 GPU. There is still room for optimization, such as converting matrix operations to lower-level CUDA operations, which could improve speed. We plan to explore this optimization in the future.

Q3: Quantitative experiments and user study.

A3: We agree with you that it may be challenging to establish reasonable quantitative benchmarks for this task. So instead of conducting experiment on the existing benchmark, we have designed 20 sketches of complex scene composition, each with more than four sub-prompts, encompassing various terrains (plains, mountains, deserts, glaciers, cities) and diverse instances (rivers, bridges, stones, castles). We have utilized CLIP-Score (Hessel et al., 2021) for the global prompt and image, and evaluated the CLIP-Score for each background prompt and instance prompts by cropping the corresponding regions. Additionally, we have conducted a user study to assess different variants of our approach, using a 1-5 rating scale to evaluate image quality, placement, and prompt-image consistency. Details can be found in "Metrics" of Section 5.1, "Quantitative Evaluation" of Section 5.2, and Appendix E. The results of quantitative experiments and user studies are shown below, both of which demonstrate good improvements of our T$^3$-S2S approach over the ControlNet model and variants:

Best regards,

Authors

Dear Reviewer,

I hope this message finds you well.

First of all, we sincerely thank you for your detailed and insightful comments during the review process. Your feedback has been incredibly valuable in guiding us to strengthen our work.

In particular, your observation about the challenge of establishing reasonable quantitative benchmarks for this task was especially insightful. To address this, we designed quantitative experiments based on 20 complex cases and conducted a user study. These additions have enhanced our paper and allowed us to better evaluate the proposed method.

We submitted our rebuttal and the revised manuscript a few days ago and are eager to hear your thoughts. Your feedback is essential to ensure that all concerns have been fully addressed. While we understand your time is limited, we would greatly appreciate it if you could review our responses and let us know if further clarifications are needed.

Thank you again for your time and effort throughout this process. Your positive evaluation would mean a great deal to us, and we are committed to continuously improving our work based on your valuable guidance.

Best regards,

Authors.

Dear Reviewer 7JUo,

Thank you for your response. Below, we will address the two remaining points that you raised. While previous studies have primarily focused on the influence of attention maps on generation, our work aims to explore the roles of text embeddings and the Value matrix within the attention mechanism. Through a thorough analysis and tests, we developed Training-free Triplet Tuning, a method that addresses the entire cross-attention process from a holistic perspective. This effectively tackles the challenges associated with generation tasks involving multiple subjects, as you also echoed in your assessment of our work’s strengths.

CLIPScore is a widely accepted metric for assessing text-image consistency. In addition to that, we also conducted user studies to give subjective assessments, including aesthetic quality and text-image consistency as you suggested previously. These two aspects assess both the visual appeal and the alignment with user expectations, offering a complementary perspective to the quantitative results. Additionally, numerous visualization cases in Appendix show high-quality image generation across diverse scenes, including black-and-white sketches, game-style scenes, and realistic scenes, demonstrating the stability of our method.

We hope this clarification better addresses your concerns regarding the novelty and evaluation of our method. We sincerely appreciate your constructive feedback and your role in helping us improve the presentation of this paper.

Best regards,

Authors

Dear Reviewer JFtx,

Thank you for your constructive and insightful comments for helping us improve our paper. Please see below our response to your specific questions.

Q1: Explanation for game scene phrases and colored sketches.

A1: Thank you for the suggestion. To ensure consistency, we have started each prompt with “Isometric view of a game scene” when generating controlled synthetic images for game settings. This helps maintain a consistent angle and style, ignoring any incoherent instance sketches that might appear in real-world scenes, thereby focusing on object placement and verifying text-image consistency. The colored sketches are used solely to distinguish between different instances, and the colors used are arbitrary without class or semantic information. To validate this, we have also used grayscale sketches as input, and the resulting images are nearly identical under the same random seed (two columns pointed by the red arrows in Figures 12, 13 and 14).

Q2: Diversity of generated scenes, including removing game scene phrases, real-world scenarios, and black-and-white sketches.

A2: To verify that our approach is not limited to game scenes, (1) We have included more results on sketches of complex composition in Figures 12 and 13 of Appendix C, each with more than four sub-prompts, encompassing various terrains (plains, mountains, deserts, tundra, cities) and diverse instances (rivers, bridges, stones, castles); (2) We have added results of diverse scenes in Figure 14 of Appendix D to verify multi-scenario applicability, including a) four common simple scenes; b) two indoor scenes; and c) three scenes featuring instances of the same type but with different color attributes. Among these diverse scenes, our approach demonstrates superior performance with improved fidelity and precision in aligning with text prompts and sketch layouts for multi-instance synthesis.

Q3: Universally applicable to various controllable generation models, such as the T2I-Adapter.

A3: Thank you for the suggestion. To validate the general applicability of our approach beyond the ControlNet model, we have applied T$^3$-S2S to the T2I-Adapter model. We have used grayscale sketches for quick validation, which contain less semantic information. The PB and CP modules remain unchanged, while the DT module is integrated into the SDXL main channel, similar to CP, as it could not be placed in a separate branch like in ControlNet. We have used the same prompts and sketches from the main results (Figure 6) and Appendix C, with all other hyperparameters unchanged. The results are presented in Figure 11 of Appendix G. T$^3$-S2S effectively improves T2I-Adapter's alignment with prompts and layouts in complex scenes, demonstrating promising generalizability and effective control across both ControlNet and T2I-Adapter models.

Best regards,

Authors

Dear Reviewer,

I hope this message finds you well.

First of all, we sincerely thank you for your detailed and insightful comments during the review process. Your valuable suggestions have greatly guided us, and we have worked diligently to address all concerns.

In particular, your suggestions regarding validating the T2I adapter and exploring diversity in black-and-white sketch scenarios have been instrumental in demonstrating the generalizability of our proposed method. We conducted several additional experiments as requested, which we believe have significantly strengthened our manuscript.

We submitted our rebuttal and the revised manuscript a few days ago and are eager to hear your thoughts. Your feedback is crucial to us, and we want to ensure that all concerns are fully addressed. We understand your time is limited, but we would deeply appreciate it if you could review our responses and let us know if further clarifications are needed.

Thank you again for your time and effort throughout this process. Your positive evaluation would mean a great deal to us, and we are committed to further improving our work based on your guidance.

Best regards,

Authors.

Dear Reviewer s5sU,

Thank you for your constructive and insightful comments for helping us improve our paper. Please see below our response to your specific questions.

Q1: Correlation between PB and CP with the TOP $K$ method, and Explanation for Dense Tuning.

A1: TOP-K is a versatile concept used across domains to select the most relevant elements, enhancing efficiency and focus. In transformer architectures [1-4], it improves efficiency by selecting the top K attention scores, creating sparse attention, and ignoring the weak maps. In our paper, our method uses the concept of top-k not just for selection but to balance and highlight each instance and its attributes in the feature map. Prompt Balance adjusts the energy distribution of instance-related embeddings in the global prompt, while Characteristics Prominence uses Top-K selection to emphasize instance-specific attributes, enhancing the relevant feature map channels (Lines 76-89). We apply dense tuning (Kim et al., 2023) in the ControlNet branch to refine attention map contours, strengthen instance regions, and enhance the modulation of the entire cross-attention process. Experimental evaluations demonstrate that our T3-S2S approach improves text-to-image models, consistently producing detailed, multi-instance scenes aligned with input sketches and prompts.

[1] Memory-efficient Transformers via Top-k Attention.

[2] Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts.

[3] Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers.

[4] Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection.

Q2: Fair comparison for Dense Diffusion.

A2: Thank you for the suggestion. We have applied the Dense Diffusion to SDXL model for a fair comparison in the "Baselines" of Section 5.1 and Figure 6, which have better generation quality than the base on the SD V1.5.

Q3: Diversity of generated scenes, presenting results on different scene categories.

A3: To verify that our approach can generate diverse scenes, (1) We have included more results on sketches of complex scene composition in Figures 12 and 13 of Appendix C, each with more than four sub-prompts, encompassing various terrains (plains, mountains, deserts, tundra, cities) and diverse instances (rivers, bridges, stones, castles); (2) We have added results of diverse scenes in Figure 14 of Appendix D to verify multi-scenario applicability, including a) four common simple scenes; b) two indoor scenes; and c) three scenes featuring instances of the same type but with different color attributes. Among these diverse scenes, our approach demonstrates superior performance with improved fidelity and precision in aligning with text prompts and sketch layouts for multi-instance synthesis.

Q4: Quantitative experiments and user study.

A4: Thank you for the suggestion. We have designed 20 sketches of complex scene composition, each with more than four sub-prompts, encompassing various terrains (plains, mountains, deserts, glaciers, cities) and diverse instances (rivers, bridges, stones, castles). We have utilized CLIP-Score (Hessel et al., 2021) for the global prompt and image, and evaluated the CLIP-Score for each background prompt and instance prompts by cropping the corresponding regions. Additionally, we have conducted a user study to assess different variants of our approach, using a 1-5 rating scale to evaluate image quality, placement, and prompt-image consistency. Details can be found in "Metrics" of Section 5.1, "Quantitative Evaluation" of Section 5.2, and Appendix E. The results of quantitative experiments and user studies are shown below, both of which demonstrate good improvements of our T$^3$-S2S approach over the ControlNet model and variants:

Q5: Some typos, such as citations.

A5: Thank you for your careful reading. We have corrected them in Lines 358 and 359.

Q6: The position of the appendix.

A6: Thank you for the suggestion. As ICLR 2025 guidelines and FAQ state, 'We encourage authors to submit a single file (paper + supplementary text)' and 'you can include the appendices at the end of the main pdf after the references'. To address potential issues with the overly large PDF, we have compressed images without compromising clarity for smoother loading. If you feel that it is better to have a separate file for the appendix, we will split it into standalone supplementary material later.

Best regards,

Authors

Dear Reviewer,

I hope this message finds you well.

First of all, we sincerely thank you for your detailed and insightful comments during the review process. Your comprehensive feedback has been incredibly valuable for improving our paper. In particular, your suggestions regarding the quantitative experiments and Dense fair comparison have significantly helped us refine multiple aspects of our work.

We have worked diligently to address all your comments and conducted additional experiments as requested, which we believe have further strengthened our manuscript. We submitted our rebuttal and the revised version a few days ago and are eager to hear your thoughts.

Your feedback is essential to us, and we want to ensure that all concerns are fully addressed. While we understand your time is limited, we would greatly appreciate it if you could review our responses and let us know if further clarifications are needed.

Thank you once again for your time and effort throughout this process. Your positive evaluation would mean a great deal to us, and we remain committed to continuously improving our work based on your guidance.

Best regards,

Authors.

Dear Reviewer s5sU,

Thank you very much for recognizing and acknowledging our work.

We truly appreciate your positive feedback and for raising the score to 6—it means a lot to us.

Best regards,

Authors.

Dear Reviewers, SAC and AC,

We have carefully responded to reviewers' every major comment, a few days ago, and uploaded a revised version following reviewers' suggestions. Please take a look at our detailed responses (both general, and individualized) and the revision, to see if your questions have been addressed, or ask further clarifying questions where needed before the due date, so that we can provide follow-up responses in a timely fashion (considering the time-lag due to time-zone difference). Thank you !!

Best regards, Authors.

Dear Area Chair zcs2,

With the deadline approaching, thank you very much for your efforts in facilitating the discussion and guiding the review process.

We truly appreciate your support.

Best regards,

Authors