TextCenGen: Attention-Guided Text-Centric Background Adaptation for Text-to-Image Generation

Thank you for your valuable review. We address each point below.

[W1]Task Importance Placement in Introduction

We'll add to the introduction: "As shown in Fig. 6, creating text-friendly images is essential for graphic design applications (validated by our 114-participant user study)."

[W2]Correlation Between Attention Maps and Target Objects

Correlation is an important assumaption for our method, which relies on attention manipulation. Strengthening the correlation between semantic content and attention maps is a critical research direction in this field [1,2]. Any training-free methods that improve this correlation could be readily integrated as upstream components to our approach. This represents a promising direction for future research that would complement rather than replace our contribution. Notably, Our results still demonstrate that attention manipulation remains an effective mechanism for our task even with current models with semantic loss (with clip score).

[W3]Alternative Approaches Comparison

To clarify differences between our task and existing approaches, we present the following comparison:

Unlike layout-to-image tasks requiring training and intensive annotation, our method only needs space region specification, which is crucial for dynamic applications like mobile wallpapers (Figure 1) or e-commerce posters.

Layout-based methods primarily focus on object arrangement rather than creating text-friendly backgrounds. Their objective is fundamentally different:

• Layout-grounded T2I: Positions objects according to a layout
• Our approach: Creates backgrounds that harmonize with planned text regions

Direct TV restriction in target regions would yield effects similar to attention reduction, which we have already validated in our ablation studies. As shown in Table 1, simply applying spatial constraints without force-directed guidance (w/o FDG) leads to higher CLIPS Loss (2.2 vs 0.32), indicating reduced semantic fidelity to the original prompt. This demonstrates that naive approaches like direct TV restriction fail to balance text-friendliness with instruction following.

Our force-directed approach enables precise object placement control while maintaining semantic coherence, as evidenced by lower CLIPS Loss and superior text-compatibility metrics.

Both alternatives would ultimately need similar attention control mechanisms to achieve our goal of text-friendly background generation that respects user-specified regions.

[W4]Testing with More Advanced Models

We have deployed our method on SD1.5, SD2.0, and SDXL to demonstrate the broad applicability of our method across different model architectures. Our approach can be adapted to MMDiT-based models like FLUX through methods similar to those demonstrated in recent work [4] similar to this code. The key difference in MMDiT models is the attention structure, which can be described as:

MMDiT Attention =  | Text-Text (TT) | Text-Image (TI) |
                   | Image-Text (IT) | Image-Image (II)| ,

In UNet models, we modify cross-attention (equivalent to TI block). For MMDiT models, one possible solution is to apply our force-directed approach to the Image-Text (IT) block (such as transpose the IT block), and maintain consistency in the Text-Image (TI) block.

This adaptation is technically straightforward based on existing implementations like Rare-to-Frequent. While implementation details differ, the fundamental principle of manipulating attention to create text-friendly backgrounds remains valid across architectures.

We appreciate your thoughtful suggestions and will address them in our revised manuscript.

[1] Yang et al. "Dynamic prompt learning: Addressing cross-attention leakage for text-based image editing," Neurips 2023

[2] Liu B, Wang C, Cao T, et al. "Towards understanding cross and self-attention in stable diffusion for text-guided image editing," CVPR 2024

[3] Zheng et al., "LayoutDiffusion: Controllable diffusion model for layout-to-image generation," CVPR 2023

[4] Park D, Kim S, Moon T, et al. "Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance," ICLR 2025 Spotlight

Thank you for your valuable review. We address your concerns with additional experiments and analysis below.

[Q1,Q3]Multi-Object and Complex Scene Handling

Our approach shows strong performance in complex scenes with multiple objects. We additionally evaluated on the Desigen dataset, which includes multiple text boxes representing a challenging multi-region scenario:

In multi-object cases, our method creates complementary forces guiding content away from specified regions. This approach extends to complex layouts as shown in our Desigen benchmark results:

In multi-object cases, our method creates complementary forces guiding content away from specified regions.

[W1,Q2] The reviewer may misunderstood our approach. Our warping force and other forces specifically address object size reduction and fragmentation caused by attention reduction, and this fragmentation equally occurs in both convex and non-convex shapes. We mentioned non-convex shapes in limitations only to consider more fine-grained region requirements from users, such as spiral-shaped regions. For non-convex shapes, our implementation approximates them using their convex hull. While a limitation, user studies show most text and icon placement scenarios involve convex regions, making this practical for common use cases.

[Q4] We're exploring image reward metrics for optimal parameters across different scene complexities, including MLLMs as judges to automatically adjust parameters.

[W3] Applicability to Other T2I Models

Our method is extensible to other text-to-image models. We have successfully applied it to multiple diffusion models (see line330 table 2 in the paper).

The implementation is straightforward. After defining the attention processor class, integration requires only a few lines:

for name in pipe.unet.attn_processors.keys():
    if "attn2" in name:
        attn_procs[name] = my_attn_processor(guidance_func, region, len(tokens), place_in_unet=name)
    else:
        attn_procs[name] = pipe.unet.attn_processors[name]
pipe.unet.set_attn_processor(attn_procs)

When combined with the finetuned diffuison model, we see improvements:

[Com 3] Computational Complexity and Runtime Analysis

We will add runtime analysis to the appendix. Our method maintains computational complexity comparable to standard diffusion models with minimal overhead from force computations. Our implementation is efficient, requiring less than 15GB VRAM (compatible with consumer GPUs like RTX 3090) and averaging approximately 30 seconds per image generation, as it requires only a single inference pass rather than multiple candidates.

[Com 1] Algorithm Pseudocode

We will add simplified pseudocode to method section in the revised paper to clarify the implementation process. Here is the algorithm that summarizes our approach:

Algorithm: Text-Friendly Background Adaptation
Input: Text prompt P, Target region R for text placement
Output: Text-friendly image I with clear space in region R

1. Initialize diffusion model with prompt P
2. For each timestep t:
   a. Extract cross-attention maps A_k^l for tokens k, layers l
   b. For each token k, layer l:
      i. Detect conflicts D(k, R, A_k^l)
      ii. For conflicting tokens:
         - Compute attention centroid
         - Calculate repulsive force F_rep and margin force F_m
         - Compute total displacement Δpos = F_rep + F_m
         - Apply warping through affine transformation
      iii. Apply constraint: A_k,new^l = A_k^l ⊙ (1-R)
   c. Apply modified maps for guidance
3. Return text-friendly image I

Application Scenarios

We have already supplemented our evaluation with the Desigen dataset(see table 2), which represents poster and advertisement design scenarios. Mobile interface design represents an interesting future direction that we plan to explore.

We appreciate your consideration and helpful suggestions.

Thank you for your valuable review.

Evaluation on Poster/Ads Datasets

We selected the Desigen dataset as a layout design dataset for evaluation. We successfully downloaded 53,577 usable images, with 52,806 used for training the Desigen version. The remaining 771 images from the validation set and their corresponding static text masks were used to construct our new Desigen benchmark.

Our core contribution is text-friendly background generation for arbitrary user-specified regions without training constraints. This approach extends beyond posters and ads to applications like mobile wallpapers where users can specify any position for text, rather than being limited by the distributions in datasets [1,2].

Comparison with Trained Versions of Baselines

Our method focuses on a training-free approach due to the diverse range of user prompts and space region positions. Training often compromises generalization across subtasks.

Table 1 shows Desigen-Trained + Attention Reduction improves over its training-free version on certain metrics. When combining our method with the trained Desigen model, we observe substantial improvements:

Table above shows Desigen-Trained improves over its training-free version.

When combining our method with the trained Desigen model, we observe substantial improvements:

Comparison with Retrieval-Based Methods

We evaluated against Text2Poster [3] across all datasets. Our approach differs from retrieval methods: we support arbitrary user regions and create novel content beyond image databases. Since Text2Poster doesn't support user-specified space regions, we tested on preset positions (top, bottom, left, right, center) and selected the best result based on VTCM for fair comparison.

Our related work section (line 083) explicitly mentions Text2Poster, which inspired us to consider background adaptation in the T2I era. We appreciate the reviewer highlighting this discussion direction, as it has provided us with valuable insights.

Our method outperforms retrieval approaches in most text-friendliness metrics while preserving prompt semantics. The emergence of autoregressive models like GPT-4o creates potential for combining Text2Poster's retrieval methodology with attention-guided RAG to achieve both text-friendliness and semantic fidelity.

Broader Impact

Our work extends beyond text placement to other UI elements including icons (Appendix Figure 1). While approaches like TextDiffuser-2 handle text with specific styles through dedicated training, our method achieves user instruction following and arbitrary space region support through attention control. This approach avoids reliance on "sign" or similar elements in prompts (line 088-089), opening a pathway to more flexible controllable generation that preserves both user intent and element compatibility.

We appreciate your reconsideration of our paper.

[1] Zheng G, Zhou X, Li X, et al. LayoutDiffusion: Controllable diffusion model for layout-to-image generation. CVPR 2023

[2] Zhou, Min, et al. "Composition-aware graphic layout GAN for visual-textual presentation designs." IJCAI, 2022

[3] Jin, C., et al. Text2Poster: Laying out stylized texts on retrieved images. ICASSP 2022 [PS: Pioneering work]