Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioning

Hi, Reviewer 4VWJ. We sincerely appreciate your careful review and so many professional and constructive comments, especially providing many useful and insightful references. We value each of them and provide responses below:

W1: Powerful LLMs as encoders

Thank you for this interesting question. We reference these three works, with particular focus on the Complex Human Instruction from [2] to enhance image-text alignment. We conduct experiments using Qwen2-VL-1.5B and Gemma2 as text encoders. (50K dataset, 5 epochs & 20 epochs, base model: DiT-ST Medium)

As shown in the table, replacing our original text encoder with LLMs for 5 epochs training yields slight improvements in both CLIPScore and FID. Considering the short training duration, the model performance may be unstable and cannot provide strong evidence. Therefore, we further tested the model performance with 20 epochs of training. We found that as the scale increases, the effects become more obvious, indicating that using LLMs as text encoders can further improve generation quality. This is a very interesting and worthwhile question for deeper investigation. Thank you for providing this valuable insight.

 

W2: Analysis of different injection processes

Thank you for your inspiration. Although we discussed the impact of different injection orders on performance in Table 8 in the appendix, we acknowledge that as you mentioned, object and relation jointly determine the layout of the generated image. Therefore, we supplement the following experiments based on Table 8: We replace the object injection at t=0 with 0.5object+0.5relation, 0.8object+0.2relation, and 0.2object+0.8relation respectively, to explore the joint effect of object and relation at the beginning of denoising.

Surprisingly, the experimental results are counterintuitive. The effect of jointly injecting object and relation at the beginning of denoising is not as good as injecting object alone. Due to time constraints, we have not yet found the clear reason for this phenomenon. We speculate that this should be closely related to the semantic entanglement we discussed in the introduction, and ensuring independent modeling of objects at the beginning is crucial. Furthermore, for the 1000-step denoising process of SD, injecting object at t=0 and relation at t=50 can both be viewed as early-stage injection. For our O-R-A injection order, it can both ensure that object and relation jointly determine the layout of the generated image in the early stage, while also taking into account the decoupled modeling of object concepts, thus achieving better results. This is a very interesting topic, and we will continue to explore it in the future to verify the above speculation. Thank you again for your inspiration.

 

W3: Injection timestep selection and overhead analysis

+injection timestep selection

To validate the generality of our cross-attention and SNR-guided injection strategy, we conduct experiments on two additional architectures: Flux .1 Dev and HunyuanDiT-v1.2. By analyzing cross-attention convergence and SNR curves, we determine the O-R-A injection steps for each model and compare them with alternative steps. As shown below, injecting at the identified steps consistently yields the best performance, while other steps lead to degradation, confirming the method’s robustness.

Although the specific injection positions vary with model architectures—e.g., Flux (0, 40, 300) shifts earlier due to faster convergence, while Hunyuan (0, 100, 500) shifts later—our strategy remains effective across all tested models.

+overhead analysis

Before training, we determine Attribute and Relation injection steps via a one-time offline analysis, enabling phase-wise semantic conditioning without extra supervision or dynamic control.

Specifically, for the Attribute injection timestep $s_{attr}$, we detect the earliest point where cross-attention stabilizes. For each sample and every layer $m=1,\dots,M$ and head $h=1,\dots,H$, we compute the normalized difference of the cross-attention matrix $Attn_t^{(m,h)} \in R^{Q \times K}$ at step $t$ (see Eq. 5), average across heads and layers, apply a moving average over the batch, and use thresholding to determine $s_{attr}$. The computational complexity is $T_{s_{\mathrm{attr}}} = O(N \times S \times M \times H \times QK)$, where $N$ is the sample size, $S$ the diffusion steps, $M$ and $H$ the numbers of layers and heads, and $Q$, $K$ the attention dimensions. As this one-time offline process uses few samples (e.g., $N=100$), its cost is negligible relative to a training epoch.

For Relation, we leverage the static SNR (Signal-to-Noise Ratio) curve provided by the diffusion scheduler, defined as $\mathrm{SNR}(t) = \alpha_t^2 / \sigma_t^2$, and compute its discrete curvature over the interval $[s_{\mathrm{attr}}, S]$ to locate the turning point corresponding to the optimal Relation injection timestep $s_{\mathrm{rel}}$ (see Eqs. 8 and 9). This method requires no model inference or input data, and only involves simple discrete operations over a one-dimensional sequence of length $S' = S - s_{\mathrm{attr}}$. Its computational complexity is: $T_{s_{\mathrm{rel}}} = O(S - s_{\mathrm{attr}})$. Since it only performs basic operations over a short vector of scalar values, the runtime is significantly lower than that of the matrix-based analysis used for $s_{\mathrm{attr}}$, and can be considered negligible.

 

W4: More comprehensive evaluation

We fully agree with you. CLIPScore and FID have inherent limitations, as they cannot adequately evaluate image naturalness, fidelity, and rationality. Therefore, they cannot reflect the model's understanding of complex captions. We additionally adopted GenEval for more comprehensive evaluation, such as semantic consistency, fidelity, clarity, diversity, and plausibility. While many other metrics exist, these three are the most widely adopted in the field. Unfortunately, when seeking to conduct broad comparative analysis, the number of applicable universal metrics becomes limited. Inspired by Reviewer 6Z8X, VQAScore can serve as a general benchmark similar to GenEval. The following presents the evaluation of our method and other baselines on VQAScore.

* denotes the simplified VQAScore used in [r1]. The other VQAScore is standard. We utilize the semantic primitives from the split-text to construct our standard VQA questions: "Is [OBJECT] visible in the image?" "Does the [OBJECT] have [ATTR]?" "Does the [OBJECT] appear [REL] the [OBJECT]?"

We can see that in the simplified VQA, the gap between models is small. However, in the standard VQA, DiT-ST shows a clear lead, demonstrating better capability in handling complex semantics and captions. Additionally, we provide extensive visual showcases in Figures 7-11 in the appendix for your evaluation. Once again, thank you for your suggestion and inspiration!

[r1] Progressive Prompt Detailing for Improved Alignment in Text-to-Image Generative Models, arxiv 2025

 

W5: Output of semantically incomplete captions

For prompts that lack explicit objects, we view this as a data-related issue rather than a limitation of our method. Our model is designed to parse complex prompts into structured semantic primitives and generate faithful images when sufficient information is provided. When a prompt omits core content (e.g., just like“dreamlike atmosphere” or “golden hour”), our method naturally falls back to global style or scene-level cues without object-level construction. Importantly, this challenge is not unique to our approach—existing models targetting at solving complex prompts would also behave similarly under such under-specified prompts. Addressing such cases typically requires prompt enrichment or additional context, rather than changes to the underlying generation framework.

 

 

Once again, we sincerely thank you for dedicating your valuable time to review our submission and for providing such insightful and professional suggestions. We are fully committed to incorporating the analyses and experiments you recommended into the final version, as a way to respect your time and effort. To be frank, you are a pivotal reviewer whose feedback may play a decisive role in the outcome of this submission, and we genuinely hope to earn your encouragement and support. Due to rebuttal policy restrictions, we regret that we are currently unable to provide the relevant visualizations for your review. However, if you have any further questions or feel that any of our explanations are unclear, please don’t hesitate to let us know—we would be more than happy to provide additional detailed clarifications. Thank you once again!

 

Best wishes and regards,

All authors of Submission 510

We sincerely thank you for carefully reading our rebuttal and for your encouraging decision to adjust the rating.We are delighted that our additional analyses and clarifications have addressed your concerns.

Additionally, we are especially grateful for your insightful suggestion regarding leveraging LLMs to further enhance the capability of text encoders. This has provided us with significant inspiration, and we plan to continue exploring this direction in our future work.

Once again, we deeply appreciate the time and effort you have dedicated to reviewing our submission. Your professional feedback has been invaluable in helping us improve this work, and we genuinely hope that this submission will ultimately achieve a positive outcome which lives up to your effort and trust.

Best wishes and regards,

All authors of Submission 510

Hi, Reviewer kbep. We sincerely appreciate your careful review and so many constructive comments, especially for pointing out some limitations that we had not previously recognized. We value each of them and provide responses below:

W1: Transferability to other models

We appreciate you highlighting this issue. We recognize that this experiment is essential for validating the transferability and generality of our method. Therefore, we conduct a comparative study. Our method initially employs SD as the base model, now, we expand our evaluation to include other Diffusion Transformer models. We apply our method to Flux .1 Dev and HunyuanDiT to assess its effectiveness across other models.

Experiment setting: 50K dataset, 5 epoch, base model: SDv3 Medium. The experiment results demonstrate that applying our method to other models, namely Flux and HunyuanDiT, also yields good performance. This demonstrates the effective transferability of our method across same model architectures Given that these models share a common transformer-based architecture, they can similarly leverage cross-attention mechanisms for injection.

 

W2 & Q1: Different caption splitting strategies

Thank you for raising this question. First, our caption splitting strategy can be transferred to other models and achieves good performance (as jointly demonstrated by Table 5, your W1, your W3 & Q2 experiment results). Second, We conduct additional comparative experiments involving two alternative splitting strategies that were explored but ultimately not adopted during this research. One is simple sentence splitting, and the other is progressive semantic primitive extraction. Specifically, the simple sentence splitting employs LLM to split all complex sentences in the caption into simple sentences. The progressive semantic primitive extraction employs LLM to extract all objects, relations, and attributes, then reconstructs the original caption into a set of three progressive sentence collections following O∪O-R∪O-R-A. The comparative results against our final adopted strategy are as follows:

Experiment setting: 50K dataset, 5 epoch, base model: DiT-ST Medium. As demonstrated, our adopted splitting strategy with hierarchical input achieves the best performance. Regarding your concern in the weakness section about whether our proposed strategy is the best one. We firmly believe that more effective splitting strategies will be developed in the future. However, among all the strategies we have explored thus far, our splitting strategy delivers the best results. If you have better ideas, we would be delighted to engage in further discussion with you, and even potentially collaborate on future research in this area.

 

W3 & Q2: Results without model fine-tuning

This is an interesting question. Building upon Table 5, we supplemented our evaluation with experiments using models without any fine-tuning. Specifically, we compare: vanilla SDv3 Medium, SDv3 Medium fine-tuned on our 100K dataset with original captions for 20 epochs (corresponding to SDv3 Medium in Table 5), and DiT-ST Medium trained on our 100K dataset using our method (corresponding to DiT-ST Medium without primitive injection in Table 5). The comparative results for both original caption and split caption are presented below.

We found that compared to original captions, split captions can achieve better performance even in vanilla SD without any fine-tuning, but fine-tuned models provide greater performance gains.

 

Q3: Ablation study for Text Encoding Refinement

Thank you for pointing out this issue, and we sincerely apologize for the absence of this necessary ablation study. During our investigation, we discovered that when utilizing MM-DiT's encoder, the concatenation of two token sequences results in partial dimension capacity remaining underutilized. Therefore, we incorporated complete-text captions in this component, which is our proposed Text Encoding Refinement. Consequently, we conduct ablation experiment on Text Encoding Refinement with DiT-ST Medium. (50K dataset, 5 epochs, base model: DiT-ST Medium)

As shown in the table, without Text Encoding Refinement, the model performance has declines of 0.62% in CLIPScore and 0.54% in FID metrics. Although the improvement from adding complete-text captions is modest, this demonstrates that Text Encoding Refinement indeed contributes positively to model performance. We commit to incorporating this ablation study into the final version. Thank you once again for helping make our paper more complete and solid!

 

Q4: Effect on downstream tasks

Thank you for raising this question. We agree that assessing a method’s generality and applicability to downstream tasks is important. Our approach is well-suited for such tasks—including editing—due to its lightweight and modular design. Specifically, split-text conditioning does not alter the pretrained diffusion backbone, but instead redesigns the conditioning mechanism by injecting semantic components (object, attribute, relation) incrementally during denoising, rather than feeding the full prompt at the initial step. This progressive injection aligns better with the evolving nature of generation and ensures compatibility with downstream pipelines.

Taking image editing as an example, which often requires localized modifications (e.g., changing object color or inserting attributes), our method provides two main benefits. First, semantic disentanglement during training allows fine-tuning to target specific elements without affecting unrelated regions, enabling precise compositional edits. Second, staged injection introduces temporal disentanglement, permitting edits at the right step (e.g., modifying attributes without disturbing layout), thus improving controllability. As an architecture-agnostic, plug-and-play module, our approach naturally complements methods like LoRA or T2I-Adapter for editing tasks.

 

L1: Scaling law for dataset size

Thank you for pointing out this overlooked issue. We design two related experiments to investigate the scaling law regarding data scale. We use DiT-ST Medium as the base model and, using the same method employed to create our 100K dataset, generate additional 50K and 500K datasets. We trained the base model on datasets of 50K, 100K, and 500K scales for 5 epochs each, then conduct comparative analysis.

Both metrics show clear benefits from data scaling: CLIPScore increases and FID reduces, indicating gains in both semantic alignment and visual quality as training data grows.

By comparing the CLIPScore performance of models trained on different data scale across various caption lengths, we observe that larger training datasets lead to better overall performance. Moreover, models trained on larger datasets demonstrate particular improvements on longer and more complex captions.

To be honest, these two experiments are insufficient to fully demonstrate the model's scaling law with respect to data scale. However, due to resource and funding limitations, we are unable to create larger datasets for further exploration. Therefore, we believe our method preliminarily follows the scaling law regarding dataset. We will incorporate this limitation into the final version with more detailed analysis. Thank you for the inspiration!

 

L2: Limitation of universality

We appreciate this important reminder. Although our discussion in W1 demonstrates that our method can be transferred to Flux and HunyuanDiT, we honestly acknowledge that our method may not be applicable to all diffusion models—a common limitation shared by similar works. Therefore, we decide to add this limitation discussion to the limitation section in the final version to enhance the depth of our research and analysis. Thank you once again for your constructive and insightful feedback.

 

 

Once again, we sincerely thank you for dedicating your valuable time to review our submission and for providing so many constructive comments. We are committed to incorporating the analyses and experiments you recommended into the final version, as a way to truly honor your time and effort. To be honest, you are a pivotal reviewer whose feedback could be decisive for the outcome of this paper. We sincerely hope to earn your encouragement and support. Due to the character limit, we can only provide brief responses at this stage. We very much look forward to engaging in further discussion with you during the discussion period. If you have any additional questions or find any of our explanations unclear, please don't hesitate to let us know—we would be happy to provide further clarifications. Thank you once again!

 

Best wishes and regards,

All authors of Submission 510

Hi, Reviewer 1VDF. We sincerely appreciate you taking time for review and providing encouraging and constructive feedback! We value each of them and provide responses below:

W1: More detailed elaboration on Equation 10

We thank the reviewer for the question regarding the design of our attention loss. Our proposed loss term $L_{attn}$ is specifically designed to regulate cross-attention behavior in the multi-stage semantic injection process of diffusion models, ensuring stable training, semantic alignment, and independence between injection stages. The loss is formulated as:

$L_{attn} = \alpha \cdot L_{inject} + \beta \cdot L_{conv} + \eta \cdot L_{mutex}$

where $\alpha = 0.6, \ \beta = 0.25, \ \eta = 0.15$ , and a global scaling factor $\lambda = 0.25$ are used in practice.

The first term, $L_{inject} = E_{t,x_0} \left[ \sum_{i=1}^3 \delta_i(t) \, ( CrossAttn_i(c_{base}, c_{inject}^{(i)}) - c_{target}^{(i)} )^2 \right]$ , enforces the alignment of cross-attention outputs with the target semantics at each activated injection stage, thereby improving semantic precision.

The second term, $L_{conv} = E_t \Big[ \sum_{i=1}^3 \big( SNR_{attn}^{(i)}(t) - SNR_{target}^{(i)}(t) \big)^2 \Big]$, supervises the evolution of the signal-to-noise ratio (SNR) of attention across timesteps to ensure timely convergence, avoiding premature or delayed semantic injection that could otherwise lead to generation bias.

The third term, $L_{mutex} = E_t \Big[ \sum_{i \neq j} \delta_i(t) \cdot \delta_j(t) \cdot \text{Overlap}(\text{AttnMap}_i, \text{AttnMap}_j) \Big]$ , prevents spatial overlap between attention maps of different stages, thereby reducing semantic interference and preserving the independence and interpretability of each injection phase.

Together, these components form a principled objective that not only stabilizes training but also enhances fine-grained semantic alignment, which is critical for the effectiveness of our split-text conditioning strategy in complex prompt scenarios.

 

W2: Visualization of attention at various injection timesteps

First, we sincerely apologize that due to NeurIPS 2025 policy restrictions, we are prohibited from using or updating any links in the rebuttal, so we cannot show you visualized images. We appreciate you pointing out this issue, which is very necessary. A comprehensive analysis needs to include failure cases.

To determine the attribute injection timestep $T_{\mathrm{inject}}$, we analyze cross-attention maps during denoising. For each candidate $T_{\mathrm{inject}}$, we inspect both the map at this timestep and the following 25 noise level $(T_{\mathrm{inject}}, T_{\mathrm{inject}}+25)$ to verify convergence. The cross-attention map is defined as:

$A_t = \frac{1}{M \cdot H} \sum_{m=1}^M \sum_{h=1}^H \mathrm{Softmax}\left(\frac{Q_t^{(m,h)} K_t^{(m,h)T}}{\sqrt{d_h}}\right)$,

where we further compute entropy $H(A_t)$ for dispersion and contrast ratio $C(A_t) = \frac{\max(A_t)}{\mathrm{mean}(A_t)}$ for focus.

At $T_{\mathrm{inject}}$, entropy drops and contrast rises sharply, indicating a shift from diffuse global attention to localized attribute focus. Over the next 25 noise levels, these metrics stabilize, confirming effective attribute-level conditioning. Furthermore, we compare attention maps at three candidate ranges: 350–375, 400–425, and 450–475.

In the first range (350–375), the model remains in a global layout construction phase, with attention maps characterized by broad, low-intensity coverage lacking attribute-specific focus. In contrast, in the last range (450–475), the model has already transitioned into an overly localized state, where attention maps exhibit prematurely concentrated patterns with reduced semantic plasticity, indicating that attribute injection would come too late. Only the 400–425 range displays a clear transition from global to localized attribute-focused attention, showing sharp, high-contrast regions precisely aligned with attribute semantics. Based on this observation, we select 400 as the attribute injection step in the main text.

These results validate our injection-step selection and provide interpretable evidence of stage-wise semantic alignment.

We sincerely thanks for your valuable suggestions and agree on the importance of this analysis. We apologize once again for not being able to include direct visualizations in the rebuttal due to NeurIPS 2025 policy restrictions. We commit to providing comprehensive visualizations and detailed analyses in the camera-ready version to address this point thoroughly.

 

 

Once again, we sincerely thank you for dedicating your valuable time to review our submission and for providing such encouraging feedback. We are committed to incorporating the analyses and experiments you recommended into the final version, as a way to truly honor your time and effort. Although the rebuttal policy limits our ability to provide relevant visualizations that could further enhance the completeness and interpretability of our work, we genuinely hope to receive your continued support in the subsequent stages, which will be crucial to the success of this submission. We look forward to further discussions with you during the discussion period. Thank you again and we eagerly await your response.

 

Best wishes and regards,

All authors of Submission 510

Hi, Reviewer 1VDF!

We would like to express our heartfelt thanks for you taking your valuable time to review and providing such inspiring feedback. Your suggestions regarding formula clarity and convergence visualization are indeed very helpful for improving the quality of our submission. Following your suggestions, we have made careful revisions in response. We are unable to provide figures for your reference due to NeurIPS policy, and we have chosen to describe our approach details in text. We sincerely hope our revisions meet your expectations. We greatly appreciate your positive affirmation and respectfully hope to gain your continued support! If there are any questions, please don't hesitate to let us know - we would be very happy to respond!

Thank you once again for your time, dedication and support!

 

Best wishes and regards,

All authors of Submission 510

Hi, Reviewer Dd4v. We sincerely appreciate you taking time for review and providing encouraging and constructive feedback! We value each of them and provide responses below:

Q1: Ablation study for Text Encoding Refinement

Thank you for pointing out this issue, and we sincerely apologize for the absence of this necessary ablation study. During our investigation, we discovered that when utilizing MM-DiT's encoder, the concatenation of two token sequences results in partial dimension capacity remaining underutilized. Therefore, we incorporated complete-text captions in this component, which is our proposed Text Encoding Refinement. Consequently, we conduct ablation experiment on Text Encoding Refinement with DiT-ST Medium. (50K dataset, 5 epochs, base model: DiT-ST Medium)

As shown in the table, without Text Encoding Refinement, the model performance has declines of 0.62% in CLIPScore and 0.54% in FID metrics. Although the improvement from adding complete-text captions is modest, this demonstrates that Text Encoding Refinement indeed contributes positively to model performance. We commit to incorporating this ablation study into the final version. Thank you once again for helping make our paper more complete and solid!

 

Q2: Analysis of memory usage and computational cost

First, we analyze the additional memory overhead introduced by Split-Text Conditioning by listing the formula. The additional memory cost mainly arises from text embedding caching and cross-attention operations.

We employ text encoders such as CLIP-L $[B,77,768]$, CLIP-G $[B,77,1280]$ and T5-XXL $[B,256,4096]$ with $B$ for batchsize, $[77,256]$ for the number of token respectively and $[768, 1280,4096]$ for embedding dimension. Given a full-text length $L$, object, relation, and attribute segments $L_O, L_R, L_A$, embedding dimension $d$ , batch size $B$, and fp16 precision, the memory required for a single text embedding is $M_{\text{text, single}} \approx B \cdot L \cdot d \cdot \mathrm{sizeof(fp)}$ . After O-R-A segmentation, we cache three segmented embeddings and the full-text embedding, yielding $M_{\text{text, ORA}} \approx 4 \cdot B \cdot L \cdot d \cdot \mathrm{sizeof(fp)}$ . Since $L_O + L_R + L_A \approx L$ , the main increase comes from storing multiple embeddings rather than longer sequences.

For cross-attention, which dominates the additional memory overhead, let the number of image tokens be $N = (imgsize // 2)²=512 \times 512$, the number of attention heads $H=28$, and the per-head dimension $d_h=64$, giving a total hidden dimension $d=H \times d_h=1792$ . The key-value (KV) cache memory for a single cross-attention layer is $M_{\text{KV}} \approx B \cdot H \cdot N \cdot L_{\text{avg}} \cdot d_h \cdot \mathrm{sizeof(fp)}$ . Since our method injects three cross-attention stages during denoising, the total KV cache memory is $M_{\text{KV, total}} \approx 3 \cdot B \cdot H \cdot N \cdot L_{\text{avg}} \cdot d_h \cdot \mathrm{sizeof(fp)}$ , while the attention logits require an additional $M_{\text{logit}} \approx 3 \cdot B \cdot H \cdot N \cdot L_{\text{avg}} \cdot \mathrm{sizeof(fp)}$ . Therefore, the overall memory cost is $M_{\text{total}} \approx 4 \cdot B \cdot L \cdot d + 3 \cdot B \cdot H \cdot N \cdot L_{\text{avg}} \cdot d_h + 3 \cdot B \cdot H \cdot N \cdot L_{\text{avg}}$ , where the first term corresponds to text embedding caching, the second to KV cache, and the third to attention logits, with cross-attention being the dominant factor while convergence analysis is negligible.

In terms of computational cost, evaluated on an NVIDIA H200 GPU, the overhead of Split-Text Conditioning is modest and results are as follow:

At 512×512, replacing the original caption with hierarchical input increases GFLO Ps by 17.4% but has negligible impact on throughput (0.2262 → 0.2248 img/s), as text processing is a minor component of the overall cost. Adding injection further raises GFLOPs by 20.2% and slightly reduces throughput (0.2248 → 0.2161 img/s) due to the additional cross-attention computations.

At 1024×1024, where image generation dominates the cost, hierarchical input adds only 1.4% GFLOPs with no throughput change, while injection incurs an extra 3% GFLOPs with a small throughput drop (0.0571 → 0.0558 img/s).

Overall, text-related overhead becomes negligible at high resolution, and injection introduces modest, manageable additional cost.

 

Q3: Discussion of challenges and benefits during training

Thank you for your question. Our DiT-ST model already incorporates staged semantic injection during training, and this design has provided clear empirical benefits. First, training-time injection explicitly guides the model to learn the temporal scheduling of object, relation, and attribute conditioning, leading to more robust semantic disentanglement compared to inference-only strategies. Second, it improves generalization and controllability: by integrating injection into the optimization process, the model not only performs well within the training distribution but also exhibits stronger interpretability and stability for long prompts and complex scenes. Third, training-time injection stabilizes the behavior of cross-attention modules, reducing the need for additional heuristic scheduling at inference time.

However, we also note that this approach introduces additional costs, including higher memory consumption and longer training time due to multi-stage injection and the associated cross-attention parameters and KV caches. Moreover, the scheduling hyperparameters for injection require careful tuning during training.

For comparison, ControlNet, a well-known approach that also integrates conditioning during training, demonstrates similar benefits: it enables the model to directly learn the correspondence between control signals (e.g., edges, depth, pose) and image generation, yielding strong controllability at inference. Yet, ControlNet also faces limitations, such as increased parameter count from additional conditional branches and the need for specialized annotated data, which reduces its flexibility.

In contrast, SCoPE performs interpolation only at inference time, which avoids the training overhead but fails to expose the model to semantic injection during optimization. As a result, SCoPE does not fully resolve semantic entanglement and lacks explicit modeling of different semantic types (objects, relations, and attributes). This inference-only design can lead to less stable performance, especially for long prompts or compositionally complex scenes.

In summary, incorporating injection during training—as we do in DiT-ST—offers clear advantages for semantic disentanglement, interpretability, and controllability, distinguishing it from inference-only approaches such as SCoPE.

 

 

Once again, we sincerely thank you for dedicating your valuable time to review our submission and providing such positive feedback. We are committed to incorporating the above analyses and experiments you recommended into the final version, as a way to truly honor your time and effort. Your encouragement is an important reason why we are willing to continue participating in the rebuttal process. We eagerly look forward to your continued support, which will play a crucial role in the success of this submission. Therefore, we very much anticipate further discussion with you during the discussion period. If you have any other questions, please kindly let us know, and we will be happy to response them enthusiastically. Thank you!

 

Best wishes and regards,

All authors of Submission 510

Hi, Reviewer Dd4v!

We would like to sincerely thank you for taking the time to review our paper as an emergency reviewer and for giving us such generous encouragement, which is extremely precious to us. At the same time, your constructive comments have also given us much valuable inspiration, and we have responded to each one carefully in our rebuttal, hoping that you will be satisfied. Your encouragement has always been a constant source of motivation for us to improve the quality of our research, and we deeply value your continued support. If there is anything that needs additional explanation or supplementation, we would be more than happy to do so!

Thank you once again for your time, dedication and support!

 

Best wishes and regards,

All authors of Submission 510

Hi, Reviewer 6Z8X. We sincerely appreciate your careful review and so many constructive comments. We value each of them and provide responses below:

W1: Difference between our work and SCoPE

Thank you for the vital reference! We were already aware of this work and cited it in Line 104. We fully agree with your suggestion and summarize detailed differences in method and experiment:

+Method

① Method process.

Both DiT-ST (Ours) and SCoPE aim to mitigate semantic overload through progressive injection, thereby enhancing the model's understanding of captions and improving generation quality, However, the two methods differ fundamentally in design.

SCoPE employs LLM to segment captions into multiple sub-captions progressing from coarse to fine-grained. The timestep for progressive injection is determined by calculating similarity among sub-captions and applying proportional similarity ratios. While SCoPE achieves caption segmentation through this training-free strategy, it cannot distinguish semantic primitives (object, relation, attribute), thereby remaining vulnerable to semantic competition and concept coupling problems outlined in our introduction.

We employ LLM to split captions into three semantic primitives—objects, relations, and attributes—constructing independent encoding streams. By utilizing SNR curvature and cross-attention convergence, we can easily identify optimal injection timesteps, enabling the model to first establish object concepts before incrementally incorporating relation and attribute. Our method effectively mitigates semantic competition and significantly enhances both generation quality and semantic consistency.

Therefore, SCoPE tends to focus on coarse-grained input control during the inference stage, whereas ours represents a fine-grained structural optimization method targeting internal model architecture and modeling processes. DiT-ST demonstrates superior interpretability and controllability.

② Model structure.

SCoPE, as a method based on the U-Net architecture, has a multi-scale convolutional design that is restricted by a local receptive field, which makes it difficult to adequately capture long-distance dependencies and has limited scalability. SCoPE tends to be a heuristic inference enhancement rather than a scalable training framework.

Ours is based on the Diffusion Transformer, which is naturally adapted to larger scales and more complex scenarios. Global self-attention mechanism has capabilities for long-distance semantic modeling and complex scenario understanding, and makes our method preliminarily follows the scaling law.

+Experiment

① Evaluation metrics.

SCoPE uses CLIPScore and VQAScore.

We use FID, CLIPScore, GenEval and VQAScore (Inspired by the reference you provided).

② Base models.

SCoPE uses U-Net-based diffusions: SD 1–4, SD 2.1, and SDXL.

We use Transformer-based diffusions: SDv3 Medium and SDv3.5 Large. Thus, our method preliminarily follows the scaling law (Table7, Reviewer kbep L1 and Reviewer kbep W3 & Q2).

③ Experiment evaluation.

SCoPE only evaluates VQAScore and CLIPScore for proposed methods and base models, without more main results or any ablation studies.

Our main experiments evaluate FID, CLIPScore, GenEval, and VQAScore (added in W2 based on your suggestion), along with model performance under different caption lengths and model sizes. We include not only base models (SD) but also Transformer-based diffusions such as PixArt-α, DALL-E 3, Flux.1 Dev, Dream Engine. In addition, extensive ablation studies further validate our design's effectiveness.

④ Method performance.

SCoPE improves CLIPScore by up to 1.3% over the base model (SDv2.1).

Ours improve CLIPScore by up to 2.78% over the base model (SDv3 Medium).

As two methods aren’t directly comparable, we use CLIPScore gain as a rough reference. Notably: 1.SCoPE is training-free, which is its advantage. 2.SCoPE adopts prompt enhancement, it’s unclear how much of the gain comes from their design instead of the added information. 3.SCoPE generates eight candidate outputs per prompt and selects the best-aligned one, while we use a single random one.

---

In addition, our models and datasets are now publicly available. Compared to 4-page reference, our paper offers greater detail, completeness and more visualizations. Generally, SCoPE is still an interesting work. We will incorporate above analysis into the final version to strengthen our study. Sincerely thank you for your insightful input!

 

W2: More comprehensive evaluation

We fully agree with you. CLIPScore and FID have inherent limitations, as they cannot adequately evaluate image naturalness, fidelity, and rationality. Therefore, we additionally adopted GenEval for more comprehensive evaluation, such as semantic consistency, fidelity, clarity, diversity, and plausibility. While many other metrics exist, these three are the most widely adopted in the field. Unfortunately, when seeking to conduct broad comparative analysis, the number of applicable universal metrics becomes limited. Fortunately, VQAScore from the reference you provided is also a general benchmark similar to GenEval. In order to reserve enough space for responding to your other valuable feedback, we kindly direct you to Reviewer 4VWJ W4 for our VQAScore evaluation. Thank you for the inspiration!

As for the user study, we acknowledge this limitation in the appendix (lines 969–973). As it's a common limitation in all similar works [9][10][22][26][39][45] (same reference in our submission) [r1][r2][r3][r4], we regret we couldn't conduct one. As an alternative, we offer extensive visualizations and available models to facilitate your subjective perception and evaluation. We apologize once again for our inability and sincerely appreciate your understanding.

[r1] Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models, CVPR 2025

[r2] Language-Guided Image Tokenization for Generation, CVPR 2025

[r3] SANA: Efficient High-Resolution Text-to-Image Synthesis with Linear Diffusion Transformers, ICLR 2025

[r4] SinSR: Diffusion-Based Image Super-Resolution in a Single Step, CVPR 2024

 

W3: LLM-related experiments

Thank you for pointing out our overlooked experiments. We add 2 experiments:

① Different LLMs. Besides initial Qwen-plus, we also use GPT-3.5-turbo and Gemini-2.0-flash and evaluate them on COCO-5K. (50K dataset, 5 epochs, base model: SDv3 Medium)

We observe that using any LLM improves performance, with Qwen performing best. This demonstrates the effectiveness and generalization of LLMs in our method.

② LLM finetuning: Prompting is viewed as LLM finetuning. Besides our prompt, we also compare a simple prompt and w/o prompt on COCO-5K. The simple prompt is: Extract all objects, relation between object and attributes from the description. (50K dataset, 5 epochs, base model: SDv3 Medium)

It shows that prompt-based LLM finetuning improves performance, with better prompts yielding better results. Without prompting, LLM tends to generate low-quality expansions [r5], hindering semantic understanding of captions.

[r5] Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models, NeurIPS 2024

 

W4: Failure case analysis

First, we sincerely apologize that due to NeurIPS 2025 policy restrictions, we are prohibited from using or updating any links in the rebuttal, so we cannot show you visualized images. We appreciate you pointing out this issue, which is very necessary. A comprehensive analysis needs to include failure cases.

+Prompt of High Complexity

The first case uses a prompt with complex syntax and layered semantics, describing a studio scene with ivy-covered windows, a weathered violin, wine glass, lace-covered table, and and a hidden cat watching dust. Both our model and SDv3 failed to fully capture some attributes (e.g., the violin’s aged texture), likely due to implicit and long-range phrasing. Still, our model better preserves spatial cues such as the cat’s occlusion and gaze, often missed by baselines. While imperfect, our method generally retains more semantic detail in such complex prompts.

+Prompt of Irrelevant Information

The second case shows failure due to missing visual grounding in the input. The prompt You may come for your appointment at 9 a.m. tomorrow. contains no visual object or scene. Although users may expect a hospital-related image, all models—including ours—produce irrelevant outputs. This reflects a limitation of T2I models when prompts lack explicit semantics.

We hope these cases help clarify the scope and limitations of our method under different prompt conditions.

 

W5: More detailed captions

Thanks for your suggestion! We promise to detail captions of tables and figures to better illustrate the content and the motivation.

 

 

Once again, we sincerely thank you for dedicating your valuable time to review our submission and offering so many insightful and constructive suggestions. We are committed to incorporating the above analyses and experiments you recommended into the final version, as a way to truly honor your time and effort. Honestly, you are the most influential reviewer for the outcome of this submission, and we eagerly look forward to engaging with you further during the discussion period and hope to gain your continued guidance and encouragement. If you have any other questions or feel that any of our explanations are unclear, please kindly let us know—we will be happy to provide further detailed clarifications. Thank you very much once again!

 

Best wishes and regards,

All authors of Submission 510

Hi, Reviewer 6Z8X!

We would like to express our heartfelt thanks for you taking the time to review our paper as an emergency reviewer and providing many constructive comments. We value each of your comments and have responded to them one by one in our rebuttal. With less than 3 days remaining until the discussion period ends, we sincerely hope our responses and additional experiments have addressed your concerns. Additionally, we would be deeply grateful for your favorable consideration. As the most important reviewer, any additional support from you could have a better decisive impact. It would not only improve our work, but also reflect the significance of your insightful suggestions. If there’s anything else we can clarify or supplement, please don’t hesitate to let us know, we're ready to respond immediately.

Thank you once again for your time, dedication and support!

 

Best wishes and regards,

All authors of Submission 510