Improving Long-Text Alignment for Text-to-Image Diffusion Models

Thank you for your positive feedback. Your insightful questions have improved our work very much. Below we respond to the comments in weaknesses (W) and questions (Q).

W1: An important paper that is missed here is ELLA $Hu et al. 2024$ . ELLA might be a valid comparison, and its adapter would likely have been a better alternative.

Baseline: Thank you for your advice. We have included ELLA as a new baseline in the updated version. Our method outperforms ELLA on both our original evaluation metrics and the new DPG-bench and VQAScore. Considering our training time is only 1/7 that of ELLA, this highlights the effectiveness and efficiency of our method for text alignment. The detailed results can be found in $\\textcolor{blue}{\\textrm{Table 2 and 5}}$ of the new version.
Adapter: In the original paper, we selected the MLP as the adapter not for its performance but because its simplicity better emphasizes our main contribution. We acknowledge that ELLA's adapter can outperform a two-layer MLP, and we are currently working on combining our method with ELLA to create a stronger foundation model.

Here is a summary of $\\textcolor{blue}{\\textrm{Table 2 and 5}}$:

W2: The paper ELLA also introduces DPG-Bench. Even on this 5k evaluation set, VQAScore might be a good option to consider.

Thank you again for your advice. As mentioned above, we have added DPG-bench and utilized VQAscore on our 5k test set as two new evaluation metrics for text alignment. The results show strong consistency across three evaluation metrics (Denscore-O, VQAscore, and DPG-Bench) for text alignment, and our method consistently outperforms the baselines.

W3: It would have been nicer to also have results with any of the newer, more performant models (e.g. SDXL).

We have added new foundation models, SDXL. Our method also extends the maximum input token limit for SDXL and improves long-text alignment. All training and testing settings match the SD1.5 version, but with 1024 resolutions. The evaluation results on 5k test set are shown below:

Q1: I do not see other implementation details (dataset, other choices etc.) regarding the training of the Denscore model.

We apologize for this. The Denscore training setup is briefly described at the end of the training part in Section 5.1. We maintain consistency with Pickscore across nearly all settings, except for the training objectives and datasets. Specifically, we train our Denscore using Pickscore’s GitHub repository, making only modifications to the dataset loading and loss function code.

W3.1: Testing with alternative segmentation designs could reveal whether simpler or more complex methods yield better alignment.

We examine different segmentation strategies by comparing the approach of treating each sentence as a segment versus grouping several consecutive sentences into one segment, as long as the total token count remains under 77. Our new ablation study in $\\textcolor{blue}{\\textrm{Figure 11}}$ in the new version reveals no significant differences in the results.

Q1: Please better explain why the proposed split-and-merge approach can address the long-text alignment issue.

We would like to clarify that the split-and-merge approach alone does not address long-text alignment. Segment encoding is a method to overcome the input limitations of the encoder. To enhance long-text alignment, we need an additional two-stage training process that includes both supervised fine-tuning and preference optimization.

Q2: Please provide the ablation study clearly.

Thank you for your suggestion. In response, we have revised the paper to include new experimental results and detailed ablation studies, with all updates clearly highlighted in blue. We hope these additions address your concerns regarding the clarity of the ablation studies. If there are specific aspects you would like us to elaborate on further, we would be happy to provide additional results or analyses upon your guidance.

Thank you for your constructive feedback. Your insightful questions have improved our work very much. Below we respond to the comments in weaknesses (W) and questions (Q).

W1: These sections seem like a workaround rather than a robust solution to handling extended texts.

We agree that segment encoding alone cannot solve all long text encoding problems, but it can enhance any existing model with a maximum input limitation. In practice, as the total maximum length increases, data collection becomes increasingly challenging. For instance, generating 250-token prompts with LLAVA is straightforward, while collecting 1000-token prompts using existing models (e.g. LLMs) is significantly harder. Our method enables us to extend an encoder trained on a maximum of 250 tokens to accommodate 1000-token inputs. Additionally, a 1000-token prompt can often be divided into several relatively independent sections, each with fewer than 250 tokens, making segment encoding for these sections a practical and sensible choice.

W1.2: The mechanics of how segmentation and merging affect the underlying model's cross-attention dynamics remain underexplored.

Thank you for your advice. In $\\textcolor{blue}{\\textrm{Figure 12}}$ of the new version, we compare the original SD1.5 with our fine-tuned version and find that their cross-attention map behaviors are similar, regardless of whether segment encoding is used. Specifically, when the prompt accurately labels each object and references them in subsequent sentences, even though segment encoding does not process multiple sentences in a single forward pass, the attention maps for individual objects across different segments remain consistent. For example, the prompt is structured as "A dog xxx and a cat xxx. This dog xxx." Even though the details about this dog are divided into two segments in our setup, the model can identify the same dog and apply attributes from both segments accordingly. This shows that T2I models with segment encoding can identify and manage information across segments of long-text inputs.

W2: Future work should include evaluations on SD-3 and SDXL to substantiate claims of superiority over other methods in a more current setting.

We have added new foundation models, SDXL. Our method also extends the maximum input token limit for SDXL and improves long text alignment. All training and testing settings match the SD1.5 version, but with 1024 resolutions. The evaluation results on 5k test set are shown below:

W3: It raises concerns about the model's performance on shorter or more varied prompts.

Diversified Length: In $\\textcolor{blue}{\\textrm{Table 6}}$ of our revised paper, we assess the generation results for prompts of lengths about 15, 60, 120, 240, and 500 tokens using Denscore-O and VQAscore $1$ . For more details on the dataset construction and evaluation process, please see Appendix C.3 of the new version. The evaluation results in $\\textcolor{blue}{\\textrm{Table 6}}$ demonstrate that our method consistently outperforms current baselines.

Here is a summary of the VQAscore results in $\\textcolor{blue}{\\textrm{Table 6}}$ in Appendix C.3:

Diversified structure: For prompts with diversified structures, we test our model on DPG-Bench $1$ , which includes test prompts for various categories such as entity, attribute, relation, and count. The new results are shown in $\\textcolor{blue}{\\textrm{Table 5}}$ of Appendix C.3. Compared to the baselines, we have also made obvious improvements.

Here is a summary of $\\textcolor{blue}{\\textrm{Table 5}}$ in Appendix C.3:

$1$ Hu X, Wang R, Fang Y, et al. Ella: Equip diffusion models with llm for enhanced semantic alignment $J$ . arXiv preprint arXiv:2403.05135, 2024.

Thank you for raising the score! The evaluation results for SD1.5 and SDXL are summarized as follows:

According to these results, we observe that (1) our methods significantly improve both SD1.5 and SDXL, demonstrating their robustness. (2) Among the two final fine-tuned versions, longSDXL clearly outperforms longSD1.5 in terms of long text alignment, indicating that a stronger foundation model achieves better performance limits. (3) The improvement is more pronounced in SD1.5, this is because the pretrained version of SDXL is already better than that of SD1.5. We have uploaded a new revision that includes these analyses in Appendix C.4, with changes highlighted in blue for clarity.

Q3: How were the 5k images in the test set specifically selected from datasets like SAM and COCO2017?

We first collect about 2 million images from open datasets, including 500k from SAM, 100k from COCO2017, 500k from LLaVA (a subset of the LAION/CC/SBU dataset), and 1 million from JourneyDB. After obtaining this new dataset, we randomly select 5k images from it as the test set without any human intervention. The 5k test set is not used in any training stage. More information about our dataset can be found in Section 5.1.

Q4: Briefly explain the selection of CLIP-H and HPSv2, as well as the chosen evaluation metrics?

Thank you for your advice. We have added more information about these methods in Section 5.1 and 5.2 of the new version. The explanations are as follows:

• CLIP-H and HPSv2: To demonstrate that our analysis of the CLIP-based preference models is generally applicable, we compare four different CLIP-based models: the pretrained CLIP, the single-value preference models Pickscore and HPSv2, as well as our segment-level preference model, Denscore.
• FID: FID evaluates the distribution distance between the dataset and generated images.
• Denscore: Denscore assesses human preference for generated images, while Denscore-O and VQAscore focuses on the text alignment of those images.
• VQAscore: VQAscore $1$ also focuses on the text alignment of generated images.
• DPG-bench: DPG-bench $2$ is a general benchmark that includes test prompts for categories like entity, attribute, relation, and count.

Thank you for your constructive feedback. Your insightful questions have improved our work very much. Below we respond to the comments in weaknesses (W) and questions (Q).

W1 & Q1: The performance of the segment-level encoding strategy under different text length conditions.

In $\\textcolor{blue}{\\textrm{Table 6}}$ of our revised paper, we assess the generation results for prompts of lengths about 15, 60, 120, 240, and 500 tokens using Denscore-O and VQAscore $1$ . For more details on the dataset construction and evaluation process, please see Appendix C.3 of the new version. The evaluation results in $\\textcolor{blue}{\\textrm{Table 6}}$ demonstrate that our method consistently outperforms current baselines.

Here is a summary of the VQAscore results in $\\textcolor{blue}{\\textrm{Table 6}}$ in Appendix C.3:

$1$ Lin Z, Pathak D, Li B, et al. Evaluating text-to-visual generation with image-to-text generation $C$ //European Conference on Computer Vision. Springer, Cham, 2025: 366-384.

W2 & Q2: Does the reweighting strategy have any quantitative results to demonstrate its effectiveness in reducing overfitting? Please provide detailed steps or pseudocode of it.

The pseudocode of the reweighting strategy is

# Calculate the text-unrelated component $V$
for image, text in dataset:
    text_emb_list.append(CLIP(text)  / ||CLIP(text)||)
common_text_emb = mean(text_emb_list) / ||mean(text_emb_list)|| 

# Calculate the reweighted loss
image_emb = CLIP(image) / ||CLIP(image)||
text_emb = CLIP(text)  / ||CLIP(text)||
# Equation 8
# Ratio controls the reweighted proportion
text_emb_reweight = text_emb - (1 - ratio) * (text_emb * common_text_emb) * common_text_emb  
loss = text_emb_reweight * image_emb 

The quantitative results for reducing overfitting are presented in $\\textcolor{blue}{\\textrm{Figure 5}}$. When the ratio approaches 1, the Denscore decreases while the FID (FID evaluates the distribution distance between the dataset and generated images) increases, indicating generation overfitting to the Denscore and a departure from the correct image distribution. At a ratio of 0.3, we can maintain a relatively stable FID, suggesting that generated images stay within the desired distribution. Visual results are available in $\\textcolor{blue}{\\textrm{Figures 17}}$ of the new version. At a ratio of 1.0, there is clear evidence of overfitting, as all images show similar patterns; whereas at a ratio of 0.3, the images align best with human preferences without obvious overfitting.
In addition to the experimental results, our paper identifies the main cause of overfitting: when training diffusion models with CLIP-based preference models, the model tends to optimize towards the text-unrelated component $\\mathbf{V}$, resulting in generated images that appear similar regardless of the text input. Based on this finding, we choose to reweight the item $\\mathbf{V}$ during training.

W3: Discuss the alignment effectiveness when dealing with texts that have complex contextual dependencies or require strong semantic understanding.

We would like to clarify that aligning long texts and complex texts are two distinct issues. A single-sentence prompt can also have intricate dependencies, but our paper focuses more on the length aspect. To demonstrate the effectiveness of our current methods in handling complex dependencies and semantic understanding we use DPG-Bench $2$ , which includes test prompts for various categories such as entity, attribute, relation, and count. The new results are shown in $\\textcolor{blue}{\\textrm{Table 5}}$ of Appendix C.3 of the new version. Compared to the baselines, we have also made obvious improvements. In addition, we agree that complex dependencies and semantic understanding are challenging tasks and remain far from being fully resolved. We have also included these points in the limitations section of the new version, with changes highlighted in blue for clarity.

Here is a summary of $\\textcolor{blue}{\\textrm{Table 5}}$ in Appendix C.3:

$2$ Hu X, Wang R, Fang Y, et al. Ella: Equip diffusion models with llm for enhanced semantic alignment $J$ . arXiv preprint arXiv:2403.05135, 2024.

Dear Reviewer 9HBG,

Thank you once again for your constructive feedback. We would like to kindly remind you that we have included additional experiments to:

• Validate the performance of our approach under varying text-length conditions.
• Assess alignment effectiveness when handling complex texts.
• Provide detailed evidence of the effectiveness of the reweighting strategy in addressing overfitting.

We have also clarified the construction of the test set, the selection of models (CLIP-H and HPSv2), and the evaluation metrics used.

As the discussion period is coming to a close in two days, we look forward to your response and would be happy to address any further comments or questions you may have.

Best,

The Authors

Thank you for your positive feedback. Your insightful questions have improved our work very much. Below we respond to the comments in weaknesses (W).

W1: Why are multiple <sot> tokens retained?

Thank you for raising this question. In the revised paper, we have added an ablation study ($\\textcolor{blue}{\\textrm{Figure 10 and 11}}$ in Appendix A) to explore three scenarios: (1) removing all <sot> tokens, (2) retaining a single <sot> token, and (3) retaining all <sot> tokens. Our findings indicate that removing all <sot> tokens leads to a significant performance drop, while keeping one or more <sot> tokens yields comparable results. This analysis is elaborated in Appendix A, with changes highlighted in blue for clarity.

W1.2: The difference in the aesthetic quality of the generated results after reweighting.

In the revised paper, we have provided multiple generation images ($\\textcolor{blue}{\\textrm{Figure 17}}$ in Appendix C.2) at different reweight ratios while keeping all other settings the same to illustrate differences in aesthetic quality. A ratio of 1 indicates the original preference loss, resulting in significant overfitting, where all images exhibit similar patterns regardless of the inputs. A ratio of 0 implies that the loss only considers the text-relevant part, leading to low image quality that does not align with human preferences. We observe a ratio of 0.3 yields the best visual quality among these options.

W1.3: Some visualizations of the outcomes from the two loss functions of Denscore.

For long-text alignment, the two losses are similar; however, for aesthetics, Equation 10 eliminates the influence of the first item on the text-irrelevant part $V$, allowing it to focus more on aesthetic aspects. In $\\textcolor{blue}{\\textrm{Figure 13}}$ of Appendix B.1 in the revised paper, we present updated visual results for retrievals with the highest scores using the text-irrelevant part $V$ under two Denscore losses. We find that retrieval results from the text-irrelevant part $V$ trained with Equation 10 align more closely with human aesthetic preferences.

W2: Can the current innovation still contribute to VLMs?

An important direction for VLM is to enable it to understand and generate visual input simultaneously $1,2$ . Our methods could also be potentially used to train VLMs to generate images similarly to T2I diffusion models. Furthermore, in CLIP, there is a symmetrical relationship between image and text. By reversing their roles—using the image as input and generating text as output—we could also enhance VLM’s image captioning capabilities $3$ , presenting another interesting potential benefit.

$1$ Team C. Chameleon: Mixed-modal early-fusion foundation models $J$ . arXiv preprint arXiv:2405.09818, 2024.

$2$ Zhou C, Yu L, Babu A, et al. Transfusion: Predict the next token and diffuse images with one multi-modal model $J$ . arXiv preprint arXiv:2408.11039, 2024.

$3$ Liu H, Li C, Wu Q, et al. Visual instruction tuning $J$ . Advances in neural information processing systems, 2024, 36.

W3: Could you clarify where the potential source of this overfitting lies?

Previous work $4$ has observed the overfitting problem in preference optimization of diffusion models, while our paper identifies its main cause and solves it. We find that when using CLIP-based preference models to train diffusion models, the training loss can be divided into a text-related component $\\mathcal{C}\_{P}^{\\bot}(p) \* \\mathcal{C}\_{X}$ and a text-unrelated component $\\mathbf{V} \* \\mathcal{C}\_{X}$. The text-unrelated component makes up a large part of the entire loss, and it is relatively easy for the model to learn since $\\mathbf{V}$ remains stable. As a result, the model tends to optimize towards $\\mathbf{V}$ regardless of the text input, leading all generated images to look similar. More information can be found in Section 4.2, and visual results can be found in $\\textcolor{blue}{\\textrm{Figures 6 and 17}}$ in the revised paper.

$4$ Wu X, Hao Y, Zhang M, et al. Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models $J$ . arXiv preprint arXiv:2405.00760, 2024.