Aligning Text to Image in Diffusion Models is Easier Than You Think

Wa: Novelty over REPA and Prompt Optimization

We thank the reviewer for the opportunity to clarify our contributions. While inspired by REPA [1], our method differs in both objective and application. REPA aims to accelerate training by aligning DiT features with external discriminative representations. In contrast, our work targets text-image alignment by implicitly aligning internal representations of two different domains through contrastive learning.

There are several works from prompt optimization. CoOP [2] introduces learnable soft prompts for CLIP models at the input level by replacing fixed prompt templates (e.g., “a photo of a [CLS]”) with learnable embeddings that help differentiate between predefined class labels. However, our approach generalizes this idea by adopting soft tokens within the model's internal layers, independent of predefined classes. This extension enables fine-grained, layer-wise modulation of semantic alignment signals.

To our knowledge, this is the first attempt to optimize soft tokens in diffusion models by modulating internal features beyond input embeddings. Our results show that these tokens significantly improve generative quality conditioning on given text by influencing feature propagation.

To validate this design, we compare with LoRA applied to the top 5 layers, which outperforms full-layer tuning. As shown in Table 1, our method—Soft Tokens + Contrastive Loss—achieves the best performance on ImageReward, CLIP, HPS, and FID, while remaining competitive on PickScore and LPIPS, demonstrating the complementary strength of our architectural and training choices.

Table 1. Quantitative comparison of parameter-efficient tuning methods

References
[1] Yu, Sihyun, et al. "Representation alignment for generation: Training diffusion transformers is easier than you think." arXiv preprint arXiv:2410.06940 (2024).
[2] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022.

---

Wb1: Compare the design with the original REPA

We sincerely thank the reviewer for the thoughtful suggestion. We agree that pretrained self-supervised visual encoders provide powerful external knowledge, and we appreciate the insight into comparing with such representations as explored in REPA [1]. However, we would like to highlight a key distinction in the training objectives and representation targets between REPA and our proposed method.

REPA aims to accelerate pretraining stage by explicitly aligning the internal features of the diffusion model (DiT) with external discriminative features obtained from a pretrained visual encoder. This is feasible because the internal generative features and the external discriminative features exhibit natural similarity in the image domain.

In contrast, SoftREPA is designed to enhance text-image alignment of pretrained models by introducing learnable soft text tokens that explore a richer textual feature space within the model. As the reviewer pointed out, our method operates entirely within the latent space of the diffusion model itself, relying solely on the model’s internal priors. Since the soft tokens are applied to textual features, there is no inherently sufficient alignment with visual representations from external encoders, making direct supervision from such representations less applicable in our setting.

That said, we believe the reviewer’s suggestion opens up an interesting future direction. It may be promising to explore soft image token tuning using pretrained visual features, which could potentially enhance the image representation space in a complementary way. While this is beyond the scope of our current work, we are excited by this possibility. Once again, we thank the reviewer for the thoughtful suggestion.

---

Wb2: Finetuning Baseline Without Contrastive Loss

We thank the reviewer for raising this insightful point, which directly addresses the contribution of soft tokens and the contrastive learning objective. To further clarify their individual roles, we conducted an additional ablation study as suggested.

To assess whether the observed improvements stem from continued training on high-quality data rather than the contrastive loss itself, we performed a new ablation where the model retains the soft token architecture but is fine-tuned using the original denoising score matching (DSM) loss on the same dataset. This setup isolates the effect of contrastive learning from soft tokens. As shown in Table 2, this variant consistently underperforms compared to our full method, confirming that the gains are not merely due to extended training.

Table 2. Quantitative comparison of extra DSM training

---

Wb3: Concerns About Hyperparameters and Generalization

We would like to clarify that although new parameters are introduced, we observed consistent trends across different models (SD3, SD1.5, SDXL). For instance:

• In terms of layer placement, all models showed best performance when soft tokens were applied only to earlier blocks (SD3) or down blocks of UNet(SD1.5, SDXL).
• The time scheduling parameter was kept constant across all models, and we did not observe high sensitivity to its value.
• The optimal number of soft tokens for SD3 lies between 4 and 8, and this range also covers the optimal settings for SD1.5 and SDXL, which show similarly strong performance across a broader range.

These consistent tendencies suggest that our method behaves robustly across architectures, requiring minimal model-specific tuning—similar to other generation techniques like classifier-free guidance (CFG).

Furthermore, to evaluate the generalizability of SoftREPA, we conducted experiments on SD1.5 and SDXL by training only with the contrastive loss, omitting the denoising score matching (DSM) term entirely. As shown in Table 3, even without DSM, the model performs competitively, further supporting that our contrastive learning framework is effective and broadly applicable.

Table 3. Quantitative comparison on the generalizability of SoftREPA

---

---

Q1: Expectation Estimation via Monte Carlo

As described in Section 3.2 of the paper, we approximate the expectation in the logit computation l(x,y) using a single Monte Carlo sample for efficiency. To further stabilize training, we use the same sampled timestep and noise across all pairs within a minibatch when computing the loss. During training, the timestep ttt is sampled from a uniform discrete distribution over [0,999], which is a standard choice in diffusion-based models. During inference, we use a fixed sequence of timesteps determined by the scheduler: DPM-Solver++ for SD3, and DDIM for SD1.5 and SDXL. We did not observe significant performance degradation with a single sample in practice, though increasing the number of samples could potentially improve stability at the cost of computational overhead.

---

Q2: Time Scheduling Parameter $\tau(t)$

The scheduling parameter $\tau(t)$ used in constructing the SoftREPA logit is set to a constant value of 0.07, which serves as a temperature parameter to scale the logits. This value is commonly used in contrastive learning literature and was found to work well empirically. We did not observe strong sensitivity to this hyperparameter in our experiments.

---

Q3: Data Source for Training

We used the COCO dataset to train the soft tokens, as described in Section 4 (Text-to-Image Generation) of the paper.

---

W1W3Q1: Lack of Discussion on Positive/Negative Pair Selection, Counting Drop, and Dataset Choice

Per reviewer's request, we conducted an additional experiment by training SoftREPA with hard negative samples. For each COCO caption, we automatically generated three hard negatives using GPT4.1-mini, by minimally modifying the original caption with respect to: Counting (e.g., altering the number of objects), Color (e.g., changing the color of one object), Position (e.g., modifying spatial arrangement). During training, for each training sample, these hard negatives were incorporated alongside other negative samples used in our original method.

As shown in Table 1, we did not observe a significant improvement from adding hard negatives to the training data. This may be due to the limited number of hard negatives, as we generated only one altered caption per attribute. We hope to conduct further experiments on the impact of hard negatives in future work.

Table 1: Effect of Incorporating Hard Negatives into Our Method

---

W2Q2: Comparisons to Relevant Methods

To provide a more comprehensive comparison, we conducted additional experiments covering both (1) parameter-efficient fine-tuning methods and (2) approaches aimed at improving text-to-image alignment.

(1) In terms of lightweight tuning strategies, we compare Soft Tokens with LoRA, applied to the top 5 transformer layers for optimal performance. As shown in Table 2, the combination of Soft Tokens + Contrastive Loss consistently outperforms LoRA-based tuning across multiple metrics, including ImageReward, CLIP, HPS, and FID, while performing competitively on PickScore and LPIPS. This confirms the effectiveness of our architectural design in conjunction with contrastive learning.

(2) Regarding alignment-oriented approaches, we would like to clarify that contrastive learning is strongly relevant to Reinforcement Learning from Human Feedback(RLHF) and Direct Preference Optimization (DPO) frameworks [1]. Recent work [1] shows that both RLHF and DPO can be reframed as contrastive learning objectives via mutual information maximization. Since our approach shares the underlying principle, we compare SoftREPA against RankDPO [2] and additionally CaPO [3] in Table 3. CaPO [3] is a recently proposed diffusion-RL method built upon SD3, which is reported to outperform Diffusion-DPO [4], as noted in our related works.

Additionally, we include CFG++ [5], a training-free technique that improves generation quality by refining the classifier-free guidance step. Since CFG++ is originally implemented on SD1.5 and SDXL, we followed the official implementation details provided in their repository.

We evaluate all models on the GenEval benchmark. As presented in Table 3, our method outperforms baselines on several key categories, including Single, Two, Colors, Position, and Color Attribute, confirming that SoftREPA not only provides an effective tuning method but also achieves strong performance in challenging alignment settings.

---

Table 2. Quantitative comparison of parameter-efficient tuning methods

---

Table 3. GenEval benchmark comparing related works

---

References
[1] Lv, Xufei, et al. "The Hidden Link Between RLHF and Contrastive Learning." arXiv preprint arXiv:2506.22578 (2025).
[2] Karthik, Shyamgopal, et al. "Scalable ranked preference optimization for text-to-image generation." arXiv preprint arXiv:2410.18013 (2024).
[3] Lee, Kyungmin, et al. "Calibrated multi-preference optimization for aligning diffusion models." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.
[4]Wallace, Bram, et al. "Diffusion model alignment using direct preference optimization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[5] Chung, Hyungjin, et al. "Cfg++: Manifold-constrained classifier free guidance for diffusion models." arXiv preprint arXiv:2406.08070 (2024).

W1Q1: Comparison with LoRA and Motivation for Soft Tokens

We appreciate the reviewer giving us the opportunity to clarify the motivation for soft tokens. The motivation of SoftREPA comes from REPA [1], which shows that aligning the internal representations of DiT with an external pre-trained visual encoder during training significantly improves both discriminative and generative performance. In this paper, we aim to enhance text-image alignment in pretrained text-to-image (T2I) generative models, leveraging these ideas to align DiT’s internal representations through contrastive learning, which is a core principle of representation learning.

The architectural motivation to use soft tokens is primarily inspired by CoOP [2], as stated in our related works. CoOP proposed prompt learning methods for CLIP-like vision-language models that replace the fixed prompt “a photo of a [CLS]” with learned soft prompts that better distinguish between a predefined set of [CLS]. This implies that we can learn an additional feature space that maximizes the discrepancy between general texts, which further enhances alignment with images. While CoOP learns tokens only in the prompt space, we expand this into the feature space within the model and remove the constraint of a predefined class set. This enables more flexible and implicit text-image representation alignment.

To evaluate the effectiveness of our design, we further investigated whether contrastive learning alone could yield similar benefits. For this, we incorporated LoRA as an alternative tuning strategy in place of soft tokens. LoRA was applied to the top 5 transformer layers, which we found to be a more stable and effective configuration than applying it to the full model.

As presented in Table 1, the combination of Soft Tokens + Contrastive Loss consistently achieves stronger performance across key metrics such as ImageReward, CLIP, HPS, and FID, while remaining competitive on PickScore and LPIPS. These results affirm that both soft tokens and contrastive learning contribute independently and meaningfully to enhancing text-image alignment.

Table 1. Quantitative comparison of parameter-efficient tuning methods

References
[1] ​​Yu, Sihyun, et al. "Representation alignment for generation: Training diffusion transformers is easier than you think." arXiv preprint arXiv:2410.06940 (2024).
[2] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022.

---

W2: Qualitative vs Quantitative Results Contradictory

We thank the reviewer for pointing out this discrepancy. In Figure 3, the original motivation to include the example of "Three zebras" was to qualitatively illustrate improvements in image quality and fidelity, rather than to make claims about counting ability. While the SD3 baseline struggles to generate complete and realistic zebras, our method produces more coherent and visually accurate outputs, including improved background details. To avoid confusion, we will replace this with other example in the final paper.

Dear Reviewer 2H9G,

As the deadline for the Author-Reviewer discussion period is approaching, we would like to kindly ask whether our responses have sufficiently addressed your concerns and questions.

We hope our responses have clarified the points you raised, and we are happy to discuss further if you have any remaining questions.

Sincerely, The Authors

W1/Q1: Necessity and Role of Soft Text Tokens

We appreciate the reviewer’s question regarding the specific contribution of soft text tokens apart from the contrastive learning objective. To investigate this, we performed an ablation study using LoRA as an alternative parameter-efficient tuning method in place of soft tokens. We applied LoRA to the top 5 transformer layers, as this configuration proved more effective than applying LoRA across all layers.

As shown in Table 1, the combination of Soft Tokens + Contrastive Loss consistently achieves stronger results across multiple metrics, including ImageReward, CLIP, HPS, and FID, and performs comparably on PickScore and LPIPS. This demonstrates that both the architectural design of soft tokens and the contrastive loss independently contribute to improved text-image alignment.

In addition, we would like to clarify the role and motivation behind the soft text tokens. Our architectural design is primarily inspired by CoOP [1], which proposed a prompt learning strategy for CLIP-based vision-language models that replaces the fixed prompt (“a photo of a [CLS]”) with learned soft prompts capable of distinguishing among a set of predefined classes. This suggests that prompt-tuning enables the model to explore richer semantic embeddings. While CoOP focuses on learning prompts in the input space, we generalize this idea by propagating learned soft tokens through the network, allowing the model to align text and image representations in the internal feature space without relying on predefined class labels.

This architectural choice enables a more flexible and implicit alignment mechanism, encouraging the model to maximize the mutual information between text and image embeddings by refining the internal feature dynamics of the model.

Table 1. Quantitative comparison of parameter-efficient tuning methods

References
[1] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022.

---

W2Q2: Clarification on LPIPS and Image Quality

We appreciate the reviewer’s concern regarding the use of LPIPS and other image quality metrics. In our study, we computed the LPIPS metric to measure the perceptual distance between the generated image and the real image corresponding to the given prompt. While LPIPS can indicate the similarity between the target and source images in editing tasks, we initially believed it could also reflect image quality in generation tasks. However, other metrics such as ImageReward, PickScore, and HPS, which effectively reflect image quality, demonstrate that our method preserves image quality while improving text-image alignment.

---

W3: Lack of Comparison with Alignment-Improving Baselines

We would like to note that our approach is grounded in contrastive learning, which shares a deep theoretical connection with recent preference optimization methods in diffusion models. As highlighted in recent work [1], both Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) can be reinterpreted as forms of mutual information maximization—a foundational principle also underlying contrastive learning. Motivated by this connection, we compare our method, SoftREPA, with the diffusion-RL-based methods RankDPO [2] and CaPO [3], which are built on SD3 backbone. Notably, CaPO has been reported to outperform Diffusion-DPO [4], which we cited in our related work.

In addition, we include CFG++ [5], a training-free approach that improves generation quality by refining the classifier-free guidance (CFG) step. Although CFG++ is implemented on SD1.5 and SDXL, we follow the official guidance provided in their public repository to ensure a consistent comparison.

To evaluate alignment quality, we report results on the GenEval benchmark. As shown in Table 2, our method performs competitively overall and demonstrates superior results in several categories, including Single, Two, Colors, Position, and Color Attribute. These results support that SoftREPA offers a robust and effective framework for improving text-image alignment compared to both preference-optimized and training-free baselines.

Table 2. GenEval benchmark comparing related works

---

References
[1] Lv, Xufei, et al. The Hidden Link Between RLHF and Contrastive Learning. arXiv preprint arXiv:2506.22578 (2025).
[2] Karthik, Shyamgopal, et al. Scalable Ranked Preference Optimization for Text-to-Image Generation. arXiv preprint arXiv:2410.18013 (2024).
[3] Lee, Kyungmin, et al. Calibrated Multi-Preference Optimization for Aligning Diffusion Models. CVPR 2025.
[4] Wallace, Bram, et al. Diffusion Model Alignment Using Direct Preference Optimization. CVPR 2024.
[5] Chung, Hyungjin, et al. CFG++: Manifold-Constrained Classifier-Free Guidance for Diffusion Models. arXiv preprint arXiv:2406.08070 (2024).

We sincerely thank the reviewer for their thoughtful and constructive feedback. The comment helped us gain deeper insight and refine the contribution of our work. Below, we address the concern in detail.

---

W1/Q1: Isolating the Contribution of Contrastive Loss

We appreciate the reviewer’s suggestion to examine the effect of contrastive loss independent of the soft token architecture. In response, we conducted an ablation study using LoRA as an alternative in place of soft tokens. Specifically, we applied LoRA to the top 5 transformer blocks, which we found to be more effective than applying LoRA across all layers.

As shown in Table 1, the combination of Soft Tokens + Contrastive Loss consistently outperforms other variants across metrics such as ImageReward, CLIP, HPS, and FID, and performs comparably on PickScore and LPIPS. This result highlights that both soft tokens and contrastive learning contribute independently and positively to text-image alignment.

Moreover, these findings support the underlying motivation of SoftREPA: to explore the text feature space to better text-image representation alignment. While LoRA modifies the attention projection layers in both the text and image encoders, this direct parameter modification may lead to suboptimal image generation, as evidenced by a higher FID score (70.33) compared to our soft token training approach (60.76).

Table 1. Quantitative comparison of parameter-efficient tuning methods

---

W2: Drop in FID and LPIPS

We acknowledge that our method may affect FID and LPIPS, as it emphasizes stricter text-image alignment. However, these metrics have limitations in evaluating generation quality alone—FID is influenced by both diversity and quality, while LPIPS is more suited to measure similarity between a real image and a generated image on the given text, rather than assessing image quality.

Instead, other metrics such as ImageReward, PickScore, and HPS reflect the image quality aligned with human preferences. These metrics consistently demonstrate that our method maintains perceptual quality while improving text-image alignment.

---

W3Q4: Counting Accuracy and Hard Negatives

We thank the reviewer for suggesting alternative approaches to improve counting accuracy. To address this, we trained SoftREPA with hard negatives added to each sample. Specifically, for each COCO caption, we generated three hard negative examples focusing on the following aspects: object count, color, and position. Using GPT-4.1-mini, we crafted prompts to minimally modify the original caption by altering the number of objects (counting), the color of an object (color), or the position of an object (position). During training, for each training sample, these hard negatives were incorporated alongside other negative samples.

As shown in Table 2, we did not observe a significant improvement from adding hard negatives to the training data. This may be due to the limited number of hard negatives, as we generated only one altered caption per attribute. We hope to conduct further experiments on the impact of hard negatives in future work.

Table 2: Effect of Incorporating Hard Negatives into Our Method

---

W4: Title May Be Overstated

We will tone down and revise it to better reflect the scope of our method.

---

Q2: Complex Compositional Understanding

We thank the reviewer for this important question regarding compositional understanding. We address this capability primarily in the Editing tasks. To evaluate this aspect, we used LLaVA [1] to generate long, detailed text prompts for images in the DIV2K dataset. These prompts contain nuanced edits involving spatial relationships, attribute modifications, and object substitutions, requiring complex compositional reasoning.

Below, we present two representative prompts we used. In each case, the target prompt modifies multiple attributes and compositional elements from the original source prompt, such as changing object positions, attributes (e.g., material, color), or introducing new objects in specific spatial contexts.

In the DIV2K Editing benchmark, our model with trained soft tokens consistently outperforms the base model on metrics that evaluate background preservation and alignment with the target prompt (refer to Table 2 in the paper). These results suggest that the incorporation of soft tokens enhances the model’s ability to reason over complex, multi-object compositions and produce more faithful edits.

---

Example 1: The image features a vase filled with a variety of flowers, including pink, white, and red ones. The vase is placed on {a table, and it is a beautiful blue color $\rightarrow$ a wooden pedestal, and it is a rich, dark brown color}. The flowers are arranged in a bouquet, creating an eye-catching display. The combination of the vibrant colors and the elegant blue vase makes the scene visually appealing.

Example 2: The image features a room with a wooden wall and a wooden table. On the wall, there are several framed pictures, including a picture of {a man wearing a suit $\rightarrow$ a woman wearing a dress}. The table is covered with a variety of items, such as books, a vase, and a bottle. There is also a {jacket hanging $\rightarrow$ a guitar hanging} on the wall, adding to the room's decor. The room appears to be a combination of a living space and a workspace, with the wooden wall and table providing a cozy and functional atmosphere.

---

References
[1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023.

---

Q3: Diffusion RL + SoftREPA Complementarity

We thank the reviewer for this insightful question regarding the potential complementarity between SoftREPA and existing preference-tuning methods. We agree that these approaches are not mutually exclusive, and we explored this direction with additional experiments. To investigate whether SoftREPA can be combined with diffusion-RL methods, we conducted experiments using two representative Diffusion RL frameworks: Diffusion-DPO [1], which performs preference optimization, and DDPO [2], which utilizes policy optimization based on an external reward (e.g., BERTScore [3]).

To test complementarity, we adopted a plug-and-play approach: we integrated pretrained soft tokens from SoftREPA into the inference pipelines of these diffusion-RL models without further training.

As shown in Table 3, the addition of SoftREPA's soft tokens leads to notable performance gains in both cases. Diffusion-DPO + Soft Tokens shows improvements in ImageReward, CLIP, FID, and LPIPS, and remains comparable in PickScore and HPS. DDPO + Soft Tokens yields consistent improvements across all metrics, indicating strong synergy between policy-based optimization and our contrastive alignment objective.

These results suggest that SoftREPA and Diffusion-RL methods are complementary, while Diffusion-DPO and DDPO aim to align outputs using preference or reward supervision, SoftREPA enhances internal representation alignment through contrastive learning. Their combination allows for leveraging strengths from both strategies—efficient text-image alignment and explicit preference feedback—to further improve generation quality.

Table 3. Complementarity with Diffusion RL methods

References
[1] Wallace, Bram, et al. "Diffusion model alignment using direct preference optimization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[2] Black, Kevin, et al. "Training diffusion models with reinforcement learning." arXiv preprint arXiv:2305.13301 (2023).
[3] Zhang, Tianyi, et al. "Bertscore: Evaluating text generation with bert." arXiv preprint arXiv:1904.09675 (2019).

Dear Reviewer 1MZz,

As the deadline for the Author-Reviewer discussion period is approaching, we would like to kindly ask whether our responses have sufficiently addressed your concerns and questions.

We hope our responses have clarified the points you raised, and we are happy to discuss further if you have any remaining questions.

Sincerely, The Authors