Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models

We are grateful for the reviewers' thoughtful evaluations and detailed comments. Our responses to the raised points are presented below.

---

W1: [Long and complex prompts]

Text encoders in diffusion models are often limited by a fixed token length, which can pose challenges for handling long and complex prompts. For example, many diffusion models using CLIP encoders are restricted to 77 tokens, and the tokens beyond this limit are typically truncated and ignored. Since DATE provides guidance through an external function and does not rely solely on the internal text encoder. This allows DATE to influence the generation process beyond the model’s fixed token-length limitation, making it a promising solution for handling long and complex prompts.

To investigate this, we perform experiments on the LongAlign (Liu et al., 2025), which contains prompts of substantial length and complexity. We sample 300 prompts and use ImageReward (20% updates) as the evaluation function. For ImageReward, each long prompt is split into individual sentences, and at each update step, one sentence is randomly selected to compute the score and guide the embedding update. We use Denscore (Liu et al., 2025) for the metric.

[Table: Results for long prompts on LongAlign dataset (SD v1.5)]

[Table: Average Denscore by sentence position]

These results show that DATE improves Denscore compared to the baseline. Notably, the improvement is more significant for sentences appearing later in the prompt, suggesting that DATE can inject additional semantic guidance in regions where the base model typically underperforms due to token truncation. These results suggest that DATE can serve as an effective mechanism for injecting auxiliary information into diffusion models, especially in scenarios where prompt length exceeds the native capacity of the text encoder.

(Liu et al., 2025) Improving Long-Text Alignment for Text-to-Image Diffusion Models, ICLR 2025.

---

W2 & Q: [More powerful generation models]

Thank you for the suggestion. First, we would like to emphasize that DATE is an inference-time method designed to improve existing image generation models by guiding the sampling process. It can be applied on top of powerful base models to further improve their performance without requiring retraining or access to large-scale paired data. That said, we do not claim that our approach outperforms methods that rely on extensive training with massive datasets and computing resources. Rather, our goal is to provide an adaptive improvement that can complement such powerful image generation models, even when full retraining is impractical or infeasible.

While DATE does not modify model weights, it does require access to model weights due to computation of gradients, which makes it challenging to apply to closed-source models such as GPT-4o or Gemini. On the other hand, for open-source models, DATE is readily applicable. As shown in Table 1 of the main text, DATE consistently improves performance across all metrics when applied to PixArt-alpha, which is a DiT-based image generation model. To further evaluate DATE’s applicability to a broader range of strong base models, we additionally apply DATE to FLUX (guided-distilled version), SD-XL and SD3. For each model, we follow the default configurations provided in the HuggingFace diffusers library and use ImageReward as the evaluation function.

[Table: Results on COCO with recent stronger base models]

The results above follow the evaluation setup in Table 1 of the main text. Across all modern architectures, DATE consistently improves performance on all metrics compared to the fixed text embedding baseline. These results suggest that DATE can be transferred to a range of stronger base models and consistently enhance their generation quality.

---

W3: [Reliance on evaluation function]

We acknowledge that DATE depends on the reward function, as discussed in Appendix E. Here, we offer theoretical, empirical, and practical perspectives.

(Theoretical perspective)

Theorem 2 shows DATE aligns text embeddings with the reward function while also regularizing with the model's text-conditional likelihood. This means that DATE does not blindly overfit to the reward function but instead applies regularized guidance informed by the underlying diffusion model. This effect can reduce the risk of overfitting to the proxy metrics.

(Empirical perspective)

Empirically, as shown in Table 1 of the main text, DATE improves performance on both the reward function used for optimization and other metrics not used during optimization (e.g., FID). To further validate this generalization, we conduct additional experiments using PickScore (Kirstain et al., 2023) as both an optimization objective and an external evaluation metric. As shown in the table, DATE consistently improves performance across all metrics. This suggests that DATE’s improvements reflect broader enhancements in generation quality. Ongoing research continues to explore more faithful evaluation metrics for text-to-image generation, and DATE can be readily combined with such advances.

[Table: Results on COCO with SD v1.5 of Table 1 in the main text, including PickScore.]

(Kirstain et al., 2023) Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation, NeurIPS 2023.

(Practical perspective)

Our method also supports multi-objective optimization, which can further mitigate overfitting to a single metric. In Figure 4 of the main text, we show results using a combined objective of CLIP score and ImageReward. Interestingly, this combination outperforms the use of each metric individually. This result highlights the potential of our method to leverage synergy between multiple objectives for more robust guidance.

In summary, while reward function dependency is an important consideration, DATE incorporates both theoretical regularization and practical flexibility to address it effectively.

We thank the reviewer for thoughtful comments. We address the questions in the following response.

---

W1: [Reliance on evaluation function]

We acknowledge that DATE depends on the reward function, as discussed in Appendix E. Here, we offer theoretical, empirical, and practical perspectives.

(Theoretical perspective)

Theorem 2 shows DATE aligns text embeddings with the reward function while also regularizing with the model's text-conditional likelihood. This means that DATE does not blindly overfit to the reward function but instead applies regularized guidance informed by the underlying diffusion model. This effect can reduce the risk of overfitting to the proxy metrics.

(Empirical perspective)

Empirically, as shown in Table 1 of the main text, DATE improves performance on both the reward function used for optimization and other metrics not used during optimization (e.g., FID). To further validate this generalization, we conduct additional experiments using PickScore (Kirstain et al., 2023) as both an optimization objective and an external evaluation metric. As shown in the table, DATE consistently improves performance across all metrics. This suggests that DATE’s improvements reflect broader enhancements in generation quality. Ongoing research continues to explore more faithful evaluation metrics for text-to-image generation, and DATE can be readily combined with such advances.

[Table: Results on COCO with SD v1.5 of Table 1 in the main text, including PickScore.]

(Kirstain et al., 2023) Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation, NeurIPS 2023.

(Practical perspective)

Our method also supports multi-objective optimization, which can further mitigate overfitting to a single metric. In Figure 4 of the main text, we show results using a combined objective of CLIP score and ImageReward. Interestingly, this combination outperforms the use of each metric individually. This result highlights the potential of our method to leverage synergy between multiple objectives for more robust guidance.

In summary, while reward function dependency is an important consideration, DATE incorporates both theoretical regularization and practical flexibility to address it effectively.

---

W2: [Multiple text embeddings]

Thank you for the thoughtful feedback. As we present dynamically adapted text embedding to the diffusion process, our method (DATE) is distinct from a static set of multiple text embeddings, such as SD3.

Regarding the related work section, we agree that methods like SD3 are relevant to the goal of improving text embeddings, and we will include a discussion of such methods in the revised version to provide a more comprehensive context. That said, our intention in lines 70-72 was to highlight a specific distinction: while methods like SD3 improve text embeddings (e.g., by using multiple text encoders), they generally do not adapt these embeddings dynamically across diffusion timesteps. In contrast, DATE is motivated by this limitation and proposes an inference-time mechanism for updating the text embedding dynamically over timesteps. Importantly, this difference makes DATE complementary to approaches like SD3. We demonstrate that DATE can be applied on top of SD3, SD-XL, and FLUX, all of which use multiple text embedding. For each model, we follow the default configurations provided in the HuggingFace diffusers library and use ImageReward as the evaluation function.

[Table: Results on COCO with recent stronger base models]

The results above follow the evaluation setup in Table 1 of the main text. Across all modern architectures, DATE consistently improves performance on all metrics compared to the fixed text embedding baseline. These results demonstrates that DATE can be effectively integrated with models that utilize multiple text embeddings, providing a scalable and complementary improvement.

---

W3: [Related works]

We agree that a more comprehensive comparison including methods for prompt optimization, text embedding tuning, and reward-based fine-tuning would strengthen the paper. In the revised version, we include a detailed discussion and additional experiments covering the following areas, as below.

As discussed in Section 2.2 of the main text, methods for refining text-to-image generation can generally target one of the following: model parameters, perturbed data, or text embeddings. The work in [4], as well as (Black et al., 2024) cited in our paper, falls into the first category. These approaches fine-tune the model parameters using a reward function. However, they require significant training time and computational resources.

Prompt-level optimization targets the input text. Several learning-based approaches (Hao et al., 2023, Mo et al., 2024) have explored prompt optimization using reward models and reinforcement learning. These methods train a language model to generate prompts optimized for a diffusion model based on reward signals. However, such methods typically require significant training time and computational resources. Additionally, if the diffusion backbone or reward function changes, the model could be retrained. In contrast, DATE is training-free and operates at inference time. Moreover, these learning-based previous works could be used in conjunction with DATE during test-time for complementary benefits.

There is also related work focusing on tuning text embeddings directly. For example, [3] learns a special token embedding using a small number of personalized images, which is then appended to prompts for personalized image generation. Similarly, (Um et al., 2025) introduce a method for generating minority instances by learning a special token embedding, further extending it to be time-dependent during sampling. While [3] and (Um et al., 2025) focus on specific tasks, DATE proposes a general-purpose framework for improving text-to-image alignment. Additionally, a key distinction lies in the granularity of updates: [3] and (Um et al., 2025) optimize a special token embedding, whereas DATE updates the entire text embedding vector.

We conduct additional experiments by combining DATE with Promptist (Hao et al., 2023), a representative prompt tuning method. We use ImageReward as the evaluation function for DATE.

[Table: Results with the prompt tuning, Promptist (Hao et al., 2023). FID values may differ in scale due to the smaller sample size (1k).]

Promptist focuses on improving aesthetic quality (Aesthetic score), often at the cost of other metrics. In contrast, DATE improves all metrics, including Aesthetic score, which is not used in its objective, indicating good generalization. When combined with Promptist, DATE helps recover image-text alignment and overall quality while maintaining high aesthetic quality. This suggests DATE can complement prompt tuning methods.

(Black et al., 2024) Training Diffusion Models with Reinforcement Learning, ICLR 2024.
(Hao et al., 2023) Optimizing Prompts for Text-to-Image Generation, NeurIPS 2023.
(Mo et al., 2024) Dynamic Prompt Optimizing for Text-to-Image Generation, CVPR 2024.
(Um et al., 2025) Minority-Focused Text-to-Image Generation via Prompt Optimization, CVPR 2025.

---

W4: [Deviation from the original intent]

We acknowledge the potential risk that adapting the text embedding might deviate from the user’s original intent. However, we believe DATE can actually help better reflect that intent through a feedback-driven refinement process. Our method updates the text embedding based on the expected image output $\bar{x}_0$ generated from the current embedding, guided by an external evaluation function. This allows the embedding to shift in a direction that better aligns with the user’s intent as captured by the evaluation signal. In contrast to fixed text embeddings, which rely solely on the static encoding of the input text, DATE introduces a feedback loop between the intermediate generated image and the text embedding, making it more adaptive to the desired semantics.

Moreover, Theorem 2 provides a theoretical justification showing that DATE updates the text embedding by aligning with the evaluation function while also accounting for the original diffusion model’s text-conditional likelihood. This regularization effect helps prevent overfitting to the evaluation function or losing alignment with the original prompt semantics.

That said, we agree that since the base diffusion model was not trained with dynamically updated text embeddings, there may be a training-inference gap. To address this, we explicitly introduce a constraint on the embedding update: $||c-c_{org}||_2 \leq \rho$ in Eq. (7). This constraint allows us to balance preserving the original text semantics while adapting to the evaluation feedback. As shown in the sensitivity analysis in Figure 6 of the main text, performance improves when $\rho$ is set appropriately, supporting the effectiveness of this trade-off.

We thank the reviewer for the valuable feedback. Below are our responses to the concerns.

---

W1: [Computational overhead]

The increased GPU memory usage in DATE mainly stems from gradient computations, as noted in the Limitations section. To address this, we apply half-precision (FP16) inference during sampling. As shown below, this reduces runtime and memory with only a slight performance drop. Still, DATE outperforms fixed embedding.

[Table: Results on COCO with SD v1.5 of Table 1 in the main text, including FP16.]

One important note is that in our implementation, casting the CLIP model used in the evaluation function to FP16 resulted in degraded performance. Therefore, we kept this component in full precision. Nevertheless, since the diffusion model accounts for most of the computational overhead, applying FP16 to the rest of the pipeline still provides substantial savings.

It is also worth noting that FP16 is a general memory-efficient strategy that can be applied across different sampling methods. In this context, our results show that DATE remains compatible with such strategies, despite its gradient computations. However, we do not claim that FP16 integration represents a fundamental improvement to DATE itself in terms of memory or speed. Rather, it shows that DATE can effectively incorporate standard efficiency techniques without loss of compatibility.

---

W2: [Dependency on reward function]

We acknowledge that DATE depends on the reward function, as discussed in Appendix E. Here, we offer theoretical, empirical, and practical perspectives.

(Theoretical perspective)

Theorem 2 shows DATE aligns text embeddings with the reward function while also regularizing with the model's text-conditional likelihood. This means that DATE does not blindly overfit to the reward function but instead applies regularized guidance informed by the underlying diffusion model. This effect can reduce the risk of overfitting to the proxy metrics.

(Empirical perspective)

Empirically, as shown in Table 1 of the main text, DATE improves performance on both the reward function used for optimization and other metrics not used during optimization (e.g., FID). To further validate this generalization, we conduct additional experiments using PickScore (Kirstain et al., 2023) as both an optimization objective and an external evaluation metric. As shown in the table, DATE consistently improves performance across all metrics. This suggests that DATE’s improvements reflect broader enhancements in generation quality. Ongoing research continues to explore more faithful evaluation metrics for text-to-image generation, and DATE can be readily combined with such advances.

[Table: Results on COCO with SD v1.5 of Table 1 in the main text, including PickScore.]

(Kirstain et al., 2023) Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation, NeurIPS 2023.

(Practical perspective)

Our method also supports multi-objective optimization, which can further mitigate overfitting to a single metric. In Figure 4 of the main text, we show results using a combined objective of CLIP score and ImageReward. Interestingly, this combination outperforms the use of each metric individually. This result highlights the potential of our method to leverage synergy between multiple objectives for more robust guidance.

In summary, while reward function dependency is an important consideration, DATE incorporates both theoretical regularization and practical flexibility to address it effectively.

---

Q1: [Long and complex prompts]

Text encoders in diffusion models are often limited by a fixed token length, which can pose challenges for handling long and complex prompts. For example, many diffusion models using CLIP encoders are restricted to 77 tokens, and the tokens beyond this limit are typically truncated and ignored. Since DATE provides guidance through an external function and does not rely solely on the internal text encoder. This allows DATE to influence the generation process beyond the model’s fixed token-length limitation, making it a promising solution for handling long and complex prompts.

To investigate this, we perform experiments on the LongAlign (Liu et al., 2025), which contains prompts of substantial length and complexity. We sample 300 prompts and use ImageReward (20% updates) as the evaluation function. For ImageReward, each long prompt is split into individual sentences, and at each update step, one sentence is randomly selected to compute the score and guide the embedding update. We use Denscore (Liu et al., 2025) for the metric.

[Table: Results for long prompts on LongAlign dataset (SD v1.5)]

[Table: Average Denscore by sentence position]

These results show that DATE improves Denscore compared to the baseline. Notably, the improvement is more significant for sentences appearing later in the prompt, suggesting that DATE can inject additional semantic guidance in regions where the base model typically underperforms due to token truncation. These results suggest that DATE can serve as an effective mechanism for injecting auxiliary information into diffusion models, especially in scenarios where prompt length exceeds the native capacity of the text encoder.

(Liu et al., 2025) Improving Long-Text Alignment for Text-to-Image Diffusion Models, ICLR 2025.

---

Q2: [Diversity]

As discussed in our response to W2, our theoretical analysis based on Theorem 2 suggests that DATE is regularized by the underlying diffusion model, which helps mitigate overfitting to the reward function. Additionally, since DATE does not modify model weights and instead performs instance-specific updates during inference, we believe its impact on sample diversity is limited.

To empirically validate this, we measure diversity using Recall, the metric for evaluating the diversity of generated images in generative models.

[Table: Results on the COCO from Table 1 in the main paper with SD v1.5 including Recall.]

As shown in the results, DATE improves both text-image alignment and sample diversity. We speculate that giving the model the opportunity to dynamically update the text embedding across timesteps, rather than relying on a fixed embedding, may have encouraged greater exploration during sampling. However, the underlying mechanism behind this effect warrants further investigation.

---

Q3: [Combined with prompt tuning]

Our method can be combined with prompt tuning and soft prompt methods. Several learning-based approaches (Hao et al., 2023, Mo et al., 2024) have explored prompt optimization using reward models and reinforcement learning. While such approaches focus on optimizing prompt tokens or prompt-generating models, DATE operates at inference time by directly adjusting the text embeddings without any additional training. Therefore, they can be used together: refined prompts from these methods can serve as the initial input, and DATE can further refine the embedding during sampling to better align with the evaluation function.

We conduct additional experiments by combining DATE with Promptist (Hao et al., 2023), a representative prompt tuning method. We use ImageReward as the evaluation function for DATE.

[Table: Results with the prompt tuning, Promptist (Hao et al., 2023). FID values may differ in scale due to the smaller sample size (1k).]

Promptist focuses on improving aesthetic quality (represented by Aesthetic score), often at the cost of other metrics like CLIP score, FID, and ImageReward. In contrast, DATE improves all metrics, including Aesthetic score, which is not used in its objective, indicating good generalization. When combined with Promptist, DATE helps recover image-text alignment and overall quality while maintaining high aesthetic quality. This suggests DATE can complement prompt tuning methods.

(Hao et al., 2023) Optimizing Prompts for Text-to-Image Generation, NeurIPS 2023.
(Mo et al., 2024) Dynamic Prompt Optimizing for Text-to-Image Generation, CVPR 2024.

We appreciate the thorough reviews and valuable comments. We address the concerns below.

---

W1 & W2: [Previous works]

Thank you for pointing out these relevant lines of work. We agree that a more comprehensive comparison with prompt optimization and guidance strategies would strengthen the paper. In the revised version, we include both a detailed discussion and additional experiments covering the following areas, as below.

(Prompt optimization method)

Several learning-based approaches [5, 7] generate static prompts via training, often with reward models or reinforcement learning. These methods are effective but costly, and fixed prompts lack adaptability during sampling. In contrast, DATE is training-free and dynamically updates text embeddings at inference time across timesteps. This enables fine-grained control and responsiveness to intermediate generations.

[6] is a concurrent method that also performs inference-time prompt optimization by updating a placeholder token for minority instance generation. While [6] focuses on a specific task and optimizes a single token, DATE is a general framework that updates the full embedding vector and is applicable across tasks. [8] proposes a training-free, gradient-free method that leverages two LLMs to iteratively refine prompts. While this method avoids training, it requires multiple LLM inferences and image generations per prompt, making it computationally expensive. In contrast, DATE uses a single image trajectory per instance and updates the text embedding sequentially at each timestep (Motivation 1 from Section 3.1 in the main paper).

We evaluate DATE individually and in combination with Promptist [7] (trained prompts for aesthetics) using SD v1.4 and ImageReward:

[Table: Results with the prompt tuning method, Promptist [7]. Note: FID values may differ in scale from those in the main paper due to the smaller sample size (1k).]

Promptist focuses on improving aesthetic quality (represented by Aesthetic score), often at the cost of other metrics like CLIP score, FID, and ImageReward. In contrast, DATE improves all metrics, including Aesthetic score, which is not used in its objective, indicating good generalization. When combined with Promptist, DATE helps recover image-text alignment and overall quality while maintaining high aesthetic quality. This suggests DATE can complement prompt tuning methods by improving balance and robustness during inference.

(Guidance strategies)

As discussed in the main paper, Universal Guidance (UG) applies external functions to guide sampling in the perturbed data space. [10] targets minority sample generation by proposing a metric that can be internally computed within the diffusion model and applying guidance in the data space.

In contrast, DATE applies guidance in the text embedding space, which introduces a fundamental distinction from data-space guidance methods. In Section 4.3, Theorem 2 provides a theoretical foundation showing that DATE updates the text embedding in a direction that aligns with the evaluation function while simultaneously accounting for the model’s original text-conditional likelihood. This enables DATE to strike a better balance between semantic alignment and preservation of the diffusion model’s distribution. Experimentally, we also show in Table 1 that DATE outperforms UG.

Meanwhile, CADS [9] shows that injecting time-dependent Gaussian noise into the conditional embedding can improve generation diversity. We empirically compare DATE with CADS and evaluate their combination. The experiments follow the same setup as Table 1 in the main paper, with DATE using 10% update with ImageReward configuration. Given CADS’s known effectiveness in improving diversity, we include Recall as an additional diversity metric.

[Table: Performance on the COCO validation set with CADS [9], including Recall metric.]

As shown in the table, CADS (third row) improves diversity, as reflected by better FID and Recall, but at the cost of reduced text-image alignment, with lower CLIP score and ImageReward. In contrast, DATE (second row) improves both alignment and diversity metrics. When CADS and DATE are combined (last row), the result shows a synergistic effect: FID and Recall improve further, while alignment metrics also increase relative to CADS alone. This suggests that while CADS excels in enhancing diversity, it does not improve text-image alignment, which can be effectively complemented by combining it with DATE.

---

W3: [Qualitative results]

Thank you for the suggestion. We currently provide qualitative comparisons for our application tasks in the main text (Figures 11 and 12) and in the appendix (Figures 17 and 18). The review system does not allow uploading additional images during the rebuttal period. In our revision, we will add diverse generated samples, e.g., COCO-prompt images corresponding to Table 1, to provide clearer visual comparisons.

---

W4: [Lightweight method]

The increase in GPU memory usage for DATE is primarily due to the gradient computations, as noted in the Limitations section. To address this, we explore memory-efficient techniques. In particular, we apply half-precision (FP16) inference during the sampling process. As shown in the below table, this reduces runtime and memory consumption, albeit at the cost of slightly decreased performance. Nevertheless, our methods still outperform fixed embedding methods.

[Table: Computational resources and performance on the COCO validation set. This is an extended version of Table 1 in the main text, including results with half-precision.]

One important note is that in our implementation, casting the CLIP model used in the evaluation function to FP16 resulted in degraded performance. Therefore, we kept this component in full precision. Nevertheless, since the diffusion model accounts for most of the computational overhead, applying FP16 to the rest of the pipeline still provides substantial savings.

It is also worth noting that FP16 is a general memory-efficient strategy that can be applied across different sampling methods. In this context, our results demonstrate that DATE remains compatible with such strategies, despite its additional gradient computations. However, we do not claim that FP16 integration represents a fundamental improvement to DATE itself in terms of memory or speed. Rather, it shows that DATE can effectively incorporate standard efficiency techniques without loss of compatibility.

---

Q1: [Another metric for optimization]

Our method is compatible with any evaluation metric that provides access to model weights. We experiment with Pickscore (Kirstain et al., 2023), another popular text-to-image evaluation metric, as the optimization objective. Similar to our results with CLIPScore and ImageReward, using PickScore also led to consistent improvements across all evaluation metrics. Additionally, when evaluating images optimized with other metrics using PickScore, our method still outperformed the fixed embedding baseline.

[Table: Performance on the COCO validation set using Stable Diffusion v1.5 with DDIM sampler. This is an extended version of Table 1 in the main text, including results with PickScore.]

(Kirstain et al., 2023) Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation, NeurIPS 2023.