CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation

We thank the reviewer for the valuable and insightful feedback！

1. Clarification on approximating an intractable expectation using Jensen’s inequality

We acknowledge the reviewer's concern regarding the use of Jensen’s inequality in our derivation of Eq. 12. Specifically, we approximate the intractable expectation as follows:

please refer to Eq.39 and Eq.40 in appendix for more details.

Although applying Jensen’s inequality in this manner provides an upper bound rather than the exact loss, our extensive empirical evaluations across multiple benchmarks (e.g., Tables 1–3) indicate that the resulting training dynamics lead to consistent improvements in generation quality and robust convergence. In practice, the bias introduced by this approximation is effectively absorbed during optimization, and the desired preferred/dispreferred split is maintained.

While a tighter approximation might further reduce any potential bias, the experimental results confirm that our current approximation does not lead to suboptimal convergence or a misrepresentation of the two distributions. Quantifying the exact impact of this inequality remains challenging due to the inherent complexity of the diffusion process. However, the empirical performance gains observed across diverse datasets and model architectures strongly suggest that the approximation error is minimal.

2. Validation on different dataset

Although the 7,459-pair dataset (OIP) demonstrates our method's high data efficiency, we have also evaluated CHATS on a larger and noisier dataset. Our experiments on the PaP v2 dataset, which comprises 851,293 preference pairs, are reported in Table 5 and confirm that CHATS consistently outperforms baseline methods on both datasets. Furthermore, our results indicate that training with the higher-quality OIP dataset yields better performance, as discussed in "A small high-quality preference dataset is enough" (Line 379, right column). Detailed information about the datasets can be found in the appendix, Line 681-687.

3. More analysis on sensitivity of $\alpha$ and $s$

We conduct additional experiments to further examine the impact of hyperparameters. The results on SDXL are summarized below.

For the guidance scale $s$, we obtained the following:

These results indicate that while our default value $s=5$ works well, a value of $s=4$ yields a slightly higher score. It is important to note that $s$ is a user-specified hyperparameter that governs the trade-off between concentrating generation on high semantic-density areas and maintaining output diversity. As such, it is not unique to CHATS and was not extensively tuned in our framework.

Regarding $\alpha$, our experiments yield the following:

These results confirm that the best performance is achieved around $\alpha = 0.5$, which is consistent with our analysis presented in Figure 5 and Line 765-769 of the appendix.

In summary, our ablation studies demonstrate that CHATS is robust to moderate variations in these hyperparameters. The default settings of $s=5$ and $\alpha=0.5$ represent a high-quality generation, with minor adjustments yielding comparable performance.

4. Computational cost

We would like to point out that Table 6 already reports computational cost metrics in terms of images generated per second, comparing CHATS with single-model methods like Diffusion-DPO. In addition, we have performed supplementary experiments to address the increased cost introduced by the dual-model architecture. As explained in Line 436-439 (right column), by simultaneously distilling both the guidance scale (i.e., $s$ in Eq. 17) and the two models into a single model, the extra inference cost can be completely eliminated while still achieving high-quality generation. For example, the following table shows that our distillation variant (CHATS-distill) not only reduces memory usage and increases throughput relative to CHATS but also maintains the improved HPS v2 score:

Thus, while the dual-model approach in CHATS does introduce additional cost in its raw form, our distillation strategy fully recovers efficiency without compromising the quality of the generated images.

Thank you for your helpful feedback and questions! Due to space limitations, we provide responses to your main comments. Further questions can be discussed in subsequent responses.

1. Theoretical foundations & convergence properties of CHATS

Given that DPO is invariant to affine transformations of the reward, for reward $R'(z_{0:T}, c) = a \cdot R(z_{0:T}, c) + b$, the optimal policy becomes (cf. Eq.29):

with $\beta$ omitted for simplicity.

CHATS decomposes reward of traditional DPO (Eq.30) into two parts:

with $\beta$ and $\log Z(c)$ omitted since they are constants for optimization. Defining the effective reward as: $R_{\mathrm{CHATS}} = R^+(\theta^+) + R^-(\theta^-)$, the optimal distribution becomes:

Under the assumption of $L$-smoothness and using standard gradient descent, we obtain the inequality:

which ensures the CHATS loss (Eq. 9) decreases and converges. Since their combination recovers the same optimal joint distribution as traditional DPO methods [1], CHATS preserves theoretical foundations of DPO.

2. Dual training signals

In our training procedure, we do not split the data into separate (dis)preferred subsets. Instead, when minimizing losses such as those in Eq. 13 and 14, we select a ranked preference pair $(z_0^+, z_0^-)$ from the entire dataset. This single pair is then used to simultaneously update both the preferred and dispreferred models within a unified dual-objective framework. This joint training approach ensures that both models receive complementary signals derived from the ranked pair.

3. Clarification on Eq. 15

As indicated in Line 220–226 (right column), when $0 < \alpha < \frac{1+s}{s}$, the dispreferred distribution $p_{\theta^-}$ is partially incorporated so that it contributes useful patterns while remaining less influential than $p_{\theta^+}$. In other words, the dispreferred model's noise prediction is toned down, partially suppressing undesirable patterns while preserving beneficial information. Conversely, when $\alpha < 0$, the terms $p_{\theta^-}(z_t | c)^{\alpha s}$ and $p_{\theta^-}(z_t)^{-(1+\alpha)s}$ actively push samples away from undesired modes, effectively suppressing the entire output of the dispreferred model, similar to using a null prompt $\varnothing$.

4. Justification on proxy prompt

Generative models represent prompts in a continuous embedding space where linear operations reflect meaningful semantic changes. For example, word embedding arithmetic [2] (e.g., “queen” ≈ “king” – “man” + “woman”) shows that semantic attributes can be linearly combined. Thus, interpolating between a prompt $c$ and the null prompt $\varnothing$ (i.e., forming $\hat{c} = -\alpha c + (1+\alpha) \varnothing$) not only reliably captures the semantic content but also reduces the number of forward passes—from 3 in Eq.16 to 2 in Eq.17, thereby cutting inference cost by about 1/3.

5. Memory efficiency

While the dual architectures in CHATS introduce additional cost (see Table 6), as noted in Line 436–439 (right column), this extra inference cost can be completely eliminated via distillation. As shown in the table below, by simultaneously distilling the guidance scale (i.e., $s$ in Eq. 17) and the two models into a single one, CHATS achieves both high efficiency and high-quality generation (Model: SDXL).

6. Justification on Table 4

Even though a single model trained on the full dataset outperforms the baseline, our CHATS method consistently delivers further improvements as shown in Table 4. Similar trends are observed with SDXL:

7. More analysis on $\alpha$

We show more ablation on $\alpha$ on table below (SDXL, $s$=5):

We observe the best choice of $\alpha$ occurs around 0.5, consistent with our analysis in Fig.5 and Line 765-769 in appendix.

References
[1] Diffusion Model Alignment Using Direct Preference Optimization, CVPR'24
[2] Distributed Representations of Words and Phrases and their Compositionality, NeurIPS'13

Thank you for your comments.

1. Novelty

We respectfully disagree with the assertion that CHATS merely combines existing technologies. In our work, we explicitly differentiate our approach from prior DPO methods for diffusion models in related work [1–3] in Line 135-146 (left column). In fact, the 3 papers you cited are evidences that support our novelty rather than undermining it. The key insight of our CHATS is to integrate human preference optimization finetuning with the sampling process, leveraging their inherent synergy to refine image generation. However, none of DPO methods you mentioned explore it.

Moreover, Null-text Inversion [4] concentrates on image editing by optimizing the null-text embeddings using gradient updates to achieve a superior DDIM inversion. In contrast, the proxy-prompt-based sampling strategy employed by CHATS is primarily designed to enhance sampling efficiency by reducing the number of forward passes from 3 in Eq.16 to 2 in Eq.17, while all prompt features remain frozen and untrained. Additionally, [4] neither utilizes human preference data nor trains separate models to explicitly capture the distributions of preferred and dispreferred images, and the dual architectures used by CHATS naturally separate these distributions into two distinct parts. As a result, our method is not simply a combination of existing techniques but rather a novel integration that leverages human preference data to improve the generative process. To the best of our knowledge, such an approach has not been previously explored in the context of text-to-image generation.

2. Clarification on performance improvement

We acknowledge that the numerical improvements, especially when comparing SDXL + Diffusion-DPO with CHATS, may appear modest on individual benchmarks. However, we emphasize that the improvements are consistent across multiple benchmark evaluations and are observed in both diffusion models (SDXL) and flow matching models (In-house T2I). Our extensive evaluations across aesthetic scores, GenEval, and DPG-Bench consistently demonstrate that CHATS improves aesthetic alignment and generation quality. The consistent gains across diverse datasets and model architectures validate the effectiveness of our approach. Moreover, these improvements are achieved with only a small high-quality fine-tuning dataset, highlighting the data efficiency of CHATS.

3. Clarification of deployment cost

While CHATS introduces a dual-model architecture that, if used naively, doubles the mode size and slightly decreases the throughputs compared to a single diffusion model (as shown in Table 6), this cost can be completely eliminated through distillation. As noted in Line 436–439 (right column), by simultaneously distilling the guidance scale (i.e., $s$ in Eq. 17) and the two models into a single model, we achieve both high efficiency and high-quality generation. For example, our distillation variant, “CHATS-distill,” attains a memory footprint and throughput comparable to or even exceeding that of the standard model, while retaining the improved HPS v2 score, as demonstrated in the following table:

Thus, while the raw dual-model approach incurs additional computational cost, our results show that a distilled version can match or surpass the efficiency of a single model without compromising the quality gains provided by CHATS.

References

[1] Diffusion Model Alignment Using Direct Preference Optimization, CVPR’24.
[2] Training Diffusion Models with Reinforcement Learning, ICLR’24.
[3] DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models, NeurIPS’23.
[4] Null-text Inversion for Editing Real Images using Guided Diffusion Models, CVPR’23.